# Working with large ROS bag files on Hadoop and Spark

**URL:** <https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314>\
**Category:** Projects\
**Created:** [July 26, 2017, 3:11pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314 "2017-07-26T15:11:39Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jan/32/870_2.png) [@jan](https://discourse.openrobotics.org/u/jan)\
**Post date:** [July 26, 2017, 3:11pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/1 "2017-07-26T15:11:39Z")

</div>

Large amount of sensor and robotic data is produced by the industry at an ever increasing peace. Be it from areas like mobility, perception, smart factory or from development tools through planing, modelling or simulation.

New effervescent robotic topics of research like self driving cars put pressure to develop new tools and techniques to deal with larger and more complex data sets. Some projects and industry players publicly announced the adoption of ROS as part of their process.

On the other hand, **Hadoop** and **Spark** Ecosystems are seeing a tremendous adoption for processing and analysing large data in parallel. (The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets)

**Why process large ROS bag files in parallel?**

 ![08](https://us1.discourse-cdn.com/flex022/uploads/ros/original/1X/d04ed3baa759c4cef69d41fac1c97122959f42f2.png)

ROS command **rosbag record** subscribes to topics and writes a bag file with the contents of all messages published on those topics. For performance reasons the messages are written interlaced as they come over the wires, with different frequencies.

**Associative** operations can be applied in parallel. Or more precisely the parallelism requires associativity. (Although concurrency technically is not parallelism it also requires associativity.) Spark provides an unified functional API for processing locally, concurrently or on multiple machines.

**Now you do not need to convert ROS bag files to work with them in Spark**

The assumption was that the ROS bag files have to be converted into a more suitable format before they can be processed in parallel with tools like Hadoop or Spark. It turns out that the format is good enough for processing with a distributed file system like HDFS but it happened that nobody has written an Hadoop InputFormat for it.

So we did it. We took the time and wrote a Hadoop RosbagInputFormat 😀 published under Apache 2.0 License.

[http://github.com/valtech/ros\_hadoop](http://github.com/valtech/ros_hadoop)

RosbagInputFormat is an open source splittable Hadoop InputFormat for the rosbag file format.

 ![16](https://us1.discourse-cdn.com/flex022/uploads/ros/original/1X/3e98bb3b9a14f0653b62244e6c03ba9906f9ea93.png)

We also prepared a Dockerfile and step-by-step tutorial that you could use to try the concepts presented here:

[http://github.com/valtech/ros\_hadoop](http://github.com/valtech/ros_hadoop)

We hope that the RosbagInputFormat would be useful for you. It would be great if you give us some feedback.

Thanks!  
Adrian, Jan

---

<div class="post-metadata">

**Author:** ![Dejan\_Pangercic](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/dejan_pangercic/32/293_2.png) [@Dejan\_Pangercic](https://discourse.openrobotics.org/u/Dejan_Pangercic)\
**Post date:** [July 26, 2017, 6:37pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/2 "2017-07-26T18:37:45Z")

</div>

Jan, this seems great. Is it meant to run in the data center or as an on-premise solution? In other words are there any special requirements on the computer HW?

Did you do any profiling on how much faster can you process data using this spliter + spark vs if you’d just do it sequentially by playing a bag from an ext4 formatted disk?

D.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jan/32/870_2.png) [@jan](https://discourse.openrobotics.org/u/jan)\
**Post date:** [July 26, 2017, 7:42pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/3 "2017-07-26T19:42:45Z")

</div>

Both are possible, data center and on-premise. There are no hardware requirements but we recommend a 3 nodes setup to see the benefits of parallelism. Start with 64-128 GB memory, 4 disks and quad-core per node.

Spark performance really scale out with multiple machines. If there are 3 splits is 3 times faster with 3 workers. etc.

---

<div class="post-metadata">

**Author:** ![joyliu329](https://avatars.discourse-cdn.com/v4/letter/j/d6d6ee/32.png) [@joyliu329](https://discourse.openrobotics.org/u/joyliu329)\
**Post date:** [December 21, 2017, 3:58am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/4 "2017-12-21T03:58:02Z")

</div>

HI JAN  
can you open rosbaginputformat\_2.11-0.9.3.jar source on github ros\_hadoop project ? i don’t know how it works. 3Q

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jan/32/870_2.png) [@jan](https://discourse.openrobotics.org/u/jan)\
**Post date:** [December 21, 2017, 9:02pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/5 "2017-12-21T21:02:04Z")

</div>

The scala source code is in the [ros\_hadoop/src/main/scala at master · autovia/ros\_hadoop · GitHub](https://github.com/valtech/ros_hadoop/tree/master/src/main/scala)

The jar is packaged by the build script [ros\_hadoop/build.sbt at master · autovia/ros\_hadoop · GitHub](https://github.com/valtech/ros_hadoop/blob/master/build.sbt)

It works similar to many other Hadoop InputFormats.

---

<div class="post-metadata">

**Author:** ![Joe\_Gifford](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Gifford](https://discourse.openrobotics.org/u/Joe_Gifford)\
**Post date:** [April 24, 2018, 6:25pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/6 "2018-04-24T18:25:07Z")

</div>

Hi Jan,

Will there be any plans to write up a tutorial for the scala api? I am able to pull the topic names but it seems a little cumbersome compaired to the python api.

Thanks

---

<div class="post-metadata">

**Author:** ![Deepak3994](https://avatars.discourse-cdn.com/v4/letter/d/c57346/32.png) [@Deepak3994](https://discourse.openrobotics.org/u/Deepak3994)\
**Post date:** [August 6, 2018, 10:18am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/7 "2018-08-06T10:18:59Z")

</div>

Hi Jan,

I was trying to implement “flux-project” from github “[GitHub - flux-project/flux: Machine Learning Stack for Big Data, Big Cluster and Big Challenges](https://github.com/flux-project/flux)”. But I am facing a error as “ImagePullBackOff”.

I wanted to know where the image “pod/flux-ros-hadoop-5f8cd4bc67-h5thl” is located.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jan/32/870_2.png) [@jan](https://discourse.openrobotics.org/u/jan)\
**Post date:** [October 16, 2018, 9:09am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/8 "2018-10-16T09:09:10Z")

</div>

Hi Joe,

Basically an Scala solution without ROS means effort of reimplementing parts of ROS in Scala. Spark might be Scala/Python but ROS is C++/Python so the glue is Python not Scala.

---

<div class="post-metadata">

**Author:** ![jie\_liu](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jie_liu/32/3141_2.png) [@jie\_liu](https://discourse.openrobotics.org/u/jie_liu)\
**Post date:** [March 8, 2019, 2:43pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/9 "2019-03-08T14:43:29Z")

</div>

![image](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/c/cda52f918afd43c7dac605f53b9ea34a517f8316.jpeg)

i have run the programme on spark-shell ,but i got a exception .  
scala\> fin = sc.newAPIHadoopFile(  
| path = “hdfs://127.0.0.1:9000/user/root/HMB\_4.bag”,  
| inputFormatClass = “de.valtech.foss.RosbagMapInputFormat”,  
| keyClass = “org.apache.hadoop.io.LongWritable”,  
| valueClass = “org.apache.hadoop.io.MapWritable”,  
| conf = {“RosbagInputFormat.chunkIdx”:“/srv/data/HMB\_4.bag.idx.bin”})  
:6: error: identifier expected but string literal found.  
conf = {“RosbagInputFormat.chunkIdx”:“/srv/data/HMB\_4.bag.idx.bin”})

do you know why ? Eagerly awaiting your reply..

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jan/32/870_2.png) [@jan](https://discourse.openrobotics.org/u/jan)\
**Post date:** [March 8, 2019, 6:10pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/10 "2019-03-08T18:10:33Z")

</div>

Hi Jie,

In our tutorial we used Python code not Scala. Please use PySpark and Python.

---

<div class="post-metadata">

**Author:** ![jie\_liu](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jie_liu/32/3141_2.png) [@jie\_liu](https://discourse.openrobotics.org/u/jie_liu)\
**Post date:** [March 15, 2019, 4:24am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/11 "2019-03-15T04:24:09Z")

</div>

thank you very much . i have a other problem,the site : [http://flux-project.org/](http://flux-project.org/) does not work. 😅

---

<div class="post-metadata">

**Author:** ![jie\_liu](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jie_liu/32/3141_2.png) [@jie\_liu](https://discourse.openrobotics.org/u/jie_liu)\
**Post date:** [March 18, 2019, 2:25am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/12 "2019-03-18T02:25:25Z")

</div>

Hi Jan ，  
did you know how to extract the pcd file from rosbag with hadoop&saprk ? I don’t see this example in the project.

---

<div class="post-metadata">

**Author:** ![raysayandeep](https://avatars.discourse-cdn.com/v4/letter/r/c4cdca/32.png) [@raysayandeep](https://discourse.openrobotics.org/u/raysayandeep)\
**Post date:** [April 10, 2019, 2:20pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/13 "2019-04-10T14:20:31Z")

</div>

Hi Jan,

I am using Azure HDInsight Spark cluster to extract data from rosbag files using RosbagInputFormat. I have followed the readme file. While running the code in pyspark I am getting the following error,

It is not able to read the idx file from local system.

```
Using Python version 2.7.12 (default, Jul 2 2016 17:42:40)
SparkSession available as 'spark'.
>>> sc.newAPIHadoopFile(
... path = "/user/spark/HMB_4.bag",
... inputFormatClass = "de.valtech.foss.RosbagMapInputFormat",
... keyClass = "org.apache.hadoop.io.LongWritable",
... valueClass = "org.apache.hadoop.io.MapWritable",
... conf = {"RosbagInputFormat.chunkIdx":"/opt/ros_hadoop/master/dist/HMB_4.bag.idx.bin"})
[Stage 0:> (0 + 1) / 1]19/04/10 14:16:35 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, wn1-avdp-h.cfzrwlyaxyyuvies4sglc0tsud.cx.internal.cloudapp.net, executor 5): java.io.FileNotFoundException: /opt/ros_hadoop/master/dist/HMB_3.bag.idx.bin (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)

```

Could you please help me with that?

Thanks,  
Sayandeep

---

<div class="post-metadata">

**Author:** ![tfoote](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/tfoote/32/3813_2.png) [@tfoote](https://discourse.openrobotics.org/u/tfoote)\
**Post date:** [April 11, 2019, 12:49am UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/14 "2019-04-11T00:49:19Z")

</div>

@raysayandeep Thanks for your question. However we ask that you please ask questions on [http://answers.ros.org](http://answers.ros.org) following our support guidelines: [Support - ROS Wiki](http://wiki.ros.org/Support)

ROS Discourse is for news and general interest discussions. [ROS Answers](http://answers.ros.org) provides a forum which can be filtered by tags to make sure the relevant people can find and/or answer the question, and not overload everyone with hundreds of posts.

---

<div class="post-metadata">

**Author:** ![mohamed\_stage](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/mohamed_stage/32/17480_2.png) [@mohamed\_stage](https://discourse.openrobotics.org/u/mohamed_stage)\
**Post date:** [May 15, 2023, 3:02pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/15 "2023-05-15T15:02:31Z")

</div>

import [org.apache.hadoop.io](http://org.apache.hadoop.io).{LongWritable, MapWritable}

val path = “hdfs://hadoop-master:9000/user/root/fichier\_ros.bag”  
val inputFormatClass = classOf[io.autovia.foss.RosbagMapInputFormat]  
val keyClass = classOf[LongWritable]  
val valueClass = classOf[MapWritable]  
val conf = new org.apache.hadoop.conf.Configuration()  
conf.set(“RosbagInputFormat.chunkIdx”, “hdfs://hadoop-master:9000/user/root/fichier\_ros.bag.idx.bin”)  
val rdd = sc.newAPIHadoopFile(path, inputFormatClass, keyClass, valueClass, conf)

I have same errors in this code  
java.io.FileNotFoundException: hdfs:/hadoop-master:9000/fichier\_ros.bag.idx.bin (No such file or directory)

plz i need to creat RDD in spark with fille rosbag if you have same ideas

---

<div class="post-metadata">

**Author:** ![Katherine\_Scott](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/katherine_scott/32/4434_2.png) [@Katherine\_Scott](https://discourse.openrobotics.org/u/Katherine_Scott)\
**Post date:** [May 15, 2023, 3:31pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/16 "2023-05-15T15:31:26Z")

</div>

I would recommend using either [answers.ros.org](http://answers.ros.org), contacting the original author via DM, or filing an issue on the source repository. This ROS Discourse thread has been inactive for four years.

---

<div class="post-metadata">

**Author:** ![Alec\_Bell](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/alec_bell/32/15318_2.png) [@Alec\_Bell](https://discourse.openrobotics.org/u/Alec_Bell)\
**Post date:** [June 9, 2023, 2:42pm UTC](https://discourse.openrobotics.org/t/working-with-large-ros-bag-files-on-hadoop-and-spark/2314/17 "2023-06-09T14:42:14Z")

</div>

Hey Mohamed! We (SensorSurf) built an open-source package to help you load all your MCAP or ROS Bag data into a Timescale database so you can run SQL queries. This might help solve some of your pains. Check it out here: [https://github.com/SensorSurf/mcap-etl](https://github.com/SensorSurf/mcap-etl)