# New perception architecture: message types

**URL:** <https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564>\
**Category:** Autoware\
**Created:** [April 2, 2019, 6:34am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564 "2019-04-02T06:34:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![yukkysaito](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/yukkysaito/32/3272_2.png) [@yukkysaito](https://discourse.openrobotics.org/u/yukkysaito)\
**Post date:** [April 2, 2019, 6:34am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/1 "2019-04-02T06:34:02Z")

</div>

We have been discussing the architecture of the perception pipeline in [issue  
#1501](https://github.com/autowarefoundation/autoware/issues/1501). However, the discussion stopped  
before it converged.

The current perception architecture does not contain the information necessary for planning to  
avoid obstacles or stop in front of an obstacle. These features were not considered when the architecture  
was developed. Based on these needs and other requirements from the planning layer, we would like  
to propose the following message types for use in the perception pipeline.

[https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit?usp=sharing](https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit?usp=sharing)

---

<div class="post-metadata">

**Author:** ![sgermanserrano](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/sgermanserrano/32/3055_2.png) [@sgermanserrano](https://discourse.openrobotics.org/u/sgermanserrano)\
**Post date:** [April 2, 2019, 6:51am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/2 "2019-04-02T06:51:01Z")

</div>

@yukkysaito have you seen [these](http://wiki.ros.org/derived_object_msgs) msgs by Autonomous Stuff? It looks like they cover what you are proposing

---

<div class="post-metadata">

**Author:** ![JWhitleyWork](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jwhitleywork/32/5614_2.png) [@JWhitleyWork](https://discourse.openrobotics.org/u/JWhitleyWork)\
**Post date:** [April 2, 2019, 7:25pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/3 "2019-04-02T19:25:39Z")

</div>

We tried to be as broad as we could with the _Object_ and _ObjectWithCovariance_ types in `derived_object_msgs` to cover as many possible sensor-specific inputs as we could with generic structures (like `geometry_msgs/Pose` and `sensor_msgs/SolidPrimitive`) but we are always open to feedback if these don’t cover a specific need.

---

<div class="post-metadata">

**Author:** ![Dejan\_Pangercic](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/dejan_pangercic/32/293_2.png) [@Dejan\_Pangercic](https://discourse.openrobotics.org/u/Dejan_Pangercic)\
**Post date:** [April 3, 2019, 1:47am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/4 "2019-04-03T01:47:41Z")

</div>

@yukkysaito I second @sgermanserrano 's opinion to try to analyse work that AS has done, that is messages from [derived\_object\_msgs - ROS Wiki](http://wiki.ros.org/derived_object_msgs), and rather improve those than to come up with a completely new definition of messages.

To comment on [[Discussion] Detection Architecture · Issue #288 · autowarefoundation/autoware\_ai · GitHub](https://github.com/autowarefoundation/autoware/issues/1501), in this issue we tried to come up with the better decomposition of nodes that would a) allow code reuse and b) fusion of sensor data on multiple levels (raw data, features, objects). This is the result of the last discussion that I, @kfunaoka and @Kosuke_MURAKAMI arrived to [[Discussion] Naming convention of topic name and msg type for v2.0 · Issue #237 · autowarefoundation/autoware\_ai · GitHub](https://github.com/autowarefoundation/autoware/issues/1409#issuecomment-429686913).

What would be our next step is what you did, which is the data modeling step.

From a quick glance there is nothing in your proposal that is not already included in [derived\_object\_msgs - ROS Wiki](http://wiki.ros.org/derived_object_msgs) so I suggest to include messages from AS.

---

<div class="post-metadata">

**Author:** ![gbiggs](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gbiggs/32/30361_2.png) [@gbiggs](https://discourse.openrobotics.org/u/gbiggs)\
**Post date:** [April 3, 2019, 3:39am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/5 "2019-04-03T03:39:11Z")

</div>

I third using the AutonomousStuff messages. They have what we need, and while I can see a couple of things I’d consider making more exact I don’t see any problems, not even small ones, in using them.

---

<div class="post-metadata">

**Author:** ![yukkysaito](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/yukkysaito/32/3272_2.png) [@yukkysaito](https://discourse.openrobotics.org/u/yukkysaito)\
**Post date:** [April 4, 2019, 11:27am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/6 "2019-04-04T11:27:03Z")

</div>

I update the slides.  
[https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit#slide=id.g55ed101741\_1\_5](https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit#slide=id.g55ed101741_1_5)

# Background

[Current message type](https://github.com/autowarefoundation/autoware/blob/master/ros/src/msgs/autoware_msgs/msg/DetectedObject.msg) is made to be able to cover various information, and a lot of information can be defined. This includes not only the requirements for planning but also the data specific to algorithms and sensors.  
The current definition has some issues.

- Issue 1: A large number of algorithm-specific data makes it difficult to define interface and modularize perception. The interface I’m talking about here is what information is filled in with detection and tracking and which information should be filled out
- Issue 2: Unstructured and redundant message type
- Issue 3: Missing information required for planning

# Descriptions

This time, we defined the message type from the information required for dynamic object (Something that can move such as a pedestrian, car, truck, etc.).  
The details are written [here](https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit#slide=id.g55ed101741_1_5).  
As for paths, it is still undefined and will be added as we discuss in the future.

 ![Perception%20architecture%20(Not%20including%20localization)](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/3/31cac45d2dfcd8fee598791c5f6973063fee509f.png)  
 ![Perception%20architecture%20(Not%20including%20localization)%20(1)](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/7/751f8ab8fda89ef701bf521cecc009dc61c87a3c.png)

# About derived\_object\_msgs

derived\_object\_msgs is made with the same idea as the current message type as @JWhitleyWork said.

 ![Perception%20architecture%20(Not%20including%20localization)%20(2)](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/1/18b206b712687a5c41cb28d7d1ad3e7f7b5fd803.png)

---

<div class="post-metadata">

**Author:** ![gbiggs](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gbiggs/32/30361_2.png) [@gbiggs](https://discourse.openrobotics.org/u/gbiggs)\
**Post date:** [April 4, 2019, 10:39pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/7 "2019-04-04T22:39:18Z")

</div>

@yukkysaito Can you give a detailed comparison of what each of the three message types can store?

---

<div class="post-metadata">

**Author:** ![Kosuke\_MURAKAMI](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/kosuke_murakami/32/3279_2.png) [@Kosuke\_MURAKAMI](https://discourse.openrobotics.org/u/Kosuke_MURAKAMI)\
**Post date:** [April 8, 2019, 7:08am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/8 "2019-04-08T07:08:25Z")

</div>

@yukkysaito @JWhitleyWork

Please feel free to add some comments.

 ![image](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/1/12132b8a885cd9d4a771abf925c10df5ca26d2ad.png)

\*1:[shape\_msgs/SolidPrimitive Documentation](http://docs.ros.org/api/shape_msgs/html/msg/SolidPrimitive.html)

\*2:[uuid\_msgs/UniqueID Documentation](http://docs.ros.org/lunar/api/uuid_msgs/html/msg/UniqueID.html)

\*3:[https://github.com/astuff/astuff\_sensor\_msgs/blob/master/derived\_object\_msgs/msg/Object.msg#L12  
 ](https://github.com/astuff/astuff_sensor_msgs/blob/master/derived_object_msgs/msg/Object.msg#L12)\*4:[astuff\_sensor\_msgs/derived\_object\_msgs/msg/Object.msg at master · astuff/astuff\_sensor\_msgs · GitHub](https://github.com/astuff/astuff_sensor_msgs/blob/master/derived_object_msgs/msg/Object.msg#L58)

---

<div class="post-metadata">

**Author:** ![JWhitleyWork](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jwhitleywork/32/5614_2.png) [@JWhitleyWork](https://discourse.openrobotics.org/u/JWhitleyWork)\
**Post date:** [April 8, 2019, 12:01pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/9 "2019-04-08T12:01:25Z")

</div>

@Kosuke_MURAKAMI - Please see [derived\_object\_msgs/ObjectWithCovariance Documentation](http://docs.ros.org/melodic/api/derived_object_msgs/html/msg/ObjectWithCovariance.html) for a version which contains the PoseWithCovariance, TwistWithCovariance, and AccelWithCovariance. The uuid is definitely a useful addition that we had not considered and would be happy to add. This was the intent of the “id” field but it is somewhat limited as an int.

---

<div class="post-metadata">

**Author:** ![yukkysaito](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/yukkysaito/32/3272_2.png) [@yukkysaito](https://discourse.openrobotics.org/u/yukkysaito)\
**Post date:** [April 9, 2019, 12:49am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/10 "2019-04-09T00:49:52Z")

</div>

> [@JWhitleyWork](#):
>
> for a version which contains the PoseWithCovariance, TwistWithCovariance, and AccelWithCovariance. The uuid is definitely a useful addition that we had not considered and would be happy to add. This was the intent of the “id” field but it is somewhat limited as an int.

thank you @Kosuke_MURAKAMI.  
As a side note, I think that the label specific shape of AS msg is No. It can not switch between polygon and SolidPrimitive. It can switch only in SolidPrimitive (Cone, sphere, bounding box, cylinder). This is also important from the point of view of multi object tracking and path planning.

@JWhitleyWork  
Who uses detection label and classification age for what?  
If we add items widely, when we modularize detection, tracking, and prediction, these modules will have to fill in these items. Some algorithms may not be filled. I think that it is better to define only what is really necessary in msg.

---

<div class="post-metadata">

**Author:** ![JWhitleyWork](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jwhitleywork/32/5614_2.png) [@JWhitleyWork](https://discourse.openrobotics.org/u/JWhitleyWork)\
**Post date:** [April 9, 2019, 6:35pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/11 "2019-04-09T18:35:31Z")

</div>

@yukkysaito - I think the field you’re referencing is “detection\_level” which is meant to indicate whether this object has been “detected” or “tracked.” From the notes in [http://docs.ros.org/api/derived\_object\_msgs/html/msg/ObjectWithCovariance.html](http://docs.ros.org/api/derived_object_msgs/html/msg/ObjectWithCovariance.html), here are the definitions:

```
A Detected object is one which has been seen in at least one scan/frame of a sensor.
A Tracked object is one which has been correlated over multiple scans/frames of a sensor.
An object which is detected can only be assumed to have valid pose and shape properties.
An object which is tracked should also be assumed to have valid twist and accel properties.
The validity of the individual components of each object property are defined by the property's covariance matrix.

```

`classification_age` indicates the number of “scans” or “detections” made of the object where the classification type is the same. When a sensor classifies an object, it usually tells you how many “scans” of that object have been sent since the object was classified as that type. This helps determine the certainty of the classification.

---

<div class="post-metadata">

**Author:** ![amc-nu](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/amc-nu/32/3014_2.png) [@amc-nu](https://discourse.openrobotics.org/u/amc-nu)\
**Post date:** [April 10, 2019, 8:50am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/12 "2019-04-10T08:50:16Z")

</div>

According to your experience in robotics and autonomous vehicles. I would like to hear your opinions @JWhitleyWork @Dejan_Pangercic and @sgermanserrano about this messsage definition not including sensor data (i.e. ImageROI, PointCloud)?  
Do you think it is necessary? Or do you think it’s better to keep it like this to add an abstraction layer?  
Thanks

---

<div class="post-metadata">

**Author:** ![JWhitleyWork](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jwhitleywork/32/5614_2.png) [@JWhitleyWork](https://discourse.openrobotics.org/u/JWhitleyWork)\
**Post date:** [April 10, 2019, 8:29pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/13 "2019-04-10T20:29:19Z")

</div>

@amc-nu I think this is based on your intent for the message. If you intend to follow a “domain-specific-controller” approach, then you are trusting that the individual sensor processing nodes know how to correctly filter the raw data and produce abstracted objects for the most part. The uncertainty is then encoded into the covariance matrix and the classification quality data.

However, if you intend to do either data fusion before object segmentation or fusion and segmentation in the same node, you would need the raw data in the message as well. My understanding is that Autoware is shooting for the first approach so I would say we don’t need the raw data.

---

<div class="post-metadata">

**Author:** ![sgermanserrano](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/sgermanserrano/32/3055_2.png) [@sgermanserrano](https://discourse.openrobotics.org/u/sgermanserrano)\
**Post date:** [April 15, 2019, 12:07pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/14 "2019-04-15T12:07:39Z")

</div>

Agreed with @JWhitleyWork comment, I would expect the raw data to be processed on a separate step so that the filtered sensor data is in a usable stage. I think it also makes sense in a setup where you might have an edge device on the sensor that is pre-processing the raw data for consumption by higher level nodes.

---

<div class="post-metadata">

**Author:** ![gbiggs](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/gbiggs/32/30361_2.png) [@gbiggs](https://discourse.openrobotics.org/u/gbiggs)\
**Post date:** [April 24, 2019, 12:25am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/15 "2019-04-24T00:25:26Z")

</div>

I think this work is being blocked by a lack of a shared understanding of what it is we want to achieve. What objects do we want to recognise, where do we want to recognise them, what sorts of data do we want to use, what data rates, should data be synchronised or can information be added on to a detection after the fact, do we or do we not use consecutive detections to strengthen an object’s presence, how interchangeable/optional do we want different algorithms and detection types to be, and so on. There are a huge number of unanswered questions that need to be defined and then answered before we can even begin to think about the messages used.

In other words, we need to define our requirements before we try to solve them. Otherwise we are solving an unknown or undefined problem.

We also need to keep in mind that we are designing Autoware for all Autoware users, not just for Tier IV’s favourite sensor set, or AutonomousStuff’s specific demonstration. I’m not saying that that is what is happening, but it is easy to forget.

Additionally, I think it would be useful to draw up a list of:

- The different types of sensors we expect to be used. Not just ones we use now, but also ones a potential Autoware user might use.
- The different types of data we might process. Obviously this closely relates to the sensors used, but don’t forget using post-processed data as an input to an algorithm, e.g. merged dense point clouds versus individual sparse point clouds, or point clouds with or without RGB data added from a camera.
- The object locating, object identifying, object tracking, object predicting, etc. algorithm types that we might use.
- Possible orderings of algorithms.

---

<div class="post-metadata">

**Author:** ![Dejan\_Pangercic](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/dejan_pangercic/32/293_2.png) [@Dejan\_Pangercic](https://discourse.openrobotics.org/u/Dejan_Pangercic)\
**Post date:** [April 28, 2019, 6:27am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/16 "2019-04-28T06:27:03Z")

</div>

> [@amc-nu](#):
>
> According to your experience in robotics and autonomous vehicles. I would like to hear your opinions @JWhitleyAStuff @Dejan_Pangercic and @sgermanserrano about this messsage definition not including sensor data (i.e. ImageROI, PointCloud)?

@amc-nu in my experience you need to decide between performance and synchronization. That is if you do not have many nodes and you have fast middleware, then you can use message types that also include raw data. If you have the opposite case then you should go with small messages.

If you split your message types too much you will have to deal with the time synchronization once the messages received by the end node. That is both hard to do and computationally expensive.

In any case I believe that we should finish the computational graph architecture first (how many nodes and composition) and then define the messages and not the other way around. I assume that AS has a solid computational graph architecture that let them define such messages.

---

<div class="post-metadata">

**Author:** ![kfunaoka](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/kfunaoka/32/2870_2.png) [@kfunaoka](https://discourse.openrobotics.org/u/kfunaoka)\
**Post date:** [May 9, 2019, 9:54am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/17 "2019-05-09T09:54:41Z")

</div>

I listed the differences between [derived\_object\_msgs/ObjectWithCovariance](http://docs.ros.org/melodic/api/derived_object_msgs/html/msg/ObjectWithCovariance.html) and [DynamicObject](https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit#slide=id.g55622ade15_0_192), and heard why [DynamicObject](https://docs.google.com/presentation/d/1jvnyUQFUNXiFfRrUYWbO9OBPtl4K4gc78iR3eiUreLA/edit#slide=id.g55622ade15_0_192) is constructed in such a way. I’d like to clarify the differences and reasons, and like to make more common view in the whole community.

@JWhitleyWork Would you comment about the different information?

# Same information

- geometry\_msgs/PoseWithCovariance pose
- geometry\_msgs/TwistWithCovariance twist
- geometry\_msgs/AccelWithCovariance accel
- geometry\_msgs/Polygon polygon and geometry\_msgs/Polygon Shape::footprint

# New common view

- uuid\_msgs/UniqueID id is [useful and happy to add](http://discourse.ros.org/t/new-perception-architecture-message-types/8564/9) by @JWhitleyWork

# Different information

 ![27](https://us1.discourse-cdn.com/flex022/uploads/ros/original/2X/9/95ab42d20622f1b18c8f6371afdfc3cd88af67f9.png)

# Note

PoseWithCovariance past\_paths is removed from DynamicObject since planning is not interested in past information. Though prediction will require past information, it can be resolved inside prediction.

---

<div class="post-metadata">

**Author:** ![yukkysaito](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/yukkysaito/32/3272_2.png) [@yukkysaito](https://discourse.openrobotics.org/u/yukkysaito)\
**Post date:** [May 15, 2019, 7:27am UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/18 "2019-05-15T07:27:23Z")

</div>

Thank you for the summary. Do you have any opinions?

---

<div class="post-metadata">

**Author:** ![kfunaoka](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/kfunaoka/32/2870_2.png) [@kfunaoka](https://discourse.openrobotics.org/u/kfunaoka)\
**Post date:** [May 15, 2019, 11:48pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/19 "2019-05-15T23:48:49Z")

</div>

I think the reasons of DynamicObject are reasonable since I created the table based on the hearing 😉 My opinion is already reflected into the table, e.g. object\_classified and past\_paths are not necessary.

@JWhitleyWork Sorry to bother you again. Would you comment about the different information? I’m not familiar with the background of ObjectWithCovariance.

---

<div class="post-metadata">

**Author:** ![JWhitleyWork](https://sea2.discourse-cdn.com/flex022/user_avatar/discourse.openrobotics.org/jwhitleywork/32/5614_2.png) [@JWhitleyWork](https://discourse.openrobotics.org/u/JWhitleyWork)\
**Post date:** [May 22, 2019, 7:42pm UTC](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564/20 "2019-05-22T19:42:55Z")

</div>

@kfunaoka I’m sorry it has taken so long to get back to you. Here are responses addressing your issues:

The object does have a header field. However, it does not need to be populated and [ObjectWithCovarianceArray](https://github.com/astuff/astuff_sensor_msgs/blob/master/derived_object_msgs/msg/ObjectWithCovarianceArray.msg) also has a header.

The types listed for “classification” are not exhaustive nor definitive. As far as I know, the message type has not been extensively used so it is open to modification. We have only done tests with it internally and have not released any packages that use it. I agree with your assessments for the “UNKNOWN\_” types. They were types provided by a sensor vendor so we included them.

Not a problem to change the classification\_certainty to a float (0-1).

Many algorithms use a convex hull bounding area to define an object. This is why “shape” was included. There is also `geometry_msgs/Polygon polygon` in the message for defining a non-normal polygons.

Regarding the rest of the comments: I think the overall concept is that our message structure for these is flexible. We can add or modify just about anything in the message, though I would prefer not to remove much (if any) of the fields.

[Next page](https://discourse.openrobotics.org/t/new-perception-architecture-message-types/8564.md?page=2)