Three bugs that changed how our sensor board talks to ROS 2

ROS ROS General
,
1

We’re working on a board that connects cameras and sensors to a robot’s compute over CAN-FD and GMSL2. Three things came up during development that changed the design more than anything we planned upfront.

The flash budget made the decision for us.

We started assuming the board would speak ROS directly. Put micro-ROS on the chip to test it, and it ate 60% of flash before we’d written any of our own code.

That settled it. The device speaks Cyphal now. ROS runs host-side behind a thin bridge, and one firmware image serves three transports instead of one.

One reboot didn’t actually reboot anything.

A link kept dying every time we swapped a board. Power cycles didn’t fix it. Full host reboots didn’t fix it either.

Turned out one chip never fully powered off. It was getting backfed through its own data lines and holding old state through every reset we tried. The fix was a single reset write, once we found where the power was actually coming from.

Also explained an older bug on the same rig we’d never tracked down. Same root cause.

Timestamps mattered more than we assumed.

We treated per-sample timestamps as optional at first. Sensor sends data, host receives it, how far off could the timing really be.

Turns out it matters a lot once you’re fusing readings from different nodes on different transports. SLAM cares when the sensor captured the reading, not when the host got it. Those two numbers aren’t close.

Now every sample is stamped at capture, on the node, with a sequence number so drops are visible.

Wrote this up mostly for our own record and I guess other people building sensor nodes run into some version of the same three things.

2

Your third one hit home. I ran into the same thing and honestly the size of it caught me off guard.

So, I’ve got a UKF fusing IMU, wheel encoders and GPS. I fed it the exact same recorded data twice, and the only thing I changed was adding 1 microsecond to every IMU timestamp. Nothing else. Literally same code, same data, and same callback order as well. With GPS available the final position only moved 0.68 m, so at a glance that looks fine. But yaw came out 109 degrees different. And when I ran the same test across a 120 second GPS outage, the final position was off by 50.6 m.

I should be careful with that number though, because it’s easy to misread. Most of it is my filter, not the timestamps. My quaternion covariance isn’t bounded, so yaw goes chaotic whenever nothing is watching it absolutely, and GPS was hiding the position side of that rather than actually fixing it. The microsecond doesn’t cause the problem, it just makes it visible. But your point still stands, and that’s basically why I’m replying: anything downstream that integrates will blow up small timestamp errors, and you won’t notice while you’ve still got something absolute to correct against.

One thing I’m curious about. What clock is that capture stamp on, and how do you keep it lined up with the host’s ROS time? Stamping at capture kills the transport latency, which is the big one. But if the node’s clock has an offset or drifts against the host, you’ve kind of traded a latency error for an offset error, and that one’s harder to catch because it looks constant instead of noisy.