I benchmarked my ROS 2 localization filter (FusionCore) against robot_localization on real-world data. Here's what happened

I ran FusionCore head-to-head against robot_localization (the standard ROS sensor fusion package) on the NCLT dataset from the University of Michigan… a real robot driving around a campus for 10 minutes. Mixed urban/suburban environment with tree cover, buildings, and open quads: the kind of GPS conditions where multipath is real, not a lab with clear sky view. Ground truth is RTK GPS, sub-10cm accuracy.

Equal comparison, no tricks: same raw IMU + wheel odometry + GPS fed to every filter simultaneously. No tuning advantage. This is strictly equal-config performance on identical sensor data.

The dashed line is RTK GPS ground truth. That’s where the robot actually was.

Left: robot_localization EKF. Right: FusionCore.

Accuracy over 600s (Absolute Trajectory Error (ATE) RMSE: lower is better):

  • FusionCore: 5.5 m

  • robot_localization EKF: 23.4 m: 4.2× worse

The difference comes down to one thing: robot_localization trusts every GPS fix equally and uses fixed noise values you set manually in a config file. FusionCore continuously estimates IMU bias and adapts its noise model in real time… so it knows when a measurement doesn’t fit and how much to trust it.

FusionCore tracks position, velocity, orientation, plus gyro bias and accelerometer bias as live states. RL-EKF has no bias estimation; gyro drift compounds silently into heading error.

I also ran robot_localization’s UKF mode. It diverged numerically at t=31 seconds: covariance matrix hit NaN, every output invalid for the remaining 9 minutes. FusionCore ran stably for the full 600 seconds on the same data. Fusioncore turns out is numerically stable even at high IMU rates. This is why RL-UKF hit NaN at 100Hz and FusionCore didn’t.

Dataset: NCLT (University of Michigan).

GitHub repo: https://github.com/manankharwar/fusioncore

ROS Discourse: https://discourse.ros.org/t/fusioncore-which-is-a-ros-2-jazzy-sensor-fusion-package-robot-localization-replacement

Currently testing on physical hardware. If you’d like to try it, the repo is open… raise an issue, open a PR, or just DM me. Happy to answer any questions… I respond to everything within 24 hours. Happy building!

2 Likes

Quick follow-up since a few people messaged me directly: the 0.31 m RMSE was standard autonomous GPS, no RTK. I’ve also uploaded the evaluation scripts to the repo if you want to run the same evo_ape benchmark on your own data: tools/benchmark/ in the repo. Would be curious if anyone gets comparable numbers on a different platform.

Interesting experiment! Just a few thoughts:

  • Any time you publish a head-to-head comparison, I would argue that the results have less meaning if you don’t provide the configuration files that you used to generate them. Where is your EKF config? Can you provide a direct link to the bag file you used?

robot_localization trusts every GPS fix equally and uses fixed noise values you set manually in a config file

  • This is false. Can you point me to the location where you derived this statement? robot_localization uses the reported GPS covariance, as given in the sensor_msgs/NavSatFix definition. This seems like the correct practice for determining how much to trust the message.

RL-EKF has no bias estimation; gyro drift compounds silently into heading error.

  • This is accurate, but we shouldn’t pretend that bias estimation solves this problem. It doesn’t. It just slows the growth of the error. That in itself is very useful, though, agreed. However,
    • If you are fusing just rotational velocity into your state estimate, this would be something I would only recommend when generating the odom→base_link transform. As this frame is typically used for local planning, the level of drift experienced over any control cycle is so small as to be negligible.
    • If you are fusing it into your map frame state estimate, and have no absolute heading reference, that’s an incorrect configuration, as it wouldn’t adhere to REP-105 (heading will drift over time). As I said, bias estimation doesn’t solve this problem; it just delays it.
  • The UKF in r_l does have numerical instability, definitely. I just haven’t had the cycles to look into it.
3 Likes

automatom, thanks for coming back to this thread too. Let me go through all of this properly.

  1. On reproducibility first

Fair point and you’re right to ask. The FusionCore config used for the NCLT run is committed here: github.com/manankharwar/fusioncore_datasets/config/nclt_fusioncore.yaml

The bag file is from the public NCLT dataset at robots.engin.umich.edu/nclt: specifically the 2012-01-08 sequence shown in that trajectory plot. The raw sensor data, the ground truth RTK, and the evaluation scripts are all in the datasets repo. I should have linked all of that in the original post and I didn’t. Fixing that now.

  1. On the GPS covariance claim

You’re right, same correction you made in the other thread. I read the navsat_transform source after you flagged it and found the covariance being read and rotated correctly. The claim that RL ignores GPS-reported covariance is wrong. I’ve updated the README and docs to remove it.

  1. On bias estimation and REP-105: this is the part I want to engage with properly

You said bias estimation doesn’t solve the heading drift problem, it just slows it. That’s accurate and I should have been more careful about how I framed it. Let me explain what FusionCore actually does here because I think the nuance matters.

For a robot running odom only with no GPS, you’re completely right. Bias estimation on a 6-axis IMU with no absolute heading reference means your yaw error grows more slowly but it still grows. FusionCore in that configuration is publishing an odom frame estimate and nothing about it is REP-105 compliant for map-level accuracy. The filter knows this too: the heading_validated_ flag only gets set true when there’s an actual external heading source: a dual antenna receiver, a 9-axis AHRS with magnetometer, or… here’s the tricky one… GPS position updates over distance.

When GPS is active and the robot is moving, sequential position fixes give the filter weak heading observability. If you’re at point A one second and point B the next, the vector A->B tells you something about which direction you’re facing. It’s not an absolute heading measurement and it degrades if you’re moving slowly or turning, but it does constrain heading drift in a way that pure gyro integration doesn’t. FusionCore uses this to set heading_validated_ after 5 meters of straight-line travel at low yaw rate. Is that a true absolute heading reference? No. Does it meaningfully reduce drift over a 600-second outdoor run compared to no GPS? Yes, and the trajectory shows it.

The honest answer to your REP-105 point is: FusionCore is fully compliant when you have a real heading source (dual antenna or magnetometer), and it produces a usable but not strictly compliant map-frame estimate when GPS position is your only heading signal. The code tries to be transparent about this… there’s a warning in the diagnostics when heading hasn’t been independently validated. But I should say this more clearly in the documentation rather than letting it be buried in a config flag.

On the UKF numerical instability

“The UKF in RL does have numerical instability, definitely. I just haven’t had the cycles to look into it.”

Thank you for confirming this. I was careful not to claim this was a configuration problem because I genuinely didn’t know if it was, and I didn’t want to be wrong twice. The data shows NaN at t=31 seconds on the 2012-01-08 sequence. I can reproduce it consistently. The same data, same timestamp, same result across multiple runs.

My working hypothesis is that it’s a gating issue. FusionCore runs a chi-squared test on every sensor update. When a GPS fix is a statistical outlier relative to the current state, it gets dropped before it touches the filter. RL’s navsat_transform uses the covariance to weight the update but doesn’t reject it. A bad fix with wrong covariance enters the filter with low weight, nudges the state slightly, the next fix is now a larger innovation relative to the corrupted state, it also gets accepted, and eventually the UKF’s sigma point computation or covariance update hits something that isn’t positive definite. That’s where NaN comes from.

If you ever do get the cycles to look into it, I’d genuinely be curious whether adding a Mahalanobis gate to the GPS update in navsat_transform fixes it. My guess is it would.

RL UKF diverges in less than a second on our robot, and I don’t think it is because of outliers. I left some evidence in the issue that reports UKF problems, but I no longer remember the details.

@peci1 I have a branch that might address it. Just need to find your comment. :slight_smile:

1 Like