A few days ago I shared a benchmark where FusionCore beat robot_localization EKF on a single NCLT sequence. Fair enough… people called out that one sequence can easily be cherry-picked. Someone also mentioned that the particular sequence I used is known to be rough for GPS-based filters. Others asked if RL was just badly tuned, or how FusionCore could outperform it that much if both are just nonlinear Kalman filters… etc
All good questions.
So I went back and ran six sequences across different weather conditions. Same config for everything. No parameter tweaks between runs. The config is in fusioncore_datasets/config/nclt_fusioncore.yaml, committed along with the results so anyone can check.
FusionCore wins 5 of 6. RL-UKF diverged with NaN on all six.
Now, the obvious question: what happened with November 2012? That’s the one where RL wins.
That sequence has sustained GPS degradation… this isn’t just occasional noise. The NCLT authors themselves mention elevated GPS noise in that session. Both filters are seeing the exact same data, so the difference really comes down to how they handle it.
Here’s what’s going on:
FusionCore has a gating mechanism. When GPS looks bad, it rejects those measurements. That’s usually a good thing… but in this case, the degradation is continuous. So, Fusioncore rejects a few GPS fixes → the state drifts → the next GPS measurement looks even worse relative to that drifted state → it gets rejected again → and this repeats. It kind of traps itself rejecting the very data it needs to recover.
RL, on the other hand, just accepts every GPS update. No gating, no rejection. That means it gets pulled around by noisy GPS, but it also re-anchors itself as soon as the signal improves. So in this specific case, that “always accept” behavior actually helps.
After discussing this with some hardware folks here in Kingston, we decided to add something we’re calling an inertial coast mode. The idea is simple:
-
If FusionCore sees N consecutive GPS rejections, it increases the position process noise (Q)
-
That causes the covariance (P) to grow
-
As P grows, the Mahalanobis gate naturally becomes less strict
-
Eventually, incoming GPS measurements are no longer “too far” and get accepted again
-
Once GPS is accepted, Q resets back to normal
Basically, instead of getting stuck rejecting everything, the filter “loosens up” over time and lets itself recover.
On the November 2012 sequence, this drops the error from 61.4 m → 28.7 m. RL still wins, but the gap is much smaller now, and everything is documented in the repo.
If your robot drives through tunnels, underpasses, agricultural land, and/or urban canyons with brief GPS dropouts, FC’s gate is a strength… it doesn’t get corrupted by the bad fixes during the outage. If you have GPS that is consistently mediocre (cheap receiver, cheap module, always noisy but never totally wrong), RL’s accept-everything approach is probably safer at least until coast mode gets smarter?
If you’ve got ideas on improving this… especially around re-acquisition or better fallback behavior… I’m all ears. Suggestions, config tweaks, PRs… all welcome.
Reproducing a run is straightforward
git clone https://github.com/manankharwar/fusioncore.git
# Download NCLT sequence from http://robots.engin.umich.edu/nclt/
ros2 launch fusioncore_datasets nclt_benchmark.launch.py \
data_dir:=/path/to/nclt/2012-01-08 \
output_bag:=./bag
python3 tools/evaluate.py --gt ground_truth.tum \
--fusioncore fusioncore.tum --rl rl_ekf.tum \
--sequence 2012-01-08
Full pipeline in benchmarks/README.md. Results per sequence under benchmarks/nclt/*/results/BENCHMARK.md.
November 2012 is an open problem. Coast mode cuts the error by 53% but RL’s no-gate approach still wins under sustained GPS degradation. Fully closing the gap requires either a smarter re-acquisition strategy or a tunable fallback threshold. Pull requests are welcome.
If you’ve got a dataset you want me to try, just send it over (or drop a link), and I’ll run it and share the results.
FusionCore accepts nav_msgs/Odometry from any source including slam_toolbox, MOLA, ORB-SLAM3, and even VINS-Mono. Same interface as wheel odometry.
Happy Building!
