Has anyone tried conformal prediction for sensor gating in a nav stack?

ROS ROS General
1

Been thinking about this for a while.

The chi-squared gate in most localization stacks assumes Gaussian noise. When when we look outdoors… it rarely is. Like doing multipath near buildings, or under tree canopy, and even around field equipment.

I came across this conformal prediction (Angelopoulos & Bates 2022). The idea is: instead of assuming a distribution, you basically test each new measurement against your own empirical data. It states coverage guarantees hold regardless of noise shape.

Has anyone tried something like this in a nav stack? And honestly… is GPS covariance mismatch painful enough in real deployments that it’s worth a proper fix, or does tuning R get you 90% of the way there?

2 Likes
2

This is a really interesting thread.

I agree that the painful part is not only “is the noise Gaussian?”, but also what score we use to decide whether a measurement or state transition is atypical.

In many EKF-based stacks, the natural nonconformity score for conformal prediction would be something like innovation Mahalanobis distance. That is convenient, but it also inherits the estimator’s covariance assumptions. If R or P is already mismatched because of multipath, canopy effects, VIO drift, delayed measurements, or linearization error, then the conformal layer may be calibrating on a biased score.

So my intuition is that conformal prediction could be very useful, but the nonconformity score should probably not be only the classical chi-squared / Mahalanobis gate.

In a related PX4/ROS project, I have been experimenting with this idea at the Offboard control boundary. I open-sourced a passive observer called AFIO — currently reframing the name as Autonomy Flight Integrity Observer — that watches:

/fmu/in/trajectory_setpoint
/fmu/in/offboard_control_mode
/fmu/out/vehicle_odometry

It computes a deterministic residual bundle:

setpointAgeMs
setpointJitterMs
staleStreams
positionTrackingResidual
velocityTrackingResidual
flightResidual

The idea is not to replace the EKF, and it is not an AI detector. It is a NARH-inspired boundary consistency score: does the incoming intent stream remain fresh, temporally consistent, and physically reflected in the vehicle response?

In controlled PX4/Gazebo SITL latency-injection tests, the residual stayed quiet for 0–80 ms injected delay, produced a consistent SETPOINT_JITTER warning around 150 ms, and reached STALE_STREAM / RESYNCING at 300 ms. The useful part was that the score captured timing / execution degradation even when simple spatial tracking error still looked fairly benign.

I think a conformal layer could sit on top of this kind of residual score:

calibration flight / nominal bag
    → collect residual scores
    → choose conformal quantile
    → online flag when new residual exceeds calibrated threshold

For GPS / localization specifically, a similar approach might combine innovation residuals with additional nonconformity terms such as measurement age, covariance consistency, inter-sensor disagreement, temporal burstiness, and odometry/IMU continuity checks.

One caveat: CP’s coverage guarantee still depends on the calibration and test data being sufficiently exchangeable. For outdoor robotics, that probably means rolling or adaptive conformal calibration rather than one static threshold for all environments.

I’d be very interested to hear whether anyone here has tried alternative nonconformity scores beyond EKF innovation Mahalanobis distance — especially for GPS multipath, canopy, or delayed VIO measurements.

https://discourse.openrobotics.org/t/release-ai-flight-integrity-observer-measuring-px4-offboard-degradation-under-controlled-ai-inference-lag/55530?u=zc_liu
AFIO repo, if useful for reference: https://github.com/ZC502/ai_flight_integrity_observer.git

1 Like
3

Really interesting work on AFIO… the timing/execution degradation angle is complementary to what I’m chasing on the measurement quality side as well.

One thing that might be useful: FusionCore already computes the normalized innovation ν’S⁻¹ν as the nonconformity score for chi-squared gating (it’s the same score, just evaluated against a χ² quantile instead of an empirical distribution). So the conformal swap is essentially: keep the score, replace the threshold with the calibration window quantile. Issue #50 is tracking this, and Whatsonyourmind’s point about exchangeability under drift is the real open question… the innovations are white under optimal filtering but correlated exactly when you need the gate most (multipath, canopy dropout).

Your rolling conformal calibration idea maps directly to that… adaptive CI (Gibbs & Candès) or exponentially weighted calibration scores both address the drift problem. Have you tried either in the AFIO context, or is the timing residual well-behaved enough that a static calibration holds?

1 Like
4

Thanks — that makes sense. This is a very clean way to frame the conformal swap.

On the AFIO/OBIO side, I have not yet implemented adaptive conformal calibration. The current implementation still uses deterministic thresholds over timing and execution residuals. The Offboard-boundary residual is somewhat different from GPS / localization residuals. It is less about environmental measurement noise and more about execution-path health:

setpoint age
setpoint jitter
stale streams
position / velocity response residual
Offboard mode semantics

In the controlled PX4/Gazebo latency-injection tests, a static threshold worked reasonably well: 0–80 ms stayed quiet, around 150 ms produced consistent SETPOINT_JITTER warnings, and around 300 ms reached STALE_STREAM / RESYNCING. But I would not claim that static calibration is enough in general. On real hardware, different setpoint rates, vehicle speeds, executor configurations, DDS behavior, and companion-compute workloads could shift the residual distribution. A rolling or exponentially weighted conformal layer on top of the OBIO residual bundle would probably be the right next step.

Also, based on feedback from the PX4 thread, I am reframing the project from AFIO to OBIO — Offboard Boundary Integrity Observer. The goal is not to be an “AI detector,” but to expose boundary pressure for autonomy managers and load-shedding policies.

I’m currently preparing a small reproducible demo around this point:

CPU% looks fine,
but lock contention / timer stalls starve the PX4 setpoint path,
and OBIO catches SETPOINT_JITTER / STALE_STREAM.

Then a second scenario uses the OBIO diagnostic stream as a trigger to pause a fake SLAM/loop-closure workload and recover the boundary.

Once that demo is up, I’d be very interested in your thoughts on whether the OBIO residual bundle would make sense as a nonconformity score for rolling conformal calibration.

1 Like
5

The OBIO residual bundle would work as a nonconformity score, but I’d treat the timing residuals (age, jitter, stale streams) separately from the execution residuals (position/velocity tracking).

I believe that the timing ones are relatively mission-independent… the failure thresholds you found (150ms jitter, 300ms stale) look stable enough that static calibration probably holds across missions.

However, the execution residuals are more dynamic-dependent and would benefit from the rolling/decayed calibration. So you might end up with a two-layer structure: static conformal on timing (fast, environment-agnostic), adaptive conformal on execution (slower, mission-specific). I’d be curious to see whether the CPU/timer-stall demo shows the two layers activating at different lag values, or whether jitter always leads execution error.

1 Like
6

That two-layer framing makes a lot of sense.​
I would phrase it as:​
Layer 1: timing contract residuals
setpointAgeMs
setpointJitterMs
staleStreams
Layer 2: execution residuals
positionTrackingResidual
velocityTrackingResidual
flightResidual
The timing layer seems much less mission-dependent because it reflects the middleware-to-flight-controller contract: is the Offboard intent stream arriving at a rate and freshness that PX4 can reasonably consume?​ It is still parameterized by expected setpoint rate, vehicle class, control mode, executor behavior, DDS configuration, and failsafe settings — so I would not call one global number universal.
The execution layer is different. A 30 cm tracking error can be severe during precision hover and completely normal during aggressive flight. That is where rolling / decayed calibration makes much more sense.​
For the immediate OBIO demo, I am keeping the focus on the first layer: timer stalls, lock contention, and fake SLAM loop-closure stalls that starve the PX4 setpoint path while CPU% may still look harmless.​
And one of the most interesting things to measure in the demo is whether timing residuals consistently lead execution residuals, or whether there are cases where execution error appears first.

7

One clarification on my side: I do not want OBIO to evolve primarily into a conformal-prediction monitor. CP is a very good calibration layer, especially for mission-dependent execution residuals, but the long-term direction I am interested in is more intrinsic: designing better spatio-temporal residual representations in the first place.

At its core, OBIO is designed as a lightweight, non-associative residual hypothesis (NARH) module. The research direction behind it is to treat timing, intent, feedback, and execution phase as coupled parts of one unified boundary-consistency problem, rather than as separate scalar errors that are only calibrated after the fact.

Longer term, I am interested in pushing the NARH direction further — possibly toward octonion-inspired spatio-temporal encoding, where timing phase and kinematic state are not treated as independent diagnostic channels. So I see the relationship as:
• NARH / OBIO: Defines richer, physically coupled boundary residuals.
• Conformal Calibration: An optional layer on top for turning these mission-dependent residuals into empirical thresholds.

The immediate demo will stay focused on the deterministic timing layer. After that, the adaptive calibration question for execution residuals becomes much more interesting.