Making a degraded-network matrix a required CI gate, and what did not reproduce

I maintain Ganglion, an open-source substrate for reaching ROS 2 robots on networks you do not control. This is a writeup of making a five-profile degraded-link matrix (clean, lossy, high-latency, asymmetric, nat-relay) a required CI gate.

One correction folded in, since I claimed otherwise in the latency simulation thread ( How do you simulate latency? - #5 by sokstherobot ): netem’s seed cannot be pinned. Loss and jitter draw from the kernel RNG and there is no seed parameter to set. A gate that fails randomly does not catch regressions, it teaches everyone to re-run the job until it goes green.

What reproduces exactly turned out to be a smaller set than netem advertises: fixed netem delay, tbf rate caps, iptables -m statistic --mode nth for loss, and route blocking for the no-direct-route case. Randomized netem still runs nightly in a non-blocking job that opens an issue rather than stopping a merge.

The section on recording what you shaped came directly out of the trigger-based capture discussion in the YERP thread (YERP + rosbag2 snapshot: can we capture why a ROS perception latency spike happened?). Injected faults that are not recorded produce failures you can see and cannot get back. Every run writes a JSON artifact with the mode, seed, exact shaping commands, duration and result.

Full writeup: Testing the claim: a degraded-link matrix as a required CI gate — automaton.run

Harness is in test-harness/degraded-link/ including the determinism contract: GitHub - RobotDen/ganglion: Ganglion is the secure connectivity and auditable tool-execution substrate for ROS 2 robot fleets on hostile networks — outbound-only reachability, signed WASM tooling, default-deny policy. · GitHub

I build Ganglion, so weigh the enthusiasm accordingly. The part I think is useful to anyone shaping networks in CI is the reproduces / does-not-reproduce split, which is independent of what you are testing.

Your tool looks interesting, but I’m having a really hard time parsing your posts. I’m not sure what “the latency thread” or “the conversation here” refer to. Do you think you can provide link/quote/summary or some additional context?

Good catch, thank you. Looks like I missed the links when I posted. I have edited it to add both: the latency simulation thread where I made the netem seed claim, and the YERP thread whose trigger-based capture discussion shaped the recording section. Short version of the correction: netem’s seed cannot be pinned, its loss and jitter draw from the kernel RNG, which is why the gate is built only from mechanisms that reproduce exactly.