Short proposal, and I would like to be argued with rather than agreed with.
When someone reports that a learned policy failed under some condition- a visual change, a reworded instruction, a sensor dropout- the number is almost always single-arm: run the condition, count the failures, publish the rate.
I think that number is not interpretable on its own, and I have measured why. Replaying identical seeds with and without the perturbation across 19 perturbation families and 8 policy adapters in simulation, the unperturbed arm fails at a non-zero rate that depends on the family. Subtracting it removes the measured effect for most families at 50 trials. So a lot of published resilience numbers are partly measuring the task setup.
The convention I would like to see, and it costs one extra column:
- the failure rate under the condition
- the failure rate with the condition removed, same seeds, same tasks
- the trial count behind each
- a two-sided interval on each
That is it. No method, no threshold, no taxonomy, and it endorses no tool.
Where I am coming from and what it is worth: I maintain an Apache-2.0 harness that does this ( GitHub - provael/provael: Provael — red-team open Vision-Language-Action (VLA) robot policies in simulation and report an Attack Success Rate (ASR). Prove it. Prevail. · GitHub ), so I have an obvious interest. The convention also costs me. It means publishing that most of our families produce no measurable effect once the floor comes out, and those zeros are on our results page.
What I am genuinely unsure about, and where ROS people will know better than I do: on real hardware you cannot pin a seed, so the matched arm needs its own trials and the run doubles. Is that cost acceptable in practice, or does it need a cheaper approximation to be adopted at all? If the honest answer is that nobody will double their hardware runs for this, then the convention needs to be written differently.
One more thing: everything I have measured is in a simulator. No hardware results at all. So I am proposing a reporting convention, not presenting a finding about real robots.