Hi @Pranav_Saroha , @Robin_Tomar
Sorry for not writing sooner, but I’ve been quite busy testing and understanding different options for the challenge
Thanks for sharing your setup. Here are my findings from the challenge in case they’re useful for you and others. I finished around 127 points (position 55), so I didn’t pass the cutoff, but I learned a lot along the way.
Started with pi0.5, switched to ACT
I initially used pi0.5. The problem was the cloud evaluation timings: model loading took ~2 seconds, there was no warmup (so the first inference was very slow), and subsequent inferences took ~200ms. With the strict configuration/loading timeouts in the cloud eval, I couldn’t make it work reliably. That’s why I moved to ACT.
The joint state ordering bug
Early on, my ACT model was completely ignoring the images. The robot followed a fixed curved trajectory regardless of what was in front of it. It looked blind.
Setting joint_positions to all zeros made the robot start “seeing” again. Debugging this, I found the issue: the order of joints in ROS didn’t match the order LeRobot was using when normalizing the dataset. Some normalized values were going crazy, drowning out the visual signal.
ROS order I was getting:
['shoulder_pan_joint', 'shoulder_lift_joint', 'elbow_joint',
'wrist_1_joint', 'wrist_2_joint', 'wrist_3_joint', 'gripper']
I had to remap to LeRobot’s expected order:
"joint_positions.0": joint_positions[3], # Wrist 1
"joint_positions.1": joint_positions[6], # Gripper
"joint_positions.2": joint_positions[1], # Shoulder Lift
"joint_positions.3": joint_positions[0], # Shoulder Pan
"joint_positions.4": joint_positions[2], # Elbow
"joint_positions.5": joint_positions[4], # Wrist 2
"joint_positions.6": joint_positions[5], # Wrist 3
This was a critical fix. Worth checking normalization carefully.
The visibility issue with my early data
I spent two weeks recording trajectories where the target port wasn’t even visible from the first frame. Then I re-read the challenge description: “the target port will always be within the view of the robot’s cameras”. I had to restart data collection. Lesson learned: read the spec carefully before automating data collection.
Resolution matters
I started recording at 288x256 and only got partial insertion on trial 1. I doubled the resolution and could be able to insert completely. To answer your question, yes, resolution matters for me.
Identifying the trial configurations
Since the trial details weren’t given upfront, I used the Task object inside insert_cable to identify them experimentally. After a few submissions and inspecting task.port_type, task.port_name, and task.target_module_name, I figured out:
- Trial 1:
nic_card_mount_2 + sfp_port_0
- Trial 2:
nic_card_mount_4 + sfp_port_1
- Trial 3:
sc_port_1
I then trained one specialized model per trial:
if task.port_type == "sc" and task.target_module_name == "sc_port_1":
model = self.model_sc
elif task.port_type == "sfp" and task.port_name == "sfp_port_0" \
and task.target_module_name == "nic_card_mount_2":
model = self.model_sfp_port0
elif task.port_type == "sfp" and task.port_name == "sfp_port_1" \
and task.target_module_name == "nic_card_mount_4":
model = self.model_sfp_port1
What worked for Trial 1
After collecting ~1000 episodes around the area I was generating data, I trained ACT for 50k steps with chunk_size=100 and n_obs_steps=1. I wasn’t expecting much, but I got 75 (full insertion) consistently on Trial 1. I have to say that this model sometimes achieved full insertions and other times it came close to the hole, so in the end this approach ended up penalizing me in the score.
Identifying task_board pose experimentally
For Trials 2 and 3 I needed to find the task_board’s actual position and orientation in the eval, which wasn’t given. Because I was running out of time to gather general data and I thought that focusing on one area would be better. I used an interesting trick:
If you train an ACT model on one configuration and then change the task_board to a different position, the model still follows roughly the same average trajectory. It gives more weight to joint states than to images. So the robot ends up at roughly the same place regardless of where the port actually is.
The cloud evaluation returns the final plug-port distance:
[aic_engine-6] [INFO] [1778918641.437966126] [aic_engine]: tier_3:
[aic_engine-6] [INFO] [1778918641.437976856] [aic_engine]: score: 0
[aic_engine-6] [INFO] [1778918641.437987216] [aic_engine]: message:
"No insertion detected. Final plug port distance: 0.41m."
By trying different task_board configurations locally and matching the resulting distance to what the cloud reported, I could narrow down where the eval was placing the board.
I then recorded targeted data around those configurations and trained new models.
The fundamental limitation: causal confusion
This is the most important takeaway. My models ended up being specialists, not generalists. They worked well when the board was in the configuration I trained them for, but as soon as you moved the board even slightly, the model would weight joint state more than visual input and execute a memorized trajectory.
This is causal confusion (de Haan et al., 2019): the model takes a shortcut by predicting actions from previous actions/state rather than truly using visual features. With chunk_size=100 and n_obs_steps=1, ACT is especially susceptible: it predicts 100 actions from a single observation and doesn’t re-observe during execution.
You can see the consequence in submissions: Trial 1 would give 75 one day and 25 the next.
What I would do differently
To answer your specific questions:
- More episodes: yes, more data helped but only up to a point. The bigger issue I think is variability. 500 episodes with wide pose variability could beat 2000 with narrow variability. The model needs poses diverse enough that it can’t memorize trajectories.
- Teleop vs CheatCode: I only used CheatCode. Teleop might generate more naturally diverse trajectories, but I can’t speak to that.
- Resolution: yes, going from 288x256 to higher resolution made a difference for me.
- Architecture changes I’d try next:
- Lower chunk_size (25-50) so the model re-observes more often.
- Higher n_obs_steps (3-5) to force temporal visual context.
- Diffusion Policy instead of ACT (generalizes better with less data, less prone to causal confusion).
- SmolVLA (compact VLA, fast loading unlike pi0.5).
- Heavy image augmentation during training to prevent the model from relying on pixel-level shortcuts.
I didn’t have time to try Diffusion Policy or SmolVLA, which I think were the biggest missed opportunities on my side.
Hope this helps. Good luck!
Here’s a video showing how my three models behave in an environment similar to cloud simulation.
