Hello, my submission has Failed, but I the results and stderr file are empty. Can you please check and guide me on how to proceed? The local tests with docker compose -f docker/docker-compose.yaml up work just fine. Here are the submission details:
AI for Industry Challenge - 2026
------
Date: 2026-05-09T13:35:40.085094+00:00
Team: Ploceus2026
Submission: 969
Run ID: 9c36ff14-25ce-4945-bc83-8bc18d6be298
------
{"submitted_image_uri":"973918476471.dkr.ecr.us-east-1.amazonaws.com/aic-team/ploceus2026:v2"}
I had same issue yesterday. I validated the container locally, as required, and it passed and scored.
But the only output I got from the submission on the site is this:
AI for Industry Challenge - 2026
------
Date: 2026-05-11T06:15:32.779808+00:00
Team: MARG_777
Submission: 1087
Run ID: 1f6ff77f-918b-449c-abca-473b2b7dbaf2
------
I checked the URI of submission and I do not see any issues with it
Without any error description, the search of problem would too broad.
I would appreciate someone look into these issues.
So I tried doing what was discussed a lot on the discourse - lazy loading, i.e. moving as many global python imports as possible into individual functions so that the AWS hosted engine discovers aic_model before the 30 sec timeout. The result was that the trials got processed but the total score was only 3, which is the 1 point per trial for successful model validation, but I don’t think the insert_cable() method got called successfully, since stdout file shows no logs or stdouts from my insert_cable() implementation.
Here are the submission details:
AI for Industry Challenge - 2026
------
Date: 2026-05-10T06:47:42.100706+00:00
Team: Ploceus2026
Submission: 1015
Run ID: 63357e44-2c72-477d-82d9-41e81e84ba27
------
{"submitted_image_uri":"973918476471.dkr.ecr.us-east-1.amazonaws.com/aic-team/ploceus2026:v3"}
And here is an excerpt from stdout file, showing the output for trial 2/3:
[aic_engine-5] [INFO] [1778395405.117917691] [aic_engine]: e[1;33m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━e[0m
[aic_engine-5] [INFO] [1778395405.117924962] [aic_engine]: e[1;33m Trial 2/3: trial_2e[0m
[aic_engine-5] [INFO] [1778395405.117933032] [aic_engine]: e[1;33m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━e[0m
[aic_engine-5] [INFO] [1778395405.117940832] [aic_engine]: e[1;36m→ Starting trial 'trial_2'...e[0m
[aic_engine-5] [INFO] [1778395405.117975383] [aic_engine]: Checking participant model readiness...
[aic_engine-5] [INFO] [1778395405.118047714] [aic_engine]: Found 1 node(s) with name 'aic_model'. Checking if it is a lifecycle node...
[aic_engine-5] [INFO] [1778395405.118062334] [aic_engine]: Service '/aic_model/get_state' is available. Participant model discovered.
[aic_engine-5] [INFO] [1778395405.118070685] [aic_engine]: Transitioning model node 'aic_model' to transition 'activate'...
[aic_engine-5] [INFO] [1778395405.120184268] [aic_engine]: Successfully transition model node 'aic_model' to state 'activate'
[aic_engine-5] [INFO] [1778395405.120213009] [aic_engine]: e[1;32m ✓ Model Ready for trial 'trial_2'e[0m
[aic_engine-5] [INFO] [1778395405.120222259] [aic_engine]: Checking required endpoints...
[aic_engine-5] [INFO] [1778395405.129505471] [aic_engine]: All required endpoints are available.
[aic_engine-5] [INFO] [1778395405.129562852] [aic_engine]: e[1;32m ✓ Endpoints Ready for trial 'trial_2'e[0m
[aic_engine-5] [INFO] [1778395405.129570843] [aic_engine]: Readying simulator for trial 'trial_2'...
[aic_engine-5] [INFO] [1778395405.129577493] [aic_engine]: Spawning task board.
[aic_engine-5] [INFO] [1778395405.313045929] [aic_engine]: Successfully spawned task_board as 'task_board'
[aic_engine-5] [INFO] [1778395405.313123221] [aic_engine]: Task board spawned successfully.
[component_container-3] [INFO] [1778395405.313437717] [aic_controller]: Received service request to tare force torque sensor!
[component_container-3] [INFO] [1778395405.313494388] [aic_controller]: Successfully tared force torque sensor.
[aic_engine-5] [INFO] [1778395405.313706883] [aic_engine]: Spawning cable.
[aic_engine-5] [INFO] [1778395405.313767174] [aic_engine]: Spawning cable 'cable_0'...
[component_container-3] [INFO] [1778395405.317992002] [gz_server]: InitializeCanonicalLinks
[aic_engine-5] [INFO] [1778395406.040913424] [aic_engine]: Successfully spawned cable_0 as 'cable_0'
[aic_engine-5] [INFO] [1778395406.040999615] [aic_engine]: Cable cable_0 spawned successfully.
[component_container-3] [INFO] [1778395406.046399696] [gz_server]: InitializeCanonicalLinks
[component_container-3] [INFO] [1778395406.114362006] [gz_server]: InitializeCanonicalLinks
[aic_engine-5] [INFO] [1778395406.679176434] [aic_engine]: Waiting for robot arm to stabilize.
[aic_engine-5] [INFO] [1778395409.060471326] [aic_engine]: e[1;32m ✓ Simulator Ready for trial 'trial_2'e[0m
[aic_engine-5] [INFO] [1778395409.060518357] [aic_engine]: Checking scoring system readiness...
[aic_engine-5] [INFO] [1778395409.374951561] [aic_engine]: Started recording to '/tmp/aic/results/bag_trial_2_20260510_064329_060'.
[aic_engine-5] [INFO] [1778395409.375008312] [aic_engine]: e[1;32m ✓ Scoring Ready for trial 'trial_2'e[0m
[aic_engine-5] [INFO] [1778395409.375016552] [aic_engine]: Starting tasks for active trial...
[aic_engine-5] [INFO] [1778395409.375025462] [aic_engine]: Sending InsertCable goal for task [task_1]
[aic_engine-5] [INFO] [1778395409.379008204] [aic_engine]: TrialState: TasksExecuting
[aic_engine-5] [INFO] [1778395409.379112266] [aic_engine]: Waiting for result...
[aic_engine-5] [INFO] [1778395444.968011533] [aic_engine]: Task [task_1] succeeded.
[aic_engine-5] [INFO] [1778395444.968066134] [aic_engine]: All tasks have been processed.
[aic_engine-5] [INFO] [1778395444.968073724] [aic_engine]: e[1;36m ⟳ Tasks Executing for trial 'trial_2'...e[0m
[aic_engine-5] [INFO] [1778395444.968079914] [aic_engine]: Checking if all tasks were completed successfully...
[aic_engine-5] [INFO] [1778395444.968085744] [aic_engine]: e[1;32m ✓ All Tasks Completed for trial 'trial_2'!e[0m
[aic_engine-5] [INFO] [1778395446.347694102] [aic_engine]: Finished scoring trial, total score is: 1.000000
[aic_engine-5] [INFO] [1778395446.347748703] [aic_engine]: Resetting after trial completion...
[aic_engine-5] [INFO] [1778395446.347760093] [aic_engine]: Transitioning model node 'aic_model' to transition 'deactivate'...
[aic_engine-5] [INFO] [1778395446.353660784] [aic_engine]: Successfully transition model node 'aic_model' to state 'deactivate'
[aic_engine-5] [INFO] [1778395446.355520252] [aic_engine]: Successfully deleted entity 'task_board'
[aic_engine-5] [INFO] [1778395446.394776458] [aic_engine]: Successfully deleted entity 'cable_0'
[aic_engine-5] [INFO] [1778395446.394841499] [aic_engine]: Homing robot to initial positions...
[component_container-3] [INFO] [1778395446.395252778] [controller_manager]: Deactivating controllers: [ aic_controller ]
[component_container-3] [INFO] [1778395446.395391441] [controller_manager]: Requested controller switch from non-realtime loop
[component_container-3] [INFO] [1778395446.395510693] [controller_manager]: Deactivated controllers: [ aic_controller ]
[component_container-3] [INFO] [1778395446.395519753] [controller_manager]: Successfully switched controllers!
[aic_engine-5] [INFO] [1778395446.399819061] [aic_engine]: aic_controller deactivated successfully.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [shoulder_pan_joint(Entity=102]). Ignoring JointVelocityCmd component.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [shoulder_lift_joint(Entity=103]). Ignoring JointVelocityCmd component.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [elbow_joint(Entity=104]). Ignoring JointVelocityCmd component.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [wrist_1_joint(Entity=105]). Ignoring JointVelocityCmd component.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [wrist_2_joint(Entity=106]). Ignoring JointVelocityCmd component.
[component_container-3] (2026-05-10 06:44:07.406) [warning] [Physics.cc:2467] Found both JointVelocityReset and JointVelocityCmd components for Joint [wrist_3_joint(Entity=107]). Ignoring JointVelocityCmd component.
[component_container-3] [INFO] [1778395447.407406917] [controller_manager]: Activating controllers: [ aic_controller ]
[component_container-3] [INFO] [1778395447.407561970] [controller_manager]: Requested controller switch from non-realtime loop
[component_container-3] [INFO] [1778395447.407685383] [controller_manager]: Activated controllers: [ aic_controller ]
[component_container-3] [INFO] [1778395447.407694553] [controller_manager]: Successfully switched controllers!
[aic_engine-5] [INFO] [1778395447.410567572] [aic_engine]: aic_controller activated successfully.
[aic_engine-5] [INFO] [1778395447.410622123] [aic_engine]: Successfully reset joints to home position, robot homed successfully.
[aic_engine-5] [INFO] [1778395447.410631643] [aic_engine]: Reset after trial completed.
[aic_engine-5] [INFO] [1778395447.410647734] [aic_engine]: e[1;32m✓ Trial 'trial_2' completed successfully! Score: 1.000000e[0m
My team is also facing this exact issue. We are able to locally run and we tested with several configs. But the submission is failing and the stderr file is empty. I’m not sure how we can go about debugging this. Also, we check various combinations of URI for submission. We have been submitting for the past 6-7 trials thinking issue is probably on algorithm side. Please share if anyone has found a solution to this!