Hi, quick question to the community. We’d really appreciate just a short answer on this one 
We aren’t getting any outputs from aic_model, just aic_engine, in our StdOut output from the submission portal. It still goes through all trials and state transitions, but it only gets the tier 1 score. It says the task has failed (‘Task [task_<i>] failed:') but we see none of our logs (we suspect yet another timeout but can’t tell without logs).
Do you get the aic_model outputs in the log file that we can access through the submission portal? Or is it normal to just see aic_engine?
Hi @jennifer
We have quite similar issues and have already seen that you posted in the other thread that your solution was to mainly load everything by doing insert_cable instead of _init_ to save time, as the submission VM is quite slow for loading and importing models.
How could you read this out of the logs? Currently, we get a stdout file, which outputs for a very long time:
[aic_engine]: No node with name ‘aic_model’ found. Retrying…
At some point, it outputs then:
[aic_engine]: Found 1 node(s) with name ‘aic_model’. Checking if it is a lifecycle node…
But afterwards, it completely fails the insertion, while the local verification of the submission container is normally successful (at least to a certain degree).
I am worried that at some point during the config/init phase we get time-out while loading a package or model and that breaks the complete following submission.
Did you discover the same outputs in the stdout logs or how was it in your case, that you realized the main issue is loading and importing (besides reading the discourse). Was there anything special in your logs which made you realize it?
Thanks already.
Sebastian
Hi Sebastian,
[aic_engine]: No node with name ‘aic_model’ found. Retrying… At some point, it outputs then: [aic_engine]: Found 1 node(s) with name ‘aic_model’. Checking if it is a lifecycle node…
If you’re getting any output at all in the StdOut file, then your timing should be fine… OR we made wrong conclusions about timeout being the reason for the completely empty StdOut logs. Meaning: we did notice that once we moved everything to task time (insert_cable), then we did finally get logs. Though we did one other thing in the Dockerfile which could have contributed. We are now bit by bit moving things back to __init__ to see when it might break. Lots of wasted submissions… but that’s how it’s gotta be it seems.
Also to note that many “retrying” outputs do indicate that it’s taking a long time.
I think it is designed that we’re not getting any aic_model output. Perhaps fellow competitors could confirm here, or let us know if that’s not the case.
1 Like
Thanks for your response, @jennifer .
That helps a lot.
@Yadunund: Could you please confirm, that if we receive a stdout file after our submission, which contains at some point the following lines, we can assume that the aic-model was definitely loaded successfully, and it is probably a policy internal error.
[aic_engine-5] [INFO] [1778233685.862236623] [aic_engine]: Found 1 node(s) with name ‘aic_model’. Checking if it is a lifecycle node…
[aic_engine-5] [INFO] [1778233685.862303376] [aic_engine]: Service ‘/aic_model/get_state’ is available. Participant model discovered.
…
[aic_engine-5] [INFO] [1778233687.083634982] [aic_engine]: Lifecycle node ‘aic_model’ is in ‘configured’ state and meets all expectations.