I’m observing a weird failure for my federated smoke test (at timestep 2023-01-23 20:23:47 UTC).
My code has successfully ran till the testing stage of the 3rd scenario, and then it abruptly failed at without any error messages. I’m not too sure if it’s a memory issue, after seeing that:
- it has successfully finished the first 2 scenarios for all clients
- the most memory intensive data preprocessing stage finished succesfully
- the same model training/inference code can finish for the central evaluation runtime
Could you please give some pointers as to why the submission failed? Would it be due to time limit or memory limit? Thanks!
Also, if it is indeed a time/memory issue, would you please consider increasing the limit for the runtime? From the public leaderboard it seems that not a lot of teams have successfully made a submission to the federated runtime.