My submission is in “starting” status from at least a few hours. This can be caused from cloud provider’s side as mentioned in the other thread.
Just wanted to make sure that it does not go unnoticed and all the submissions can be completed on time.
Same here. I wonder if we submit near the deadline but it finishes inference after the deadline whether our submission will be accepted!
Hi @rajparmar, thanks for flagging. This was indeed an Azure issue, which caused the submission to hang. We’ve got it running now.
@johntiger1, all code submissions that have been properly submitted before the end of the competition are eligible and will be executed and added to the leaderboard, even if they have not completed execution by the submission deadline.
That said, we are adding some more resources right now so wait times will decrease in these final days!
I have been waiting for 24 hours now for my submission to start…
Thanks @emily and team for taking this on priority.
On the similar note, my submission again failed with incomplete logs without any error trace. Smoke test was successful on the same code.
Is this also caused due to glitch from Azure and will resubmitting my code should work fine?
Hi all, we just spun up some more resources so there are now 4 nodes (instead of 1) running submissions. This should significantly decrease wait times. @rajparmar go ahead and resubmit and let me know if you’re still having problems.
@emily still the same issue. This is really getting frustrating.
Please confirm if it is memory issue at least so that I can work on optimizing in that direction without any wasted efforts.
Yes @rajparmar it looks like it’s running out of memory. As you work to implement some fixes, I’d suggest adding more logs to the “Generating predictions using Pytorch models” code block to get a better view. Good luck!