QUEUED: There are currently 12 submissions ahead of yours

[jsensio]

😵‍💫😵‍💫😵‍💫

Are there plans to upgrade the available hardware for evaluating submissions?

[Max_Schaefer]

hey @jsensio.

The reason for long queues might be that most participants use the code provided in the benchmark notebook, which does inference on CPU, instead of GPU. I just wrote about the issues:

IMPORTANT: How to speed up Inference & Queue time

Hey fellow participants. I think i found the reason for long queue waiting time. I noticed that the code provided in the benchmark notebook doesn´t leverage GPU, but instead lets your model do inference on CPU. Since probably most participants start with that notebook, i´m guessing that most didn´t adjust the code in main.py to let their model do inference on GPU. I adjusted the code and tested the difference: CPU: 1h 44 minutes GPU: 0h 23 minutes Here is a little hard coded solution to spee…

Please make sure that your code leverages GPU for inference.

[jsensio]

Hi, my code uses the GPU and I think most participants do as well because doing inference on the CPU with a neural network takes too much time, and even GPU inference in a non-batched way may exceed the time limit. The problem is that there are a lot of test samples, with a big resolution, so inference is slow. The solution is to enable multiple machines for processing submissions in parallel, I guess right now there is only one with a queue system. Since this competition is backed by Microsoft and the Planetary Computer I think there should not be much of a problem to put in place a scalable infrastructure, adding and removing machines as required, so the queue never exceeds 2 or 3 pending submissions…

[Max_Schaefer]

You are right. Allocating more ressources to this competition is the ultimate solution. Since this problem persists for several weeks now, i don´t feel like things going to change. Are any of the competition organizers even in this forum?

[rbgb]

Thanks for your patience. We’re tracking wait time for this competition; currently the median wait time is about 12 hours. If it gets much higher than that, we’ll consider adding another code execution node.

[loweew]

wow, are there many more participants in this competition vs the previous Flood competition? My submissions usually ran immediately in that competition.

[rbgb]

New year, new node! We’ve increased the number of nodes available for code execution so things should speed up. We’ll continue to keep an eye on wait times and try our best to keep things moving. Thanks for your patience!

[loweew]

another great improvement would be to provide all of the extra bands in test_features. participants are likely wasting a lot of bandwidth/time all downloading the same files repeatedly. IIRC, they were provided midway through the flood competition as well.

[ajijohn]

One more option would be to allow 3 submissions per day even if you have a submission that is pending.