Back to DrivenData | Blog

QUEUED: There are currently 12 submissions ahead of yours

😵‍💫😵‍💫😵‍💫

Are there plans to upgrade the available hardware for evaluating submissions?

5 Likes

hey @jsensio.

The reason for long queues might be that most participants use the code provided in the benchmark notebook, which does inference on CPU, instead of GPU. I just wrote about the issues:

Please make sure that your code leverages GPU for inference.

Hi, my code uses the GPU and I think most participants do as well because doing inference on the CPU with a neural network takes too much time, and even GPU inference in a non-batched way may exceed the time limit. The problem is that there are a lot of test samples, with a big resolution, so inference is slow. The solution is to enable multiple machines for processing submissions in parallel, I guess right now there is only one with a queue system. Since this competition is backed by Microsoft and the Planetary Computer I think there should not be much of a problem to put in place a scalable infrastructure, adding and removing machines as required, so the queue never exceeds 2 or 3 pending submissions…

2 Likes

You are right. Allocating more ressources to this competition is the ultimate solution. Since this problem persists for several weeks now, i don´t feel like things going to change. Are any of the competition organizers even in this forum? :sweat_smile:

Thanks for your patience. We’re tracking wait time for this competition; currently the median wait time is about 12 hours. If it gets much higher than that, we’ll consider adding another code execution node.

2 Likes

wow, are there many more participants in this competition vs the previous Flood competition? My submissions usually ran immediately in that competition.

New year, new node! We’ve increased the number of nodes available for code execution so things should speed up. We’ll continue to keep an eye on wait times and try our best to keep things moving. Thanks for your patience!

8 Likes

another great improvement would be to provide all of the extra bands in test_features. participants are likely wasting a lot of bandwidth/time all downloading the same files repeatedly. IIRC, they were provided midway through the flood competition as well.

8 Likes

One more option would be to allow 3 submissions per day even if you have a submission that is pending.

1 Like