Hello,
Wow, 3 subs per 7 day. I am wondering why we have such strict limit. Could we somehow have more chance to submit?
Cheers,
M
1 Like
Hey Motoki…
I am sure it is because of the resource crunch. Meanwhile, I ran out of all my submissions… that gives 7 more days to think and come up with a new idea. Is’nt that great.
I do understand the painful wait, will be good to have 1 submission per day. With you on it.
Regards
Shankar
2 Likes
Yes, I would go for more subs.
I agree. It will be better to have 1 sub per day.
Hi all, just want to make a quick point here: you can run locally with the Docker setup an unlimited number of times. The submission limit is there to prevent overfitting to the leaderboard data.
If your workflow requires frequent runs on the test data you may want to make sure you are using good train/test/eval splits locally to avoid overfitting and get a model that generalizes well to new data.
1 Like
Some less money for the prize and some more money on the servers would have been a good idea IMHO. After more than 4 hours I still have 3 persons ahead in the queue.
My train, val, test is around 75%, 10%, 15%, but it seems to generalize badly. My local score was better than the benchmark (about 10% improvement) but the final score on the leaderboard was slightly worse. Now (tomorrow, I guess) let’s see this new model (about 20% better than the benchmark).
Thanks for the heads up that waiting times were growing. We just added some compute resources which should help work off the backlog right now.
This isn’t really a compute cost issue, just a confluence of submissions at the same time where some are taking a long time to finish and are therefore lengthening average wait times.
That being said, in this competition we really encourage everyone to think hard about generalization and test locally.
1 Like
One sub per day would be great. People who joined this competition 3 or 2 weeks before the ending don’t have enough submissions to test their models.