Hi!
I’m new on this platform. Am I right that here I shouldn’t choose my submission? What submission will be chosen: the last one or the best one on private?
Hi @BorisV ! You are correct - you don’t need to choose a submission. Your best-performing submission will be used by default.
Good luck!
Why is it done this way? A) it gives a big advantage to people who submit more, possibly making it like a lottery and the winner being the person who has the most “tickets”. B) Our best submission might be one that we didn’t save reproducible code for, and thus it doesn’t seem useful for it to be considered for the Private LB.
1 Like
@BrandenKMurray Each user or team has a limited number of submissions they can enter, so folks cannot “hack” the competition by simply having the most submissions.
While the private LB is not exactly the same as the public LB, the general trend is similar so you have some indication of which of your submissions is likely to be the best. I would recommend maintaining notes and reproducible code for your top few submissions based on the public LB.
I respect your point, but I can’t entirely agree with it. Of course, if you are in 50th place, it is doubtful (but possible) that you could reach the top3 places, but between the top5, it will be a lottery =)
And for the organizers, it is not the best idea too. If you should choose between two final submissions, your goal is to create your model as stable as possible, which is suitable for a model in real life, but for this type of competition, you should make as volatile models as possible (but with low enough mean), because if you do this, you have much more chances to win =) So, I would not recommend using the winning model in the production environment, and also, there is a high chance of missing the best model =)
1 Like
Yes, exactly this. I can get a score that would move me down to 4th or 5th on the Public LB simply by changing the random seed of my “best” model. There’s an incentive for me right now to just keep submitting the same model with a different seed hoping that at least 1 of them will be good enough for 1st on the Private LB. The competitors would just be throwing things at the wall and seeing what sticks.
The current format of competitions is based on many years of experience and what we have found works best, but we are always open to feedback about how to make competitions a better experience for our solver community! We appreciate your engagement with the competition structure, and will take these ideas into consideration for future competitions.