First time competing in the platform, don’t see the typical submission selector for final private LB eval.
Will this appear later, or does drivendata scores all team submissions on private?
thx
For this competition, every submission is scored on both a public and private leaderboard. Final rankings are based on the private leaderboard. Good luck!
What is the percentage of private leaderboard data out of the total test data? Or are we not entitled to that information?
i think it is a flaw in the competition design if participants are not asked to select the final submission.
especially in the regression problem, I can make a submission x first. then I can submit x+1, x+2, x-1, x-2. if there is test data bias, at least one of these submissions will give a good score. i did nothing to learn or prevent the data bias and the best solution is automatically selected.
hi @tglazer one clarification, imagine a team has two submissions that rank private LB top 1 and top 2. but top 1 is not reproducible or valid under competition rules. Will submission nº2 from the same team be considered? or only a single top 1 private leaderboard submission per team will be considered for final rankings?
@rquintino The leaderboard ranks teams by their best scoring submission; it does not rank submissions. You should not make any submissions that are not valid under the competition rules. If you believe you have a submission that is not valid, let us know ASAP and we will remove it.
thanks @tglazer I was mostly assuming that would be the case, but it’s good to have confirmation
Have to say this makes the participation really an additional challenge, not the best experience compared to other platforms where we pick submission to eval. Raises some concern on bias more submissions=more chances to win, even if random variation. But overall gets really tricky because every “competitive” submission we do must then be fully noted, tracked,etc for later reproducibility if it happens to be top scored on private LB. Would risk to say some teams wont notice this until it’s too late. Hope drivendata allows this feature soon, otherwise not highly motivated to participate in another one without submission selection.
(might worth surfacing this more clearly on competition home/rules)
much thx! Rui