I’m having some confusions regarding the rules, According to the rule,
Scores displayed on the public leaderboard while the competition is running may or may not be the same as the final scores on the private leaderboard, depending on how samples from the Data are used for evaluation. For that portion of the Competition evaluated qualitatively, the results will be determined by a panel of judges, as more fully described on the Competition Website. As more fully described on the Competition Website, the Competition may have a quantitative winner, a qualitative winner, a combined quantitative and qualitative winner, or all three.
So, my concerns are,
How exactly public and private leaderboard is calculated. In other words, what portion of the data is reserved for private leaderboard?
How judge will determine the result in qualitative evaluation?
What happens when there are both quantitative and qualitative winner?
Hi there! Happy to help, apologies for the delay in response.
Test data is randomly split between the public dataset, which is used to determine placement on the leaderboard throughout the competition, and the private dataset, which is used to determine the final winner. A submission’s score is calculated the same on each dataset (using mean absolute error). Scores tend to be very similar between the private test dataset and the public test dataset.
The portion of the rules discussing qualitative competition winners does not apply to the Deep Chimpact challenge, which has only a quantitative winner. The submission with the lowest mean absolute error on the private leaderboard will be the competition winner (assuming their submission is validated).
Same as question 2
Don’t hesitate to follow up with any other questions! Thanks for being part of the DrivenData community.