I’m a bit unclear on how Phase 2 works, perhaps you could help clarify the process for me. We now have the training and validation ground truth (labels). We will now train our models with the combined training and validation data. How does the testing work now? Are we still running predictions on the test features and submitting it for scoring against the hidden test labels? The following comment is confusing:
“Note that the validation set samples are whose scores determine the public leaderboard. That means that the scores on the leaderboard during Phase 2 may not be representative of true performance due to the availability of the ground truth.”
Hi @jimking100, that is correct. You should still run predictions on both the validation and test sets and submit those to the challenge. The submission format remains the same between Phase 1 and Phase 2.
The validation set score, which is what is shown on the public leaderboard, may not be representative of your true generalized performance. Relatedly, others’ scores on the public leaderboard may not be representative of their true performance. We encourage you to use techniques like cross-validation for estimating how your models are performing. You can compare that score against the snapshotted leaderboard from the end of Phase 1.
The test set score, which is hidden, will be revealed at the end of the challenge and will be used to determine prizes.
I’ve updated the announcement to clarify. Please let me know if you still have further questions.
Okay, I think I understand now. We will still make predictions based on both validation and test sets, but only the results of the validation set will be shown to us in the rankings. At the end of Phase 2 (April 18th) the results of the test set will be shown and the winner determined by the best score on the test set. Also, I assume the best score of our individual submissions (as is the case now) will be the one used in the final test set rankings, instead of say our last submission. We are still limited to 6 submissions per 14 days, so we need to be careful with our submissions! Let me know if I’ve got this right - thanks!