is the track divided into A/B leaderboards? On the publicly available datasets I have seen, it seems that the training set + test set have fully covered 2058 patients. So will different parts of the test set be used at different stages of the competition to determine the leaderboard order? For example like kaggle using a small portion as live leaderboard and the other large portion to determine the final prize order. I think a fixed leaderboard to the end creates a great risk of overfitting.
On the rules page it is only stated that publicly available computer vision models are allowed, is it permissible to use publicly available with speech pre-training models such as wav2vec?
Yes, there is a private and public leaderboard. While the competition is ongoing, the leaderboard displays the standings based on the public test set. When the model arena closes, submissions will be reranked based on the private leaderboard. This private ranking will be used to determine cut-offs and prizes (note that final rankings for this competition also depend on judged qualitative reports).
On the rules page it is only stated that publicly available computer vision models are allowed, is it permissible to use publicly available with speech pre-training models such as wav2vec?
Good catch! Any publicly available pre-trained models are allowed, including wav2vec, as long as they comply with these guidelines:
(1) available freely and openly in that form at the start of the competition and (2) not trained on any data associated with the ground truth data for this challenge.
We’ll update the challenge text to reflect, thanks for flagging that!
Thanks for the clarification, but I have a further question. May I ask in what way the private leaderboards are presented? Do we need to submit inference code to re-predict the results on a test set we have never seen before? Or do we choose one or two existing submissions to score on different portions? Thanks.