Private testset evaluation questions

I’m new on drivendata and have several common questions:

  1. Which submission will be evaluated on private data? Last one or the best by LB or we will be able to choose.
  2. How many submissions will be evaluated? One or two?
  3. The container must complete execution within a timeout of 8 hours . Is it only regarding public leaderboard or there is a limitation for testset evaluation? If yes, could you please provide timeout and size of testset?
    Thank you in advance.
3 Likes

@bull Could you please help with the questions, the most interesting now about time limitation because it may significantly affect the algorithm.

Which submission will be evaluated on private data? Last one or the best by LB or we will be able to choose.
How many submissions will be evaluated? One or two?

We will run the one submission that has the best score on the private leaderboard (season 11). Just one submission will be executed.

The container must complete execution within a timeout of 8 hours. Is it only regarding public leaderboard or there is a limitation for testset evaluation? If yes, could you please provide timeout and size of testset?

We only will execute the best submission (according to the private leaderboard), which had to complete within the time limit for the subset (8 hours). The runtime limit for the final evaluation will be proportional to the change in the number of files evaluated, so as long as the original submission respected the submission time limit for the current subset then you should be fine.

Thanks!

1 Like

@bull Does the submission need to be submitted and evaluated before the deadline or only submitted. If later, evaluation would finish a few hours after the deadline.

@isms @bull I hope someone can address my question (see above) before the deadline.

Submissions must be made before the deadline. Execution can happen after the deadline.