Hello. Default (local) validation is not representative since it is > 10x smaller than the one on testing server. Make local scenarios by hand is super time consuming.
I understand that computational time is limited, but would you consider to increase the number of total submissions since
- it is directly serving to developing the better solutions for the problem
- current rules create inequality for people who joined the competition in the mid/end.
Thanks for reply in advance.
I can certainly understand your perspective, but the submission limit is something defined prior to the competition start by the competition organizers and involves not just computational time, but also the overall size of the dataset, risks of overfitting to the test set, and other considerations. In general, it’s not something that we can change mid-competition.
Sorry, I know this is not the answer you were looking for.
Hi, my I ask if there is a private leaderboard? If so, how many submissions could I choose as final submissions?
Good question, I can’t see a way to select submissions.
Usually here you cannot choose, and the final submission you send will be counted for private LB. So make sure that your last submission is your best. (CV or LB )
I got it. Could you please fix only one thing which is totally not understandable and unfair. Now is 13 days left till the end (>1 week). Why I have only 3 submissions left if second week is not yet finished?
I consider this as a bug and it shouldn’t be like that.
I may got it wrong from the rules page. If it is showing ‘1 week left’ does it mean that I have 3 submissions before the end or it’ll be according to the number of actual days?
I think you’ve got it now. The submission limit is 3 submissions in the last 7 days and the submission limit indicates the number of submissions currently available to you. Since you have not made any submissions yet, you currently have a budget of 3 submissions. Once you start making submissions, you will start using up this budget. But as long as you have not made 3 submissions in the past 7 days, you can make another submission. Each 7 day window rolls over at UTC midnight and it should say on the submission page when your next submission is allowed, so that you can plan.
By the way, submissions that fail to be scored are not counted against your limit.
I hope this helps to clarify.
Interesting! That makes only one submission to choose.
Thanks for the response.
Yeah, I got it now. Thanks for your explanation!
Huh! I thought all submissions count towards the private LB. @dd-mike could you please confirm if all submissions are considered for the private LB or just the last submission?
@karelds @kirusha @flamethrower @Ammarali32 @sheep
In this competition:
- There is a private leaderboard that is hidden and only revealed at the end of the competition. It is used to determine final prize rankings.
All submissions will be considered for the private leaderboard. This means that your final score will automatically be your submission with the best score. You do not need to choose final submissions through the UI, nor do you need to ensure your last submission is a particular one.
Note that other DrivenData competitions may only consider the last submission, or give you the ability to choose a fixed number of final submissions through the UI. In those cases, the competition documentation will specify the rules.
Hope that clears things up. Good luck in the final stretch of the challenge!