Back to DrivenData | Blog

Preparing for the Nov 9th Sprint #1 Final Scoring Invitation

First off, everyone who has submitted to DP Prescreening has now passed. Congratulations! If you haven’t submitted to prescreening yet, our last review will be held at 9am ET, next Wednesday 11/4. Make sure to get your algorithm write-up in before then!

Second, we’re getting closer to the Nov 9th deadline for invitations to Final Scoring. In order to be invited to final scoring and be eligible for the remaining $25,000 in prizes for Sprint #1, you need to do the following:

(1) Have passed prescreening (11 teams have done this!)

(2) Have submitted your solution to the “prescreened arena” (7 teams have done this!)

(3) Get a reasonably competitive score (better than the naive baseline solution) on the prescreened leaderboard (7 teams have done this!)

Note! If you’ve been following along on the open arena, but aren’t sure you’ll be able to make it in through the remaining steps for the Sprint #1 final scoring by Nov 9th, remember that this is only the first sprint of a three sprint match. Stay tuned! You’ll have another opportunity to jump into the competition this winter, and even a third opportunity next spring! Participating in Sprint #1 isn’t required to join in later sprints.

Once you’ve completed the above steps and are invited to final scoring, you’ll have until November 15th to prepare your final scoring submission. The final scoring will include a source code review and a much more thorough differential privacy verification (please be kind to your reviewers and comment your code!).

We’ll ask that you provide with your final scoring submission:

(1) Your executable code (just like you submit to the prescreen leaderboard)

(2) Your source code (commented and organized (but not so drastically last-minute refactored that you accidentally break it immediately before submission (this week, for instance, is a good refactoring week)))

(3) A Code Guide which shows where exactly in the source code to find each step of the privatization algorithm
…and also tells us which category each file/component of the code falls into:

  • Pre-processing (processes each individual’s records independently of all others to incur 0 sensitivity cost),
  • Privatization (processes whole data-set, incurs sensitivity cost and adds appropriate noise to account for it)
  • Post-processing (only touches privatized data and has no contact with raw input data).

(4) And finally, a clear and complete write-up of your differentially private algorithm and privacy proof. This should be a bit longer and more thorough than your prescreen submission if you’re doing something more than simple counts… it needs to clearly define what you’re doing at each step, so we can double check your logic. Make sure all of your notation is clearly defined. If you’re referencing a research paper, add a link to an accessible pdf, and include relevant excerpts in your writeup.

If we have questions about steps in your approach, we’ll reach out to you (you’ll receive an email from DrivenData asking for clarification), and we’ll need you to respond in order to maintain prize eligibility. Rather than anxiously watching your inbox over thanksgiving, it’s best to be clear and thorough in your write-up the first time around. :slight_smile: That may even help you catch any small mistakes yourself, before we get to them.

Hi Christine,
It looks like the leaderboard is not reliable, because someone submitted the non-private ground truth. Can we know what is the highest score among differentially private solutions ? Just to check with our solutions.

Thanks

So, you ask an interesting question. Because the prescreened arena is still scoring your submissions using the publicly available 2019 data-set (your algorithms don’t get to see the final scoring data until the development phase has fully completed), it’s possible for solutions to correctly provide differential privacy for their input dataset… and still score extremely high because they’ve been tailored to closely fit the publicly available 2019 data.

As we said in the webinar tips, it’s perfectly allowed (and even advisable) to use the publicly available data to learn about the general behavior of features and map segments, and hardcode these lessons into your solution. But you may want to be cautious about overfitting. :slight_smile: Distributions of real world data will naturally shift between datasets, and different datasets from this schema will be used for final scoring. To the best of our knowledge all solutions in the prescreened arena currently are differentially private, but their performance on this leaderboard may or may not reflect their performance in the final scoring.

Competitors concerned about the possibility of overfitting may find it useful to try out the bootstrapping technique for estimating variance from a single data-set:

Hello, if we already have entered the prescreened area but want to change our method for the final submission, do we need to submit a new write-up before the 11/9 deadline?

So, in any case, you will need to include a clear and complete updated write-up (and code-guide) with your final submission on Nov 15th.

You can change your algorithm between now and Nov 9th if you like, and try out the new algorithm on the prescreened leaderboard, and you don’t have to resubmit to prescreening.

However, the risk of switching to a new algorithm that hasn’t been prescreened is that if your new approach has any fundamental, significant DP violations, we won’t have the opportunity to catch them before the final DP validation, and that would make you ineligible for prizes.

If you can submit an updated write-up for prescreening by thursday afternoon, we can take a look at it on friday for you.

Ok thank you! I will submit an update to get some feedback. Thanks!

Hi Christine,

We just received the feedback regarding our submission. However, we found some something confused here, probably there is a misunderstanding here. Is there any way (like email) we can discuss with NIST SME panel regarding our write-up.

Thanks,

(Making the connection by e-mail as requested.)