Sprint #2 Differential Privacy Prescreening (For Prize Eligibility!)

Differential Privacy Temporal Map Challenge
1

Hi all-- I wanted to provide a quick overview of the DP Prescreen process for everyone joining us in Sprint #2 (this post will look familiar if you were with us in Sprint #1, but check out the shiny updated links and dates!)

We’ll go more into detail on this during the webinar this week, but for anyone who wants an early start I wanted to quickly go over the DP Prescreen process. The goal of the challenge is to develop an algorithm that provides good utility on the temporal map problem… while preserving privacy. The leaderboard metric only scores utility, which means the easiest way to get a high score is just to take the ground truth data and submit it right back again with no privacy protection. Which is fair game for the open arena leaderboard, but not really to the objective of the challenge.We also need to ensure your approach satisfies Differential Privacy. The first step to doing that is, we ask that you submit a write-up explaining your algorithm and proving that it satisfies differential privacy.So! For your convenience–

  • Resources explaining differential privacy are here
  • Directions for prescreening are here
  • Prescreen submissions are accepted here
  • And a sample submission is in the competitor’s pack, here

SME Panel Prescreening happens every Wednesday morning. If you submit by Tuesday night, we’ll take a look Wednesday and get you feedback promptly. Prescreening is a high-level check for obvious/significant mistakes. We also do a DP Validation review during final scoring, which involves a much closer read of the proof as well as a source code review. Naturally, only valid differentially private solutions are eligible for final prizes.Prescreening is important for fun and profit! Once you’ve passed prescreening, you’ll be cordially invited to submit to a second, prescreened-only leaderboard. All submissions in the prescreened arena rank above all submissions on the open arena board. The easiest way to win a $1K progressive prize on Jan 25th is to make sure you’ve passed prescreening by Jan 19th, and then enter the prescreened arena and submit to the leaderboard there.Note that you must enter the prescreened arena before the end of the sprint on February 15th, in order to be eligible to be invited to final scoring. If you’re new to differential privacy, it’s a good idea to submit to prescreening early, and then (optionally) resubmit each time you make a significant algorithm change, to confirm you’re continuing to satisfy differential privacy and to better ensure you’ll be eligible for final prizes. We’ll give you feedback and help you understand any oversights or violations. Welcome to the challenge!

2

For Progressive prize, we only need to submit (1) a write-up before the deadline (2) and a csv submission file, and no need to submit implementation/codes. Is that correct ? Because it looks so easy to get 1K progressive prize as long as our write up is correct for DP part.
Or do we need (3) to submit also the codes, and the codes will be evaluated on another private data (not the public data )

3

If I am competing in a team, does every member have to submit a solution for prescreen separately to be included in the pre-screened arena? I only see an option for individual submission.

4

That is mostly correct (technically you don’t even need to submit the write-up), but it’s worth noting that when handing out prizes, everyone who’s on the pre-screened leaderboard (where you submit code rather than csv) is ranked above everyone who’s on the open leaderboard. We hand out four progressive prizes to the top four ranked teams on the leaderboards. That means that if there’s only three teams that have submitted code to the prescreened leaderboard, and you’re the top ranked team on the open leaderboard, you get a prize. But as soon as there’s four teams that have gotten through to the prescreened leaderboard, then you won’t be eligible. Submitting a write-up that passes the differential privacy sanity check is important because that’s what gets you access to submit to the prescreened leaderboard.

5

So if you’re on a team, your team captain should be the one doing all of the official communication with the website (including submitting the write-up). We only need one write-up per solution/team. I’ll let the DrivenData folks clarify more regarding how teaming works on the platform.

6

@joiewu @Christine_Task Regarding teaming, that is correct. Only one pre-screening write-up should be submitted per team, and if it’s approved then all members of that team will be permitted to enter the Prescreened Arena.

7

Hi Greg, one of our team members already passed prescreening. We didn’t create a team yet so they tried to add us to the team after the fact, but it won’t work since the other members are still in the Open Arena. Is there any way to form a team now?

8

@joiewu Sure thing, let me know who the participants are that are part of the prescreened team and we can add them to the Prescreened Arena. Then they will need to join the team when they enter.