Evaluation Submission Process

Snowcast Showdown
1

Hi,
I wanted to clarify the submission process during the Evaluation Stage. I’ve missed the first submission date of 1/13/22, but I’ve successfully submitted for 1/20/22 on 1/20/22. My submission has Nans in all the columns except 1/20/22 which has my predictions for each grid cell. Questions:

  1. I believe I cannot change the 1/13/22 values at this point, correct? Any values prior to 1/20/22 are ignored.

  2. I do have the option of forecasting into the future, but Nans will work if I choose not to forecast future values, correct?

  3. My window for submitting for 1/20/22 lasts for 3 days (until 1/23/22) and I can make multiple submissions during that time - I assume the last submission is the one used, correct?

  4. What happens if you submit one between 1/23/22 and 1/27/22, after the 3 day window, but before next week’s window?

I realize this is stage 2a and everything gets re-set on 2/15, but can you explain how the score works over multiple weeks? Is the score I see the cumulative score for week 1 and week 2? So if I missed week 1 my score is not going to look very strong, correct? Is there a way to see the individual week score?

Thanks!

2 Likes
2

Hi @jimking100 - Good questions.

  1. Correct
  2. Correct
  3. Correct
  4. Your predictions would be used for future dates, but any predictions for 1/20 or before would be ignored

As you pointed out, the period before 2/15 (Stage 2a) is for practice. So the scores will reset as of 2/15, and any previous weeks will be ignored in the real-time evaluation scoring that will be used for prizes (stage 2b). Here’s the timeline for reference.

The way scores are calculated is described here. Your first week will have been populated by no-snow (0.0) predictions. The published scores are cumulative. We’d encourage you to use the Development Stage for scoring feedback, and Stage 2a to get your inference pipeline and submissions set up for real-time evaluation.

Hope that helps!

4

Suppose that on 1/20 I make a prediction for 1/27 as well, if I don’t submit again on 3day window on the 27th you will use my latest submission, otherwise I can overwrite my 27th prediction on that 3day window, right?

5

@nima.shahbazi Right, you got it.

6

Thanks Glipstein,

Since many teams missed the first submission, there is no way we can use the leaderboard scores for scoring feedback then! (First submission is very tricky since you will fill it with 0 if not submitted!)

Any chance that for this test period you update the score weekly please? We just want to check the pipeline and validation strategy on a sample which weekly is good enough! Since there is no award, what difference does it make if you update weekly instead of cumulative, if that’s possible! Thanks so much.

1 Like
7

Hi @nima.shahbazi - That’s right. The scorer is set up to be cumulative throughout this period. This can’t be changed. Again, for further scoring feedback you’re encourage to test the system using the data you have available in the Development Stage.