Several Questions about the Evaluation Phase

Snowcast Showdown
1

Hi team,

I have several questions about the evaluation phase:

  • Can we decide not to submit the prediction in a particular week? Or should we submit the prediction for each week in order to be eligible for the prize?

  • How will the model verification be done? Will the host rerun the code for all submission points after the evaluation phase ends? What if there is a discrepancy between the prediction submitted by the participants due to input data changes? Is it possible that the data from HRRR/MODIS updated?

  • Is it possible that ground truth from past dates is updated/revised/added/removed later due to measurement errors or anything?

Predictions will be evaluated against ground truth labels as they become available, and your position on the public leaderboard will be updated each week. It is reasonable that you may need to implement minor fixes to your data processing pipeline to resolve any unforeseen changes to the input data (e.g., format, type, or API updates) during this time period

  • What kind of data processing pipeline modification is allowed? Can we update the pipeline on handling missing data?

I am afraid of data availability and any problems that might happen in the future. If the data is missing, the model will predict incorrectly. My model depends on HRRR and more than 50% of the data are missing for the latest submission because the data source from Azure is not accessible. I did not use AWS source because it is not permitted previously.

Thank you

2

Hi @rasyidstat!

We encourage weekly submissions but this is not required. If you don’t submit for a week, we’ll use your previous week’s submissions, as outlined here.

At the end of the competition, finalists will need to re-submit their code and weights. We’ll then re-generate submissions using this code as part of the verification process.

The only code changes you’re permitted to make during the prize-eligible evaluation period are things to make your submission run (e.g. handling an unexpected data type that throws an error).

Given the issue with HRRR and the recent approval of the AWS source, you are of course permitted to make a code change to use the updated data. In your case, that sounds like a change worth making!

1 Like