Different stages understanding confirmation

Water Supply Forecast Rodeo
1

Hi,
[EDIT: Please ignore first 2 points, 3. point includes both of them]
Could I get a confirmation about which models should be taken into account for Hindcast evaluation criteria?

  1. Do I understand correctly that the LB results submitted yesterday don’t count for anything as evaluation criteria will be based on LOOCV results from 2005-2023? Or does the LB result count for Hindcast Stage Prizes?
  2. I’d like to ask if we have to use the same models already submitted to the Hindcast Stage or the models could still be changed for that evaluation criteria. If it’s the latter, what is the deadline for changing models for the Forecast Skill (Hindcast) criteria, the same as for Hindcast model report deadline (January 26, 2024)? Then also, should we include models already submitted to the Forecast Stage in our Hindcast model report?

EDIT: 3. I’ve just read Forecast stage clarification - #2 by jayqi topic. Do I understand correctly that there are in fact 4 stages of the competition with different prizes included? Please correct me if I misunderstood some of the points.

  1. First one, already closed, for which Hindcast Stage prize (Prize Pool of $50,000) will be awarded for 3 best teams based on LB results and Hindcast model report (January 26 deadline). LOOCV evaluation isn’t used in this stage, just LB results.
  2. Second one (Forecast Stage, with Prize Pool of $50,000) for which models can be changed. Submissions for this part end on January 11 and the results from this part will be known in July after all forecasts for this water year are made, based on the models submitted on January 11. The requirement for competing in this stage is to make a successful submission for Hindcast Stage (1), the report from Hindcast Stage isn’t needed. Not sure if results for this stage will be updated every month or the models will just be run at the end of July.
  3. Third one (Overall, with Prize Pool of $325,000) - in this part models could be still updated up to March 15 when Final Model Report is submitted. LOOCV evalutation over the 20-year hindcast period is only for this stage. The requirement for competing in this stage is to make a successful submission for Hindcast Stage (1) and Forecast Stage (2), so we will know if we are eligibible for the price only at the end of July when Forecast Stage (2) is finished. If Forecast Stage (2) solution gets an error in May, our result from Overall stage won’t count as we get disqualified also from this Overall stage. Evaluation criteria from Competition: Water Supply Forecast Rodeo: Development Arena are only for this Overall stage.
  4. Fourth one,The Explainability and communication bonus prize (with Prize Pool of $75,000) - the deadline for this stage is yet unknown but to be eligible for this stage, all previous stages must have been completed.
1 Like
2

Hi @progin,

Your “Edit 3” has the general structure correct. This is a pretty large challenge with a complex structure of prizes, timelines, and submissions, so we welcome questions as well as feedback on the best way to explain it. The description of it being 4 stages (or some other word for stage) is potentially a useful way of looking at it, and it is something we have been considering about updating the documentation with.

A few points about the Forecast Stage are not correct. Please see the Forecast Stage website released earlier today.

Not sure if results for this stage will be updated every month or the models will just be run at the end of July.

Models will be run four times each month on the designated issue dates to generate predictions. These predictions won’t be scored against the ground truth until August.

If Forecast Stage (2) solution gets an error in May, our result from Overall stage won’t count as we get disqualified also from this Overall stage.

You will have the opportunity to fix errors. Additionally, errors that don’t get addressed won’t disqualify your submission—previous forecasts will be filled forward. See this section.

1 Like