Hi @mmiron,
does this mean it’s possible to rank in a prize winning position, but not receive a payment because of the judges?
I’m just going to assume that it’s reserved solely for cases where the person legitimately shouldn’t have won, then (e.g. constructing the hold out set from publicly available data and training on it specifically or something like that).
The Hindcast Stage prizes (and also the Overall prizes later) are explicitly awarded based on an evaluation process that incorporates both leaderboard performance and evaluation of a model report by judges together. The leaderboard will be shown to give people an idea of their quantitative performance, but you should not think about ranks on the leaderboard as “prize winning positions”.
Forecast skill (i.e., the quantitative metrics shown on the leaderboard) will be the largest part of the evaluation criteria for solutions, but it is not the only evaluation criterion. More details about the evaluation criteria will be shared soon when the Evaluation Arena opens for testing.
The judging process in this competition exists because the task of seasonal water supply forecasting involves a very small set of ground truth data. In the Hindcast Stage, with 10 years for 26 sites, there are only 260 unique ground truth values. Because of this limitation in statistical rigor, solutions will additionally be evaluated qualitatively based on additional considerations of technical merit.
Solutions which violate challenge requirements like training on test data will be disqualified, and that is separate from the judging evaluation criteria.
personally I’m having difficulty clearly defining the specific data points that I’m required to predict. For example, I ended up noticing that during the hindcast stage, we’re apparently only required to predict one out of the 2-3 month season (season length varies by site). We’re literally given 2/3 of the data, and need only predict the remaining 1/3.
This is not a correct framing. Your task is to predict the seasonal water supply using only data from before the issue date. So even though we provide historical naturalized flow data for April, May, and June, you cannot use those months data as a feature if they don’t fall before a given issue date. For example, and issue date of March 15 may only use February and earlier naturalized flow observations as features. Please carefully no future data requirement.
but the data_read API is still being finalized
The data_reading code in the runtime repository is supplemental sample code that is provided to help participants get started. It is not an API for the challenge in the sense that participants are required to use it in their solutions. You should not wait for or depend on data_reading code for a data source to be added if you want to use that data source in your modeling.
The data_download code in the runtime repository does represent an API for the challenge, and you should make sure that you use it to download data in the same format. Data download code for approved data sources is mostly complete, and upcoming changes will primarily be for requested data sources that get approved.