Final Outcome: Model or Algorithm?

Water Supply Forecast Rodeo
1

A bit late may be to ask this question, but I realised it now. What is the final expected outcome of this challenge between the following two:

  1. A model/set_of_models that can be used in future issue dates (i.e. going ahead from now 2024, 2025, 2026 etc)? This model expected to be developed such that it has minimum LOOCV error across last 20 yrs.
    OR
  2. An machine learning algorithm that outputs lowest errors in last 20 yrs designed in a such a way that each year is left while training on others. The example.py given seemingly follows this route as it saves model with name having combination of site, year and quantile( line 168 f"{site}-{year}-{quantile}.joblib"). Here, e.g, model saved with year 2005 wont have any future use but it is developed in such a way that it gives minimum error for year 2005. Hence, this example code looks just like an ALGORITHM aimed to minimize LOOCV scores across all years but without any model/set_of_models for use in the future.

In my opinion, if we are doing CV then approach 1 should be the expected outcome.

Pls clarify.

Thanks!

2

it is developed in such a way that it gives minimum error for year 2005

To my ears this sounds like it is using validation data for fitting, which is bad modeling and should not be accepted as a solution in my opinion.

1 Like
3

Hi @kmande,

The goal of the challenge is to get the most accurate modeling methodology that can lead to models that can be used operationally in the future. The cross-validation is an evaluation procedure to estimate the accuracy of a modeling approach. As such, your “approach 1” describes this most closely.

What you describe in “approach 2”—producing a set of models that each individually minimize error for their respective test years—is certainly possible to do, but would be considered overfitting. This will be assessed from your model report, and solutions that are overfitting will be considered as having weak statistical rigor.

The example code saves out the models from each cross-validation iteration mainly for reproducibility and diagnostic purposes.

The conceptual “final product” would be a model trained on all 20 years of the cross-validation period, and the cross-validation procedure is an estimate of the performance of that final model. We don’t do this in the example because we’re not expecting any submission based on it in the Final Stage. However, the model you submitted to the Forecast Stage is basically such a final model.

1 Like