Hello! Given that a portion of the model evaluation is the model’s “explainability”, would it be possible to further clarify how this term is being used in this context? Clearly, understanding how predictors drive forecasts and changes in forecasts is important to operational decision makers. But how is this typically done? Is it a matter of being able to extract feature importance, or choosing an easy to explain model, or do decision makers care about something else? How specifically does explainability affect decision making in water supply forecasting?
In addition, who will be the technical experts who assess the explainability of our models? Are they the decision makers themselves or engineers who have built similar data-driven models?
Finally, why do previous data-driven models that try to predict water supply rely on principal components regression? Is this related to explainability, since PCR models are simpler to explain? Or related to ease of making quantile predictions, since PCR combines easily with quantile regression?
Any help to clarify these three questions would be much appreciated
First responding to your initial questions as it relates to the challenge prize structure, submissions, and evaluation.
There are a few different places in the challenge where judges will qualitatively evaluate your solutions, so I want to be clear about distinguishing them.
Later, for the “Overall” prize evaluation, you will submit an updated and expanded final model report, and there will be a more comprehensive set of evaluation criteria. Details about the final model report and the overall evaluation criteria are not yet available.
Finally, there is the “Forecast Explainability and Communication” bonus track. This bonus track’s purpose is specifically to ask participants to innovate in the most effective way to explain and communicate their models’ predictions. Additional details about the submission format and evaluation criteria will be provided in the future.
I will follow up with the challenge organizers regarding your other questions. Thanks for your patience!
Hi @jitters, following up on the other questions in your post based on some input from the challenge organizers and SMEs.
You can see some published examples of how operational forecasts are presented linked on the problem description page. In general, helping decision-makers understand the forecast is important to them being trusted and used. Some helpful guiding questions include: Why is this forecast these values? Why has it changed from a previous forecast from an earlier date? Exploring effective approaches here is why we will have the Explainability bonus track later in the challenge.
The judging panel is not finalized. However, for the Explainability bonus track, the plan is to have both forecast issuers and forecast users represented on the judging panel.
Principal components regression is the traditional modeling approach with a long legacy in water supply forecasting. It improves upon more standard linear regression (which was used before PCR) by addressing colinearity and model reduction/feature selection. This 1992 paper is the main paper cited regarding the use of PCR in water supply forecasting. The last two links on the about page may also be helpful references on statistical modeling techniques in water supply forecasting.
