After reading problem description and evaluation metric section, it’s still unclear how IoU is calculated. In particular:
- IoU score can be calculated per each 512x512 tile independently and averaged across all test images.
- IoU score can be calculated per each flooding event and averaged afterwards.
In case of option 1, there are important to clarify edge cases, when target mask has no positive targets. What would IoU score for that tile would be, if model predicts no positive pixels and if it predicts some positive? I assume it’s 1.0 and 0.0 accordingly. But would be great if you can clarify this.
I wonder if it’s possible to include official scoring code to GitHub - drivendataorg/floodwater-runtime: Code execution runtime for the STAC Overflow: Map Floodwater from Radar Imagery competition?
Thanks in advance, Eugene