Leaderboard aggregation: pooled over public-test pixels, or mean of per-chunk scores?

GEMS Prize Challenge
1

Hi DrivenData team, and thanks again for the earlier clarifications on known-fault masking and the 300 m rule, they were very helpful. One more question on how the distance-weighted Tversky score is aggregated:

  1. Is the public leaderboard score computed by pooling the distance-weighted TP, FP and FN over all public-test pixels and then taking a single Tversky index, or as the mean of per-chunk Tversky indices?
  2. If it is per chunk: what is the chunk size, and are chunks with no new-fault pixels excluded from the mean?
  3. Is the same aggregation used for the private leaderboard and for the final-round re-evaluation?

Thanks for running such a well-documented challenge.

2

Hi @joeyfezster,

  1. The public leaderboard score is computed by pooling over all pixels in the public subset and computing a single Tversky index.
  2. We are not providing any information on the chunking / subsets.
  3. The private leaderboard score is computed the same way. The final re-evaluation will be on the entire GeoDAWN area.

Hope this answers your questions!

-Chris