Two questions about how submissions are scored, since they change the optimal way to build a submission:
The problem description says the test set consists of newly identified faults that are not in the USGS database, and the metric penalizes every predicted pixel of probability mass that is not within 300 m of a ground-truth pixel. If a submission predicts the known USGS/INGENIOUS fault traces (which we are asked to train on), do those predictions count as false positives, or are pixels near known faults masked out / excluded from the FP term when scoring against the new-fault labels?
For the Final Prize Round, is the “entire updated label set” the new-fault set (Initial Round labels + expert-verified additions), or does it also include the existing USGS/INGENIOUS faults?
Related: the description says predictions should cover “all faults in the region”. Should we read that as “predict the union of known and unknown faults”, or “predict the unknown ones”?
Pixels corresponding to known USGS/INGENIOUS faults are masked / excluded from evaluation, so they do not count towards penalty terms.
Re-evaluation will also mask/exclude the existing USGS/INGENIOUS faults.
We’ll consider changing the description, but for scoring purposes it should not matter whether these known faults are included with predictions or not.
Thanks for the earlier confirmation that pixels corresponding to known
USGS/INGENIOUS faults are masked/excluded from evaluation in both rounds.
I’d like to pin down the geometry of that mask, because the metric’s 300 m
kernel means a pixel-exact mask and a buffered mask imply opposite
submission strategies.
Three specific questions:
Is the mask pixel-exact — only the rasterized known-fault pixels
themselves — or does it extend a buffer around them (for example the
300 m / 3 px kernel support)?
Consider a predicted pixel that is not itself masked but lies 1-3 px
from a known fault trace. Is its false-positive contribution computed
normally, i.e. penalized at [1 - max_g k(d)] against the new-fault
ground truth only? Or is it also excluded?
Can a ground-truth pixel in the new-fault test set lie within 300 m of
a known fault trace, or are such labels removed from the ground-truth
set as well?
Why it matters: a lot of what an expert would add to an existing map sits
just off the mapped line — splays, along-strike tip extensions,
hanging-wall structures. Under a pixel-exact mask, predicting those
corridors costs real false-positive mass. Under a buffered mask, the same
predictions are free. I’d rather not spend a submission slot working out
which regime applies.
The mask is indeed pixel-exact - it is identical to the provided set of training fault labels.
Only new-fault ground truth is considered for scoring purposes. A predicted pixel that is near a known fault trace but far from a new-fault ground truth pixel will be fully penalized, i.e., the buffer does not apply to known faults.
A new-fault ground truth pixel can indeed lie within 300m of a known fault trace. Such pixels would constitute corrections or modifications to existing fault traces. Identifying these corrections is one outcome we are aiming for as part of this competition. Such corrections may already exist in the new-fault set, and may also exist in the final round evaluation set.