Task 2 metric clarification

N+1 Fish, N+2 Fish
1

Hi, I don’t quite understand this remark in the description of how score for “Task 2: Identify species of fish” is calculated. The remark is

Note: For situations where AUC is undefined, MAE is calculated instead (e.g., for a video where all fish are the same species).

Why I find it confusing - I thought that AUC (I presume it’s ROC AUC) is calculated for each class on all frames across all videos. But this remark (“a video”) makes it sound like AUC is calculated on one video (or MAE if AUC is undefined) - is that correct? Are AUCs/MAEs on videos then averaged?

2

Ho @lopuhin, the metric is calculated per-video and then averaged. I have updated the problem description to make that clearer.

3

Thanks for clarification @bull.
That looks quite an interesting metric to optimise for. Any chance we can get a reference implementation of the whole scoring function?

4

Looks like it is here: https://github.com/drivendataorg/metrics/blob/master/metrics.py#L129