With ~1,300 scans and only a few thousandths of log loss between ranks, I suspect much of the current ordering is noise. Since we’re predicting a clinician’s read rather than dopaminergic status, there may also be a label-noise floor that many of us are already close to. Has anyone tried estimating that floor—and whether a big shake-up would simply confirm it?
1 Like
Hi, as a nuclear medicine physician ; I tried to estimate it, by going back and re-reading the scans blind.
The setup. my model gets 115 of 1,362 training scans wrong, about 8%, and those 115 carry 57% of my total loss. So i took the worst of them and had a reader look again, blind: anonymised renders, no labels, no model output, shuffled order, and confident normals mixed in as controls so the reader could not tell cases from controls by the shape of the worklist.
Three things came out of it.
First round, 14 of the model’s most confident false negatives. The reader sided with the model, against the stored label, on 9 of them. On the control scans the reader did not diverge, so it was not just a reader who says abnormal to everything.
Second round, 60 blinded reads against the stored labels. Agreement was 39 of 60. On scans stored as normal the reader called abnormal in 17 of 40.
Third, and the one I find hardest to argue with, we re-presented the same 14 scans to the same reader weeks later. They changed their own answer on 4 of the 14. Same reader, same images, no new information.
The caveat is that none of this is a random sample. We deliberately picked the scans the model found hardest, so these rates describe the difficult tail, not the whole dataset. They say nothing about the easy 90%.
What I take from it. If a reader disagrees with themselves on something like a quarter to a third of the hard tail, then for those scans there is no single correct answer to predict, only a probability. Roughly 40 scans in our set, 3% of the data, behave that way and carry a fifth of our loss. That is a real floor, and it is worth more than the entire spread between first place and seventh (mine currently) on this leaderboard.
But it does not explain the ranking, because every one of us pays it. What costs us extra is being confident on those scans rather than uncertain. We have 10 scans where the model is past 90% sure and wrong.
And I could never find them in advance. The thing that surprised me most is that my ensemble does not disagree with itself on its own errors. The mistakes are unanimous. A contested scan does not look uncertain to the models, it looks confidently wrong, so there is no way to hedge those without hedging everything.
So on a shake-up: I do not think it would confirm a floor, because the floor is shared. It would mostly reshuffle the mid-table, where ranks are separated by less than the run-to-run noise anyway.
4 Likes
Thank you for this—the test–retest is the part I couldn’t produce, and it’s the most convincing evidence.
I approached the same question from the other side, as a non-clinician unable to re-read the scans. So I had to infer rather than measure, and it’s interesting where our findings converge—and where they don’t.
Where we agree: My hardest 5% of scans account for 47% of my loss, close to your 8% accounting for 57%. My confident errors are similarly lopsided: 26 abnormal scans called normal, versus 11 normal scans called abnormal. The worst cases are bright, symmetric, comma-shaped scans stored as abnormal.
I also found that image-based difficulty is barely predictable. An out-of-fold model can identify the “moderately hard” scans, but the catastrophic cases are invisible to it. That fits your finding that an ensemble makes the same mistakes unanimously: a difficult scan doesn’t look uncertain to the model; it looks confidently wrong.
Where I had to stop: Without re-reads, I could measure the pattern but not its cause. At matched severity, scans in the most symmetric third were 32 percentage points less likely to be stored as abnormal than those in the least symmetric third. The effect peaked at intermediate severity and disappeared at both extremes. My interpretation was that the reader may be anchoring on asymmetry, since every reference describes it as the hallmark—and one explicitly warns that abnormal does not necessarily mean asymmetric.
But I genuinely can’t distinguish between “readers under-call symmetric reductions” and “symmetric scans at the same severity are genuinely less likely to be abnormal.” Your blind reads could answer that; mine can’t.
Love this discussion!
As a pure machine learning scientist, I did a similar investigation to see where my model fails. A few samples resulted in high loss and to my surprise it was mostly pathological. Since I was working with ResNets, I was interested to see how the samples distribute in the models feature space and came up with the following UMAP plot
This beautifully demonstrated that the model finds a great seperation for most samples but mistakes a few pathological samples for healthy ones. Also note the asymmetrie - There are no healthy controls overlapping into the pathological cluster, only the other way around. Matching the previous peoples findings, I could reproduce this plot for 20 different models trained on different splits, indicating a data-central issue.
My hypothesis (again, I am a computer scientist with no biological/medical background) was that early stages or some variant of this disease looks really similar to the healthy controls and my first idea was to reformulate this issue into a 3 class problem. Sadly, I ran out of time since I started this challenge a few weeks ago, but I think using this information could have been a good way improve performance.
That plot is the clearest version of this yet, and the asymmetry in it has a plausible clinical explanation. The scans I get most confidently wrong are bright, symmetric, comma-shaped and stored as abnormal. Symmetric bilateral reduction fits PSP/MSA better than typical Parkinson’s, and the references warn explicitly that abnormal does not mean asymmetric. So your “early stage or some variant” hypothesis may be right in shape, but about syndrome rather than severity.
The discouraging part: I tried predicting per-scan difficulty out of fold, to hedge rather than classify. It finds the moderately hard cases and is blind to the catastrophic ones which is your point about models not disagreeing on their own errors. If a contested scan looked contested, we could all shrink toward the prior and stop paying for it.
So I suspect your three-class split only helps if the third class comes from outside the images follow-up, or a second reader since the model’s own confidence is least informative exactly where it’s needed.
1 Like