# Sharing CV scores and LB scores **URL:** \ **Category:** VisioMel Challenge\ **Created:** [April 30, 2023, 10:11am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912 "2023-04-30T10:11:51Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Optimo](https://avatars.discourse-cdn.com/v4/letter/o/97f17d/32.png) [@Optimo](https://community.drivendata.org/u/Optimo)\ **Post date:** [April 30, 2023, 10:11am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/1 "2023-04-30T10:11:51Z")
Hi everyone, I wanted to make this thread so that contenders can share their CV vs LB scores. In fact I have seen a very poor correlation between CV and LB so I wondered if the same happened to you. Here are a few examples of my CV vs LB score (note that my CV is computed using the out of fold predictions of a 5 fold cross validation): CV 0.390 → LB 0.4364 CV 0.376 → LB 0.4361 CV 0.360 → LB 0.4462 CV 0.377 → LB 0.4627 The correlation is very poor as you can see, what about yours ? ---
**Author:** ![gbrought](https://avatars.discourse-cdn.com/v4/letter/g/b38774/32.png) [@gbrought](https://community.drivendata.org/u/gbrought)\ **Post date:** [May 1, 2023, 1:03am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/2 "2023-05-01T01:03:37Z")
So far, mine have been fairly correlated, with LB only 0.02 - 0.03 higher than CV, using metadata and a very simple 2 layer conv with attention on the patches and MLP head to classify the attention scores + embedded metadata. ---
**Author:** ![sheep](https://avatars.discourse-cdn.com/v4/letter/s/c67d28/32.png) [@sheep](https://community.drivendata.org/u/sheep)\ **Post date:** [May 3, 2023, 3:16pm UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/3 "2023-05-03T15:16:43Z")
Mine is similar to yours with nearly no correlation. Logloss is not suitable for this case. Let’s assume we have a false negative prediction and the total sample size in the public or private dataset is 260. If we yield 0.01 for a positive instance, this would result in a 0.018 influence on the average loss. ---
**Author:** ![Optimo](https://avatars.discourse-cdn.com/v4/letter/o/97f17d/32.png) [@Optimo](https://community.drivendata.org/u/Optimo)\ **Post date:** [May 3, 2023, 3:36pm UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/4 "2023-05-03T15:36:37Z")
This is a simulation from out of fold predictions, where I consider that 1/3 of the test data is in public leaderboard and 2/3 in private. I make random selection of a test set with the same size as the real one: ![image](https://canada1.discourse-cdn.com/flex028/uploads/drivendata1/original/2X/7/7a9d98f92f35d035ad82bd9f7e9326e19073257a.png) Looks like it’s hard to infer private score from public score. I also compared two different models of mine with similar CVs 0.376 and 0.377. Here are their corresponding scores on different random public set: ![image](https://canada1.discourse-cdn.com/flex028/uploads/drivendata1/original/2X/b/b87c367c88a0fff2f6f7b1258c37cf8c07ee454f.png) There could be a large a shakeup… ---
**Author:** ![BruhMann](https://avatars.discourse-cdn.com/v4/letter/b/87869e/32.png) [@BruhMann](https://community.drivendata.org/u/BruhMann)\ **Post date:** [May 4, 2023, 12:41am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/5 "2023-05-04T00:41:31Z")
I am suffering from the same problem as Optimo and was wondering the same thing. It could be that some pre-processing or attention needs to be applied. ---
**Author:** ![zsolt.bedohazi](https://avatars.discourse-cdn.com/v4/letter/z/4af34b/32.png) [@zsolt.bedohazi](https://community.drivendata.org/u/zsolt.bedohazi)\ **Post date:** [May 11, 2023, 10:00am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/6 "2023-05-11T10:00:00Z")
We achieved pretty high ROC AUC and accuracy scores locally, but we also experienced the exact same problem and we can also confirm that there is no strong correlation between the local CV fold scores and the public leaderboard scores based on the log loss metric. This is due to the fact that the distribution of the public test set is unknown which is a problem only due to the log loss metric, it wouldn’t be a problem if any other metric had been selected. It is highly probable that fitting the models to the distribution achieves much better log loss scores than letting the model learn meaningful signals. ---
**Author:** ![Optimo](https://avatars.discourse-cdn.com/v4/letter/o/97f17d/32.png) [@Optimo](https://community.drivendata.org/u/Optimo)\ **Post date:** [May 11, 2023, 10:15am UTC](https://community.drivendata.org/t/sharing-cv-scores-and-lb-scores/8912/7 "2023-05-11T10:15:58Z")
My best models have around 0.34 logloss and 0.82 AUC, what about yours @zsolt.bedohazi ?