Congrats and 1st place brief summary

Many thanks to DrivenData team, Radiant Earth Foundation team, and all organizers and sponsors for this great challenge! Congrats and thanks to all participants! It was really hard to get to the top.

Special thanks to all people involved in data creation and preparation. High quality of the dataset makes it invaluable for any kind of research.

My solution is an ensemble based on several different architectures which were trained with different backbones and different depth of previous frames. Local cross-validation is based on 5-fold GroupKFold split by storm.

CNN + LSTM (unidirectional, without attention)
CNN + Transformer (1-layer, 6-layers, 12-layers)

ResNet (50, 101, 152)
SE-ResNet (50)
DenseNet (121)
EfficientNet (B0, B3, B4, B7)

I used default resolution of 366x366, augmentations (flips and rotations) and corresponding TTAs. Best single model is CNN+LSTM trained on 24 consequent historical frames with ResNet50 backbone. In my experiments deeper historical data did not substantially improve the score but contributed to ensemble. I trained everything with Adam optimizer. Batch sizes and learning rates are different with aim to have largest batch which can fit in memory. Learning rate schedule is simple reduction on plateau.


Thanks @vecxoz! And congratulations. Would it be possible for you to expand a bit on how you did the CNN+LSTM. specifically how you coded the historical frames. I’m knew to this and trying to figure it all out. Pointing to any resources would be great. Or even just terms and techniques I could search for.
Thanks again!

Thank you and great work @vecxoz! More or less we have shared the same approach. May you share the public and private scores of your best model?

1 Like

Thanks @cbenson,

Related paper:

My implementation is close to the last example (at the bottom) of the following post:

Thanks @cayala,

7.05, 6.63 – this is for EffNet-B3 backbone and this is self-ensemble of 5 folds and 9 TTAs. Actually I don’t have LB scores for mentioned ResNet50 backbone but based on my CV you can expect it to be a bit better than B3.

1 Like

Congratulations @vecxoz and thanks for sharing.
I am curious to know your HW setup. Can you share it with us?

Thank you @KarimAmer,

It was TPU.

1 Like