Data Extraction and Processing Times

[kovila]

Hello, just for info, i would like to know, on average, how long it’s taking you to download satellite data and processing it into the correct format for it be ready to feed into the model as features. Thank you!

[cszc]

Hi @kovila. I can’t speak for others, but our benchmark notebooks take about 1-1.5 hours to run end to end:

New Approved Datasets and Blog Post Tutorials

Hello Airathon Community! We’ve got just over two weeks left on the Airathon competitions. To help you perfect your models, we’ve added some new resources. Approved Datasets Requests for new datasets are now closed, but several datasets were approved last week, including: ECMWF IFS CY41r2 High-Resolution Operational Forecasts GEOS Composition Forecasting (GEOS-CF) CALIPSO Lidar (Particulate Matter Track only) Head to the announcements (pm2.5, no2) and the problem description pages (pm2.5, n…

In addition to code for downloading and processing data, the notebooks include code for training a rudimentary model. They only process MAIAC and OMI data for the pm2.5 and NO2 blog posts, respectively. I imagine TROPOMI will take the longest to download and process given the size of the dataset. FWIW, using a machine in the same region as the bucket greatly increases the speed of download.

[vkmr123]

Where are you getting satellite data from?
Send me the links and procedure to download
Also, send me the train_labels.csv file with SWE values for 11K cell_ids. Could not find it.

Thanks.

[cszc]

Hi @vkmr123 . Instructions for accessing the satellite data for the NASA Airathon competitions can be found on the data download pages (no2 and pm2.5).

You can direct SWE related questions to the Snowcast Showdown - DrivenData Community topic. Posts not related to the Airathon competitions posted under this topic will be deleted.