# Data Extraction and Processing Times

**URL:** <https://community.drivendata.org/t/data-extraction-and-processing-times/7489>\
**Category:** NASA Airathon\
**Created:** [March 13, 2022, 6:37pm UTC](https://community.drivendata.org/t/data-extraction-and-processing-times/7489 "2022-03-13T18:37:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kovila](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@kovila](https://community.drivendata.org/u/kovila)\
**Post date:** [March 13, 2022, 6:37pm UTC](https://community.drivendata.org/t/data-extraction-and-processing-times/7489/1 "2022-03-13T18:37:54Z")

</div>

Hello, just for info, i would like to know, on average, how long it’s taking you to download satellite data and processing it into the correct format for it be ready to feed into the model as features. Thank you!

---

<div class="post-metadata">

**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\
**Post date:** [March 14, 2022, 8:05pm UTC](https://community.drivendata.org/t/data-extraction-and-processing-times/7489/2 "2022-03-14T20:05:13Z")

</div>

Hi @kovila. I can’t speak for others, but our benchmark notebooks take about 1-1.5 hours to run end to end:

> [@New Approved Datasets and Blog Post Tutorials](https://community.drivendata.org/t/new-approved-datasets-and-blog-post-tutorials/7433):
>
> Hello Airathon Community! We’ve got just over two weeks left on the Airathon competitions. To help you perfect your models, we’ve added some new resources. Approved Datasets Requests for new datasets are now closed, but several datasets were approved last week, including: ECMWF IFS CY41r2 High-Resolution Operational Forecasts GEOS Composition Forecasting (GEOS-CF) CALIPSO Lidar (Particulate Matter Track only) Head to the announcements ([pm2.5](https://www.drivendata.org/competitions/88/competition-air-quality-pm/announcements/), [no2](https://www.drivendata.org/competitions/91/competition-air-quality-no2/announcements/)) and the problem description pages ([pm2.5](https://www.drivendata.org/competitions/88/competition-air-quality-pm/page/424/), [no…](https://www.drivendata.org/competitions/91/competition-air-quality-no2/page/435/)

In addition to code for downloading and processing data, the notebooks include code for training a rudimentary model. They only process MAIAC and OMI data for the pm2.5 and NO2 blog posts, respectively. I imagine TROPOMI will take the longest to download and process given the size of the dataset. FWIW, using a machine in the same region as the bucket greatly increases the speed of download.

---

<div class="post-metadata">

**Author:** ![vkmr123](https://avatars.discourse-cdn.com/v4/letter/v/ecd19e/32.png) [@vkmr123](https://community.drivendata.org/u/vkmr123)\
**Post date:** [March 14, 2022, 8:35pm UTC](https://community.drivendata.org/t/data-extraction-and-processing-times/7489/3 "2022-03-14T20:35:57Z")

</div>

Where are you getting satellite data from?  
Send me the links and procedure to download  
Also, send me the train\_labels.csv file with SWE values for 11K cell\_ids. Could not find it.

Thanks.

---

<div class="post-metadata">

**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\
**Post date:** [March 14, 2022, 9:04pm UTC](https://community.drivendata.org/t/data-extraction-and-processing-times/7489/4 "2022-03-14T21:04:55Z")

</div>

Hi @vkmr123 . Instructions for accessing the satellite data for the NASA Airathon competitions can be found on the data download pages ([no2](https://www.drivendata.org/competitions/91/competition-air-quality-no2/data/) and [pm2.5](https://www.drivendata.org/competitions/88/competition-air-quality-pm/data/)).

You can direct SWE related questions to the [Snowcast Showdown - DrivenData Community](https://community.drivendata.org/c/snowcast/49) topic. Posts not related to the Airathon competitions posted under this topic will be deleted.
