# Dowloading Data **URL:** \ **Category:** NASA Airathon\ **Created:** [January 27, 2022, 10:10am UTC](https://community.drivendata.org/t/dowloading-data/7188 "2022-01-27T10:10:43Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![smcoder5](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@smcoder5](https://community.drivendata.org/u/smcoder5)\ **Post date:** [January 27, 2022, 10:10am UTC](https://community.drivendata.org/t/dowloading-data/7188/1 "2022-01-27T10:10:43Z")
I am trying to download using boto3 and cloudpathlib in python, but I am getting error “Unable to locate Credentials”, I don’t know why I am getting this error but someone can tell me what is the name of bucket of amazon s3, from where I can Download??? ---
**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\ **Post date:** [January 27, 2022, 4:06pm UTC](https://community.drivendata.org/t/dowloading-data/7188/2 "2022-01-27T16:06:59Z")
As mentioned in another post, the name of the buckets and the individual filepaths can be found in the `satellite_metadata.csv` and data download file found on the [Data Download page](https://www.drivendata.org/competitions/88/competition-air-quality-pm/data/). Cloudpathlib should work, but to be explicit you can set the `no_sign_request` boolean kwarg when [initializing the s3 client](https://cloudpathlib.drivendata.org/v0.6/api-reference/s3client/#cloudpathlib.s3.s3client.S3Client. __init__ ). Hope that helps! ---
**Author:** ![smcoder5](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@smcoder5](https://community.drivendata.org/u/smcoder5)\ **Post date:** [January 28, 2022, 7:48am UTC](https://community.drivendata.org/t/dowloading-data/7188/3 "2022-01-28T07:48:54Z")
Hello CSZC, I am using Googe Colab as of now, for initial experiment: I am using cloudpathlib as follows: cp = CloudPath(“s3://drivendata-competition-airathon-public-as/pm25/train/maiac/2018/20180201T191000\_maiac\_la\_0.hdf --no-sign-request”) cp.download\_to(“1.hdf”) but still I am getting credentials error. I have tried both i.e. by not using “–no sign-request”. This is the first link of satellite metadata.csv Thank You ---
**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\ **Post date:** [January 28, 2022, 6:06pm UTC](https://community.drivendata.org/t/dowloading-data/7188/4 "2022-01-28T18:06:28Z")
If it’s not working for you out of the box, I would recommend instantiating an s3 client directly: ```auto from cloudpathlib import S3Path, S3Client s3_cli = S3Client(no_sign_request=True) ``` From there you can create an s3 path to a specific file, as in your example, or to the whole directory. In either case, you want to pass the s3 client you instantiated. Here’s an example to the `maiac` directory: ```auto maiac_path = S3Path("s3://drivendata-competition-airathon-public-as/pm25/train/maiac", client=s3_cli) maiac_files = list(maiac_path.rglob("*.hdf")) # get first file fp = maiac_files[0] # download file to temp directory fp.fspath # download to local fp.download_to("1.hdf") ``` ---
**Author:** ![eyastaifour](https://avatars.discourse-cdn.com/v4/letter/e/9dc877/32.png) [@eyastaifour](https://community.drivendata.org/u/eyastaifour)\ **Post date:** [February 1, 2022, 6:50am UTC](https://community.drivendata.org/t/dowloading-data/7188/5 "2022-02-01T06:50:33Z")
personally, I had some issues accessing the files until I ran the aws cli (without arguments) and it asked me to setup some AWS things - I entered random digits, and moved forward ---
**Author:** ![Eisenberg](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@Eisenberg](https://community.drivendata.org/u/Eisenberg)\ **Post date:** [February 2, 2022, 7:07pm UTC](https://community.drivendata.org/t/dowloading-data/7188/6 "2022-02-02T19:07:09Z")
Hello, I’m reaching over 90 GB only for the train dataset… But in the instructions they mention something like 7 GB. Is that normal? ---
**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\ **Post date:** [February 2, 2022, 7:17pm UTC](https://community.drivendata.org/t/dowloading-data/7188/7 "2022-02-02T19:17:52Z")
To which track/product are you referring? ---
**Author:** ![Eisenberg](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@Eisenberg](https://community.drivendata.org/u/Eisenberg)\ **Post date:** [February 2, 2022, 7:37pm UTC](https://community.drivendata.org/t/dowloading-data/7188/8 "2022-02-02T19:37:18Z")
I used this command line to download the directory on my local machine: aws s3 cp s3://drivendata-competition-airathon-public-eu/no2/train/ train/ --no-sign-request --recursive is that normal? ---
**Author:** ![kar95ar](https://avatars.discourse-cdn.com/v4/letter/k/e0b2c6/32.png) [@kar95ar](https://community.drivendata.org/u/kar95ar)\ **Post date:** [February 3, 2022, 4:33am UTC](https://community.drivendata.org/t/dowloading-data/7188/9 "2022-02-03T04:33:19Z")
I think you’ve downloaded the entire directory - which includes both maiac and misr satellite data. ---
**Author:** ![Eisenberg](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@Eisenberg](https://community.drivendata.org/u/Eisenberg)\ **Post date:** [February 3, 2022, 12:06pm UTC](https://community.drivendata.org/t/dowloading-data/7188/10 "2022-02-03T12:06:39Z")
Gotcha I figured out, its just that’s my first competition so I don’t really know by where should I start, how many files to dl ect 🙂 ---
**Author:** ![snande23](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@snande23](https://community.drivendata.org/u/snande23)\ **Post date:** [February 3, 2022, 7:49pm UTC](https://community.drivendata.org/t/dowloading-data/7188/11 "2022-02-03T19:49:01Z")
Is there a way to copy the competition data in aws s3 bucket to an Azure Blob Storage? Please help me with this because this is my first competition. ---
**Author:** ![JM1000](https://avatars.discourse-cdn.com/v4/letter/j/7993a0/32.png) [@JM1000](https://community.drivendata.org/u/JM1000)\ **Post date:** [February 18, 2022, 10:42am UTC](https://community.drivendata.org/t/dowloading-data/7188/12 "2022-02-18T10:42:59Z")
Hi, is anyone else struggling to connect to the s3/bucket endpoint? I have the following error: _fatal error: Could not connect to the endpoint URL: “[https://drivendata-competition-airathon-public-eu-central-1.s3.London.amazonaws.com/no2/train/tropomi/2019/20190101T213357\_tropomi\_la\_0.nc](https://drivendata-competition-airathon-public-eu-central-1.s3.London.amazonaws.com/no2/train/tropomi/2019/20190101T213357_tropomi_la_0.nc)”_ I’ve successfully pinged [s3.amazonaws.com](http://s3.amazonaws.com)… and have tried various region permutations. I’ve also tried pulling data with a scripy per cszc above response but i get the same error. Just wondered if anyone else had this problem, how did you solve it. Help… pulling my hair out 😂 Thanks ---
**Author:** ![Carl\_Malings](https://avatars.discourse-cdn.com/v4/letter/c/8c91f0/32.png) [@Carl\_Malings](https://community.drivendata.org/u/Carl_Malings)\ **Post date:** [February 18, 2022, 7:02pm UTC](https://community.drivendata.org/t/dowloading-data/7188/13 "2022-02-18T19:02:58Z")
Hello, I’ve had success with the following snippet of Python code: ```auto # Import the needed libraries: from cloudpathlib import S3Path, S3Client # URL of the file I want to download, from the satellite metadata csv file: s_url = 's3://drivendata-competition-airathon-public-eu/no2/train/tropomi/2019/20190101T213357_tropomi_la_0.nc' # Where I want the downloaded file to go on my computer: s_file_location = 'C:\\Users\\carlm\\Downloads' # Access the file: Remote_file = S3Path(s_url,client=S3Client(no_sign_request=True)) # Download it: Remote_file.download_to(s_file_location) ``` Hope this helps. ---
**Author:** ![cszc](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@cszc](https://community.drivendata.org/u/cszc)\ **Post date:** [February 22, 2022, 8:51pm UTC](https://community.drivendata.org/t/dowloading-data/7188/14 "2022-02-22T20:51:41Z")
Hi @JM1000 - is this working for you yet? Not sure where the url `https://drivendata-competition-airathon-public-eu-central-1.s3.London.amazonaws.com/no2/train/tropomi/2019/20190101T213357_tropomi_la_0.nc` is coming from - could you please post more of your code?