# Data download from server/cli

**URL:** <https://community.drivendata.org/t/data-download-from-server-cli/4910>\
**Category:** Genetic Engineering Attribution\
**Created:** [August 22, 2020, 5:57am UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910 "2020-08-22T05:57:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![olivier](https://avatars.discourse-cdn.com/v4/letter/o/d2c977/32.png) [@olivier](https://community.drivendata.org/u/olivier)\
**Post date:** [August 22, 2020, 5:57am UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/1 "2020-08-22T05:57:22Z")

</div>

Hello,

I’d like to know if there is a way to download the datasets using a command line on a server.

Thanks.

---

<div class="post-metadata">

**Author:** ![davebel](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@davebel](https://community.drivendata.org/u/davebel)\
**Post date:** [August 22, 2020, 10:49am UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/2 "2020-08-22T10:49:24Z")

</div>

Hi. I was able to download the data using _ **wget -O train\_values.csv --no-check-certificate --no-proxy "url\_link\_to\_csv"** _. The url needs to be inside quotation marks.

---

<div class="post-metadata">

**Author:** ![olivier](https://avatars.discourse-cdn.com/v4/letter/o/d2c977/32.png) [@olivier](https://community.drivendata.org/u/olivier)\
**Post date:** [August 22, 2020, 10:57am UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/3 "2020-08-22T10:57:57Z")

</div>

Thanks a lot @davebel that works !!!

---

<div class="post-metadata">

**Author:** ![binarysurface](https://avatars.discourse-cdn.com/v4/letter/b/e79b87/32.png) [@binarysurface](https://community.drivendata.org/u/binarysurface)\
**Post date:** [September 9, 2020, 11:24pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/4 "2020-09-09T23:24:17Z")

</div>

Are the links still valid - there have been some issues with obtaining data? An _aws ls_ command indicates the bucket doesn’t exist.

---

<div class="post-metadata">

**Author:** ![KieranLitschel](https://avatars.discourse-cdn.com/v4/letter/k/3da27b/32.png) [@KieranLitschel](https://community.drivendata.org/u/KieranLitschel)\
**Post date:** [September 10, 2020, 8:55am UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/5 "2020-09-10T08:55:02Z")

</div>

The URLs at least seem to expire after 24 hours. Have you checked to see if the URL has changed since you copied it?

---

<div class="post-metadata">

**Author:** ![binarysurface](https://avatars.discourse-cdn.com/v4/letter/b/e79b87/32.png) [@binarysurface](https://community.drivendata.org/u/binarysurface)\
**Post date:** [September 11, 2020, 3:33pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/6 "2020-09-11T15:33:28Z")

</div>

Kieran–

Thank you for the response. It seems there may be some other issue - the repository seems now to be entirely unavailable or inactive. Even _aws cli_ attempts were unsuccessful.

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [September 11, 2020, 8:56pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/7 "2020-09-11T20:56:04Z")

</div>

Take a careful look at what happens to the URL when you paste it into your shell. Some shells will “helpfully” escape some of the URL characters like `&`, and that escaping will break the link if it is in quotes, but not if you leave the quotes out.

If you can click the link in the browser and the download starts, then your URL is not yet expired and there is some problem with what you pasted in the shell.

---

<div class="post-metadata">

**Author:** ![binarysurface](https://avatars.discourse-cdn.com/v4/letter/b/e79b87/32.png) [@binarysurface](https://community.drivendata.org/u/binarysurface)\
**Post date:** [September 14, 2020, 9:17pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/8 "2020-09-14T21:17:03Z")

</div>

Initial attempts to obtain relevant data provided a redirect-link which permitted downloading/opening datasets only to eventually timeout. Most recent attempts to obtain data lend the following:

`AccessDenied`Request has expired864002020-09-09T22:17:35Z2020-09-14T21:15:21Z466A2211FDD5CD9B1m7jOH8IX0NHNQGSzIm/z+rEFuaw8bS0VQsH9dfivjBpNOANsQtNbPLiJlwzhLxXk3j6eh1DMOE=

Several attempts were made to obtain datasets via _aws-cli_ to no avail.

It appears as though the datasets have been either removed or the accessibility otherwise restricted.

Anyone on this - several weeks in on obtaining data?

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [September 14, 2020, 9:49pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/9 "2020-09-14T21:49:22Z")

</div>

Does clicking on the download link on the data page start a download for you? (If you get access denied, refresh the page and try again—users get specific expiring URLs generated for them).  
[https://www.drivendata.org/competitions/63/genetic-engineering-attribution/data/](https://www.drivendata.org/competitions/63/genetic-engineering-attribution/data/)

The wget command mentioned above also works if you are not on a machine with a web browser.

You can DM me the command you pasted in the shell, and I’ll take a look.

We haven’t had other reports of problems downloading the data, and have had many successful submissions, so I expect there is an issue with the command you are trying to run.

---

<div class="post-metadata">

**Author:** ![binarysurface](https://avatars.discourse-cdn.com/v4/letter/b/e79b87/32.png) [@binarysurface](https://community.drivendata.org/u/binarysurface)\
**Post date:** [September 15, 2020, 6:18pm UTC](https://community.drivendata.org/t/data-download-from-server-cli/4910/10 "2020-09-15T18:18:25Z")

</div>

Thank you for your help!
