Hi Bull (and others),
I downloaded the original files with this command:
aws s3 sync s3://drivendata-competition-sfp-public-eu/train/annotated_regions/ /mydir --no-sign-request
Now I try to use it but get this error:
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
How can I download the new extracted regions?
Thanks in advance.
Thanks for letting us know, @booster! Looks like the permissions are still propagating. I’ll drop a note in here when it is finished for all regions (looks like it is almost done).
Looks like they are done now. Let me know if that works for you!
It’s working now, thanks!
I am getting an error when running the same command but with the target being another s3 bucket. The downsampled image folder works but tif folders seem to have different permissions?
Adding --no-sign-request will prevent you from writing to your S3 bucket if it is private (since no credentials are used). If you were using --no-sign-request, remove that from your command.
If you already had removed that, the permissions problem is likely with your bucket (the destination) rather than the hosting bucket. Make sure you have all of the permissions set properly in that bucket and your AWS credentials configured.
One rule of thumb is that if you can download the file locally, then the problem is with the destination if it fails when copying to an S3 bucket. You can test by uploading a file from your local machine to the same destination bucket.
One other note: the AWS tools treat large files differently than small ones (i.e., it does “multipart uploads”) so you’ll want to test with a large file.
If you still have problems, DM me the file URL and the region. I spot checked copying a few different files of different sizes and types across all the regions directly to a different bucket in a different account and everything worked as expected.
Thanks for the prompt reply and the suggestions. I tried some of your suggestions.
Also to note is that the command works for the folders “train/annotated_regions” and “train/downsampled_data” without throwing a problem but fails for the TIF folder. Maybe it some of the bucket policies did not properly propagate?
Indeed when the sync command goes to a local folder, it works. However when it’s another bucket I get the error below. The GetObjectTagging to me suggests it’s a problem getting the meta or list infos rather than the Put operation?
I also found the following which might be relevant? https://medium.com/collaborne-engineering/s3-copyobject-access-denied-5f7a6fe0393e
copy failed: s3://drivendata-competition-sfp-public/train/tif_images/C01_B205_S01.tif to s3://my_bucket/train/tif_images/C01_B205_S01.tif An error occurred (AccessDenied) when calling the GetObjectTagging operation: Access Denied
Ah interesting, this seems to need the newer AWS CLI 2.xx to repro the same error. It worked using the pip-installable 1.xx version which was on my test machine.
Made some updates and now things seem to work with AWS CLI 2.xx as well.
Thanks for the report!