Submission upload fails after 1 hour - Increase session timeout

Youth Mental Health: Automated Abstraction
1

Hi,

I’m not able to upload my submission because server returns an error (please contact administrator) after 1 hour of upload. It seems to be related to HTTP request token timeout according to the browser error in console:

{
“name”: “c”,
“message”: “Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:ae6e3416-b01e-0075-6b69-12fbd7000000\nTime:2024-09-29T12:14:26.4503513Z”,
“code”: “AuthenticationFailed”,
“authenticationerrordetail”: “Signed expiry time [Sun, 29 Sep 2024 12:14:26 GMT] must be after signed start time [Sun, 29 Sep 2024 12:14:26 GMT]”,
“statusCode”: 403,
“requestId”: “ae6e3416-b01e-0075-6b69-12fbd7000000”
}

Thanks.

2

@kwetstone Is it possible to increase upload session time out to 3 hours or more instead of 1h?
It will benefit to all competitors.

Thanks.

3

Just curious, how much size in GB is your submission ?. Probably an estimation of the upper bound on submission size in GB will result into >1 hour upload time will be very useful for all people here. Because we can understand how big the model that we should use.

I know the answer might depend on other factors like Internet bandwidth. Just an estimation is enough. Personally, I tried to upload a submission (zip file) of size 30GB and it still worked well.

4

It’s around 50GB. And yes, my internet bandwidth is not the best.

5

@MPWARE We’ve increased the upload time to 3 hours as requested, to help folks work around low internet bandwidth.

A 50GB submission is still very large – note that your submission will be limited by the runtime constraints (16GB GPU, 56GB CPU, 8 hour execution time). Code execution resources are shared across participants, so please follow the instructions on the code submission page to make sure your submission will both run without error, and run within the time limit. Before submitting on the full test set, please test your submission locally with Docker and then with a smoke test submission.

1 Like
6

@kwetstone Thanks, it worked.

7

I just got the problem of “There was a problem uploading your submission to the cloud. Please try again”
Is there any issue with disk space in cloud ? Or because so many people are uploading at the same time ?
I uploaded a test submission that has very small files. It is showing pending (uploading status not running status) now. Should I cancel the job, and resubmit it ?

8

I got this error yesterday after 1% upload. Then I tried again (cancel the failed submission first) and it worked fine. I’ve uploaded something this morning without any problem. If you have this problem again, open browser console (before getting the error) and you should see some detailed logs about the problem.

9

@kwetstone Last question: What is the disk size limit?

“The container runtime has access to a single NVIDIA T4 GPU with 16GB VRAM. It also has access to 8 vCPUs with 56GB VRAM”.

Thanks again.

10

@MPWARE The node type of the VM is “Standard_NC8as_T4_v3”, which has 352 GB of temporary disk space.

However, it is unlikely that disk size is limiting your solution in practice. The constraints of the RAM (16GB GPU and 56GB CPU) will still apply. Anything that you need to load into memory while running your solution must fit in the RAM available.

11

Even 3 hours is very limited for me, given my internet speed. Could you please consider removing the upload time limit and instead just limit the file size? This would make the process more equitable for all competitors.

Thanks!

12

Hi @ouahab7, thanks for your suggestion. How large is your submission? We can look into increasing the upload time allowed, but just wanted to note that even after upload large submissions may still be limited by the runtime constraints outlined here.

13

My submission is about 40G, and in 3 hours I can upload only 7G.

14

How about limit the uploading file size if you really want to have a limit? Otherwise it is just a compete for network speed!

15

@ouahab7 @legend We’ve increased the upload time to 20 hours, which should work for your 40GB submission. Feel free to let us know if you continue having issues.

16

Thanks, 20 hours is enough!

17

@kwetstone
How can we check the duration of our submission? Is there any log for this?
If not, could you please add it to the log file?

18

@bamps53 When you submit a smoke test, you’ll be able to see all the logs generated during the process. You can add logging to your main.py to help track how long different steps are taking. For example, the benchmark blog post uses loguru.

Note that you will be able to see your submission logs for a smoke test, but not for a full submission. You can use the smoke test logs to estimate the amount of time your submission will take on the full 1k test samples, and ensure that it will run in the 8 hour time limit.

19

@kwetstone
It only works if the text lengths in the smoke test and the real test set match, but we’re not certain about that.
Also, the duration for the full test set isn’t hidden; if you monitor the code job, you can track it. However, due to the long pending time, it’s hard to tell when it will actually start. Please include the submission duration in the detail columns on the submission page.

20

@kwetstone
Could you please increase the log text length? My log is filled with unnecessary unzip information, so I cannot see what I really want to check.
10,000 would be enough, but more is safe.

Thanks.