Same code for fails for Run # 3 due to timeout but suceeds in Run #4 within 900s

Same code for fails for Run # 3 due to timeout but succeeds in Run #4 within 900s.
What could be the reason?
Is there some more limitation when time limit of run is 1020s. I have lost many submissions due to this weird behavior if there is no extra limitation? May be containers underlying compute machines are not equivalent, some are slow or some are fast. Can this be looked into?
Can I get some help for tracking time in purchase phase?

Following is mentioned for compute_budget:
The compute_budget argument holds a floating point number representing
the time available (in seconds) for BOTH the pre_training_phase and
the purchase_phase.
Exceeding the time will lead to a TimeOut error.
I have following questions on the above description:
a) Is it possible to catch TimeOut error and save the model weights? I think may be not but then wonder what is the purpose of mentioning TimeOut error.
b) How do track time when training as compute_budget also includes time for pre-training, which is not known? Or I assume some time based on previous run. In this scope it limits the number of experiments in pre-training phase as first I have to determine pre-training time.

Regards
Aditya

View post 1

Internet in enbaled in training, purchase , prediciton

tried to submit 1 public code. Now this time got following

Time left: 10800
timeout: the monitored command dumped core
/home/aicrowd/run.sh: line 62: 365 Segmentation fault timeout $AICROWD_TIMEOUT_INFO python aicrowd_client/launcher.py

now submission are over.
It seems need to spend time on how to submit and not on the problem.

View post 0

Internet in enbaled in training, purchase , prediciton

Now facing cuda is not available
/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py:115: UserWarning: torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.
warnings.warn(“torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.”)
any idea

View post 0

Internet in enbaled in training, purchase , prediciton

Hi All
May be people have already figured it , so I like to know
Whether internet is enable so the code can download pre train weights.
My first submission says it is disabled.

If it is disabled then can we include weights in out gitlab repo, whether they will be copied.
Or pretrain models other than torchvision is not allowed, what are the rules.

If it is enable then I like to share my logs to organizers, how I can do that. Logs are saying that there is connection timeout

Regards
Aditya

View post 1

Waiting for a workspace - check your inbox

I have requested for workspace on 5/5/2021. No conformation mail from Aridhia Service Desk. Do I need to send the request again.

View post 0

Submission is stuck

Following submission is stuck, please look into this

View post 0

Lost submission due to internal server error

I lost submission due to internal server error.
Such issues should not result in failed submission.
Please look into this
Following is the submission ID

View post 0

Round 2 Data Availability

As round 1 is finished, when round 2 data will be available.

View post 0