I just realized the most time consuming aspect of the submission module is the Entry Point.
ie. In entrypoint.sh script, conda command kick starts the execution.
conda run -n py python main.py
However, it takes eons to enter into the main function. Same observed from the logs of Docker environment and actual submission logs.
ie. Initiating the conda environment takes most of the time compared to the actual submission file creation.
Can anyone confirm my observation, Is my understanding wrong… Please help.
Regards
Shankar