# Version consistency between CUDA, PyTorch and other related modules **URL:** \ **Category:** PETs Prize Challenge\ **Created:** [January 19, 2023, 2:18am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462 "2023-01-19T02:18:37Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ryokawa](https://avatars.discourse-cdn.com/v4/letter/r/f475e1/32.png) [@ryokawa](https://community.drivendata.org/u/ryokawa)\ **Post date:** [January 19, 2023, 2:18am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/1 "2023-01-19T02:18:37Z")
Hi, When I was checking the versions of CUDA and PyTorch in the docker container that is used in pets-prize-challenge-runtime, I noticed that the version numbers returned through various interfaces seem to be inconsistent. This topic is somewhat similar to a past question here. [CUDA 11.0 and cuDNN / NVIDIA driver versions](https://community.drivendata.org/t/cuda-11-0-and-cudnn-nvidia-driver-versions/8227) - PyTorch: 1.12.1.post201 - CUDA version by PyTorch: 11.2 - CUDA version by nvcc: V12.0.76 - CUDA runtime version: 11.0.x I used the commands shown at the bottom of this message to collect the version information. These version numbers should be consistent. The last part of the PyTorch version number should correspond to a version number of CUDA, such as 1.12.1+cu116. However, post201 is an unknown number. The reason of this inconsistency seems to be in the choice of the base image and the python modules. In Dockerfile ```auto FROM nvidia/cuda:11.0.3-base-ubuntu20.04 ``` In environment-gpu.yml ```auto - nvidia::cuda-nvcc=12.0.76 - pytorch-gpu=1.12.1 ``` The exact binary package of PyTorch is determined by conda automatically from conda-forge channel, which resulted to choose 1.12.1.post201, which seems to support CUDA 11.2. However, that is different from the CUDA bundled in the base image (CUDA 11.0.x). Separately, nvcc of a different version is installed. I am not sure this inconsistency is still compatible at this moment, but at least I can say that this is not a popular or “standard” configuration. In the official website of PyTorch, PyTorch 1.12.1 binary package is assumed to be used with CUDA 10.2, 11.3, or 11.6. These are distributed in `pytorch` channel. > **[PyTorch](https://pytorch.org/get-started/previous-versions/)** > > An open source machine learning framework that accelerates the path from research prototyping to production deployment. As a result, other PyTorch-related modules distributed by the ecosystem assume the combination of the versions of CUDA and PyTorch shown above. I was trying to use PyG (torch-geometric, a graph neural network library), but I could not make PyG to work in the current runtime environment because it does not support the combination of CUDA 11.0 and PyTorch 1.12.1. The installation of PyG is successful but it causes a runtime error. Is there any way to upgrade the CUDA version to 10.2, 11.3 or 11.6? I think I can submit a pull request to change the base image and change the versions of the related packages correspondingly. (I know it is almost too late to propose this …) I used the following commands to collect the version numbers within the container. ```auto $ conda run -n condaenv python -c "import torch; print(torch._version_)" $ conda run -n condaenv python -c "import torch; print(torch.version.cuda)" $ conda run -n condaenv nvcc -V $ ls -la /usr/local/cuda-11.0/targets/x86_64-linux/lib/ ``` ---
**Author:** ![jayqi](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/jayqi/32/924_2.png) [@jayqi](https://community.drivendata.org/u/jayqi)\ **Post date:** [January 20, 2023, 8:44pm UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/2 "2023-01-20T20:44:36Z")
Hi @ryokawa (and also @kzliu given the other thread), We’re doing some further investigation to make sure things currently installed in runtime image work correctly. However, to give an initial response regarding the base image version (CUDA version 11.0.3): this version shouldn’t have any impact on the runtime because a different version of the CUDA runtime will be installed in the conda environment via cudatoolkit. We get drivers and other things from this base image version, but the CUDA runtime that comes with the base image is not used during evaluation. Accordingly, the base image version should not need to be changed, and the managing of CUDA runtime version compatibility happens entirely within what is installed in the conda environment. We’re taking a look at the packages in the environment and will follow up. ---
**Author:** ![jayqi](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/jayqi/32/924_2.png) [@jayqi](https://community.drivendata.org/u/jayqi)\ **Post date:** [January 21, 2023, 6:18am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/3 "2023-01-21T06:18:49Z")
After testing, we are making the following changes to the runtime environment in [this pull request](https://github.com/drivendataorg/pets-prize-challenge-runtime/pull/35) to standardize on CUDA runtime 11.2, which we believe should still work for folks depending on the current versions of packages but also resolve errors that were seen. All of the GPU-utilizing libraries like PyTorch, Tensorflow, JAX, or XGBoost in the environment are being pinned to CUDA 11.2 builds. This does not actually change any of these packages’ versions—they were previously resolving to `cuda112` builds already and the new PR is just making this explicit and transparent. We are pinning cudatoolkit to 11.2. Previously, cudatoolkit was not pinned and resolving to version 11.8. This turned out to be mostly compatible with the aforementioned `cuda112` builds of other packages, but appeared to sometimes break (torch-geometric was reported). Downgrading to 11.2 should ensure the best and most consistent compatibility across packages. Finally, we are installing NVCC via `cudatoolkit-dev` rather than the `cuda-nvcc` package from the `nvidia` channel. We were previously installing `nvidia::cuda-nvcc` to meet the requirements of JAX, per the [JAX documentation](https://github.com/google/jax#conda-installation). However, the important part is that JAX requires `ptxas`, and this is also available from `conda-forge::cudatoolkit-dev`. By installing `conda-forge::cudatoolkit-dev`, we can get versions that are compatible with CUDA runtime 11.2. We have tested the [`jit` example](https://github.com/google/jax#compilation-with-jit) from the JAX documentation in this setup and find that it seems to work. * * * @ryokawa @kzliu Please test out the new image and let us know if you have any issues. @ryokawa : We did some testing with `pytorch_geometric=2.2.0` and found that this environment seemed to work with it. If you would like to add `pytorch_geometric` to the runtime environment, please let us know or open a PR. ---
**Author:** ![kzliu](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kzliu](https://community.drivendata.org/u/kzliu)\ **Post date:** [January 21, 2023, 7:22am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/4 "2023-01-21T07:22:55Z")
Hi @jayqi, thanks a lot for looking into this! I was trying a centralized smoke test around 20mins ago, and am observing an unexpected error: ```auto ... Traceback (most recent call last): File "/opt/conda/envs/condaenv/lib/python3.9/site-packages/graph_tool/ __init__.py", line 373, in __del__ def __del__ (self): File "/opt/conda/envs/condaenv/lib/python3.9/site-packages/ray/_private/worker.py", line 1618, in sigterm_handler sys.exit(signum) SystemExit: 15 ``` This suggests that the error happens when importing `graph_tool` which was fine to import in my previous submissions. Could this be related to the recent changes? Thanks! UPDATE: I believe it has something to do with passing a `graph_tool.Graph` object for multiprocessing with the `multiprocess` package. The same multiprocessing code runs fine in my local environment and I’m curious if `multiprocess` (or python native `multiprocessing`) is allowed for the official runtime? Thanks! ---
**Author:** ![jayqi](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/jayqi/32/924_2.png) [@jayqi](https://community.drivendata.org/u/jayqi)\ **Post date:** [January 21, 2023, 4:07pm UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/5 "2023-01-21T16:07:55Z")
Hi @kzliu, I don’t think there should be any limitations regarding the use of multiprocessing specific to our runtime. Based on the traceback message and the error, it appears that the system sent your process a SIGTERM to kill it. It’s possible that this is an out of memory error. The pandemic datasets are relatively large, and if you spawned 5 multiprocessing workers and data gets copied, you could easily run out of memory on the system. ---
**Author:** ![ryokawa](https://avatars.discourse-cdn.com/v4/letter/r/f475e1/32.png) [@ryokawa](https://community.drivendata.org/u/ryokawa)\ **Post date:** [January 22, 2023, 1:51pm UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/6 "2023-01-22T13:51:04Z")
@jayqi Thank you for the updates! I will test the new environment and check whether my torch-geometric program can be executed or not. ---
**Author:** ![ryokawa](https://avatars.discourse-cdn.com/v4/letter/r/f475e1/32.png) [@ryokawa](https://community.drivendata.org/u/ryokawa)\ **Post date:** [January 23, 2023, 8:47am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/7 "2023-01-23T08:47:17Z")
@jayqi > > > This is related to discussion: > :: Version consistency between CUDA, PyTorch and… other related modules - PETs Prize Challenge - DrivenData Community https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/3 I submitted a pull request to add pytorch\_geometric=2.2.0. Would you merge this PR if it is not too late.? Sorry about this last-minute change. I have confirmed that a PyG program can be executed with the latest environment + this change. Thank you for the update. (FYI) However, I could not execute it if I manually install pytorch\_geometric using conda install -n condaenv -c conda-forge from the container after I built the official runtime environment. (The installation was successful, though. The following error occurred when importing torch\_geometric.) ```auto terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped) ``` ---
**Author:** ![ryokawa](https://avatars.discourse-cdn.com/v4/letter/r/f475e1/32.png) [@ryokawa](https://community.drivendata.org/u/ryokawa)\ **Post date:** [January 23, 2023, 2:51pm UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/8 "2023-01-23T14:51:40Z")
@jayqi To prepare for the case that the PR is not merged, I think I can manage to install pytorch\_geometric locally and offline by pre-downloading the required binary packages as explained in the following URL. [https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/installing-with-conda.html](https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/installing-with-conda.html) FYI: The above error from std::bad\_alloc has been fixed. Now, I can install pytorch\_geometric from the inside of the container. The root cause of the error seems to be that slightly different versions of pytorch\_scatter, pytorch\_sparse, and pyg-lib are installed in the case of the separate install. I needed to fix the versions precisely. ```auto conda install -c conda-forge --freeze-installed pytorch_geometric=2.2.0 pyg-lib=0.1.0=cuda112py39h83a068c_1 pytorch_scatter=2.1.0=cuda112py39h83a068c_0 pytorch_sparse=0.6.15=py39h83a068c_0 ``` Then, we can get a complete list of the dependent packages during the installation. We can find those .bz2 and .conda files in /opt/conda/pkgs or elsewhere. ---
**Author:** ![jayqi](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/jayqi/32/924_2.png) [@jayqi](https://community.drivendata.org/u/jayqi)\ **Post date:** [January 23, 2023, 3:40pm UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/9 "2023-01-23T15:40:46Z")
Hi @ryokawa, This is merged into main and the CI build is currently pushing the new image to the container registry. Once [this build](https://github.com/drivendataorg/pets-prize-challenge-runtime/actions/runs/3987763830/jobs/6838014175) is successfully completed, the new image with `pytorch_geometric` should be used by new submissions. ---
**Author:** ![ryokawa](https://avatars.discourse-cdn.com/v4/letter/r/f475e1/32.png) [@ryokawa](https://community.drivendata.org/u/ryokawa)\ **Post date:** [January 24, 2023, 3:30am UTC](https://community.drivendata.org/t/version-consistency-between-cuda-pytorch-and-other-related-modules/8462/10 "2023-01-24T03:30:46Z")
Thank you, @jayqi . It looks good to me.