Reproducibility issue, runtime environment-gpu.yml isn't installable on ubuntu 20 with gpu

On Cloud N
1

to reproduce:

mamba env create -f environment-gpu.yaml

it installs but then the blog post notebook doesn’t run and fails when importing pandas

import pandas

ImportError: cannot import name 'DtypeArg' from 'pandas._typing' (/home/rave/miniconda3/envs/condaenv/lib/python3.9/site-packages/pandas/_typing.py)

this looks related to this issue

https://github.com/pandas-dev/pandas/issues/42506
2

also, xarray-spatial is not included in the environment but it is used in the blog post notebook. installing xarray-spatial after the fact downgrades pytorch to use the cpu version, so it needs to be included to be able to run the blog post notebook locally.

3

I was able to reproduce the blog post nb with this env file

name: condaenv
channels:
  - fastai
  - nvidia
  - conda-forge
  - conda-forge/label/broken  # required imagecodecs is marked broken
dependencies:
  - xarray-spatial
  - albumentations=1.0.3
  - catboost=0.26.1
  - cudatoolkit=11.1.74=h6bb024c_0
  - cudnn=8.2.1.32=h86fa8c9_0
  - fastai=2.5.0
  - geos=3.9.1
  - geotiff=1.6.0
  - h5py=2.10.0
  - imageio=2.9.0
  - keras=2.4.3
  - loguru=0.5.3
  - mkl=2020.4
  - numpy=1.20.3
  - numba=0.54.1
  - pillow=8.2.0
  - pip=20.3.4
  - pydantic=1.8.2
  - pydap=3.2.2
  - pyepsg=0.4.0
  - pygeos=0.10.1
  - pyproj=3.3.0
  - pystac-client=0.3.1
  - pytest=6.2.4
  - python=3.9.7
  - pytorch-lightning=1.4.1
  - pytorch=1.8.0=cuda111py39h37e5b68_1
  - rasterio=1.2.6
  - scikit-image=0.18.2
  - scikit-learn=0.24.2
  - scipy=1.7.1
  - segmentation-models-pytorch=0.2.0
  - shapely=1.7.1
  - stackstac=0.2.2
  - tifffile=2021.8.8
  - tiledb=2.3.2
  - timm=0.4.12
  - torchvision=0.9.1=py39cuda111hcd06603_1_cuda
  - tqdm=4.62.0
  - typer=0.3.2
  - xarray=0.19.0
  - pip:
    - cloudpathlib[azure]==0.6.2
    - lightning-flash[image]==0.5.0
    - mmcv-full==1.3.9
    - mmsegmentation==0.16.0
    - pandas_path==0.3.0
    - planetary-computer==0.4.2
    - segmentation-models==1.0.1
    - tensorflow-gpu==2.6.0
    - ttach==0.0.3
    - crfseg==1.0.0

I had to add xarray-spatial in the conda section and remove pandas from the conda section (it installs an old version of xrspatial but the nb runs). I also edited the nb to use a smaller batch size and turned on gpu options in the pytorch lightning classes.

4

Does xarray-spatial in the pip section work better for you, like in this PR? I’m also interested in using xarray-spatial at runtime, while also keeping pandas around.

Ștefan

5

I haven’t tried xarray-spatial with pip because I didn’t want it to overwrite any libraries installed with conda when it installs pip dependencies. The above env file works for me. I think minimizing mixing pip with conda will lead to less issues down the road (like this one: BUG: can't import pandas · Issue #42506 · pandas-dev/pandas · GitHub), but if you need a new version of xarray-spatial pip might be the way to go.

6

@rbavery 'm not able to reproduce that error with the current environment file on Ubuntu 20.04; is there any more detail you can provide? You make a good point about conda and pip-specified dependencies. To some degree it’s unavoidable, since not all packages are in both sources, but I did clear up a few of the conda/pip conflicts in the most recent PR.

The xarray-spatial continues to be a bit of a struggle, but I think @stefan.istrate 's solution might be our best option. I’ll point out that in the benchmark xarray-spatial is only used for visualization, not for actual modeling, so there’s a possibility of just skipping it.

7

@rbgb This details the issue with the runtime env file: environment file installs but can't import pandas · Issue #11103 · conda/conda · GitHub The env file I posted above works because I edited it. Is that the env file that works for you or are you saying the runtime env file works for you and you are able to import pandas after it finishes installing?

8

Right, I was able to use the runtime env as it is in the repo now. Importing pandas is even part of the build tests, so I’m stumped as to why it would be different in your case.

9

@rbavery There were some updates to the env today that touch on pandas, you might try again with the most recent version of the environment-gpu.yml.

1 Like
10

Yep it looks like the updates today fixed it. The pandas version changed in PR 14 (side note, why does this forum block links to github with a message "you can’t link to that host??)

the pandas version that the runtime .yml used before today was the source of my conda/pip issue. The environment file I posted above reflects the pandas environment used in today’s PR.

1 Like