Generate Data Subset Error

Video Similarity Challenge
1

When I tried to run the Descriptor Track Code Execution Runtime, I executed make pull succesfully, but make data-subset returned an error:

rm -f data/test/query_subset.csv data/test/subset_ground_truth.csv data/query/.mp4
process_begin: CreateProcess(NULL, rm -f data/test/query_subset.csv data/test/subset_ground_truth.csv data/query/.mp4, …) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:139: _clean_subset_data] Error 2

Seems like it has error cleaning the data. So I try to execute the following command directly:

python scripts/generate_data_subset.py --dataset test --subset_proportion 0.01

Which should create the data subset, if I did not misunderstand its purpose. However, this returns another error:

DLL load failed while importing _multiarray_umath: The specified module could not be found.

Full error message is here:

error_msg
error_msg1611×872 18.6 KB

What I learn from the message is that the issue is due to Numpy. So I first check my numpy and python version. Numpy version is 1.24, and python version is 3.9. Seems fine, so I reinstalled numpy, but the issue persists.

Another thing is that according to the git repo, run make should return all possible make commands, but when I try to run make, there is an error:

process_begin: CreateProcess(NULL, uname, …) failed.
Makefile:179: pipe: Bad file descriptor
‘test’ is not recognized as an internal or external command,
operable program or batch file.
ECHO is off.
“$(tput bold)Settings based on your machine:$(tput sgr0)”
SUBMISSION_IMAGE=ab42bf8d56d1 “\t# ID of the image that will be used when running test-submission”
ECHO is off.
“$(tput bold)Available competition images:$(tput sgr0)”
“”
ECHO is off.
“$(tput bold)Available commands:$(tput sgr0)”
ECHO is off.
‘sed’ is not recognized as an internal or external command,
operable program or batch file.
make: *** [Makefile:179: help] Error 255

I am not very familiar with make, so I do not really know how to understand this error message.

I wish to get some insights on what can cause this issue and what methods can potentially fix it or some explaination on the error messages. Any help is appreciated, thanks!

2

Hey @Xiaoyang!

It looks like you’re running on Windows, which may be related to the issues you’re seeing.

It may not be enough to ensure numpy is installed - you may also have to add the necessary paths to the PATH environment variable so they can be discovered.

The makefile was designed to run on a UNIX system, where sed is typically installed by default. The makefile also relies on UNIX-style paths, which likely won’t work on Windows (perhaps related to the bad file descriptor error?). To get this to work on Windows, you may need to make significant modifications, and install some version of the UNIX tools that the Makefile relies on for its various commands.

For development purposes, you may find it easiest to create a development environment in a Docker container based on a UNIX operating system in which you can run these commands natively.

I hope this helps - please feel free to follow-up with any additional questions!

-Chris

1 Like
3

Thank you for your reply! I met another issue working with the vsc2022 repository:
Video Similarity Challenge 2022 Codebase
When I try to run it in docker, I used the docker Dev Environment created directly from the repository, but I have no idea how to install conda in it. I tried to install it with pip, but the conda installed with pip “cannot be execute as an app”, and it ask me to install with an installer. I am not sure how to use installer in a dev environment, what I tried was this:

wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && /bin/bash ~/miniconda.sh -b -p /opt/conda

export PATH=$CONDA_DIR/bin:$PATH

But it does not do any thing.

When I try to use windows to install the vsc-vscl and run the tests, it has error on five tests, shows errors like this:

PermissionError: [Errno 13] Permission denied: ‘D:\temp\tmp_xlb0je2’

and this:

ImportError: Failed to import test module: tests.test_localization

Could you provide some help on this? Thank you very much!

4

The provided Docker environment isn’t really meant to be a development environment. If you build it locally, by default it will not allow internet access, so you wouldn’t even be able to download pip or conda to install them. I would recommend creating your own development environment using Docker (you can pull from the same base image) and installing the same conda environment as in the runtime.