Error when running "make test-submission"

Video Similarity Challenge
1

Hello, thanks for bringing this copy detection competition to us! I really expect to join in.

But I met a problem when I running make test-submission in GitHub - drivendataorg/meta-vsc-descriptor-runtime: Containerized runtime for the Descriptor Track of the Meta Video Similarity Competition
Here is the main part of log in submission/log.txt

Validating submission...
+ conda run --no-capture-output -n condaenv python /opt/validation.py --query_features subset_query_descriptors.npz --ref_features reference_descriptors.npz --query_metadata /data/query_metadata.csv --ref_metadata /data/reference_metadata.csv --subset /data/query_subset.csv
+ echo 'Running similarity search to generate subset rankings for scoring...'
Running similarity search to generate subset rankings for scoring...
+ conda run --no-capture-output -n condaenv python /opt/descriptor_eval.py --query_features subset_query_descriptors.npz --ref_features reference_descriptors.npz --candidates_output subset_rankings.csv
2022-12-19 07:23:52 INFO     Starting Descriptor level eval
2022-12-19 07:23:52 INFO     Loaded 83 query features
2022-12-19 07:23:52 INFO     Loaded 40318 ref features
2022-12-19 07:23:52 INFO     Performing search for 99600 nearest vectors
2022-12-19 07:23:54 INFO     too many results 12255872 > 199200, scaling back radius
2022-12-19 07:23:54 INFO     too many results 328200 > 199200, scaling back radius
2022-12-19 07:23:55 INFO     too many results 237113 > 199200, scaling back radius
2022-12-19 07:23:55 INFO     too many results 213381 > 199200, scaling back radius
2022-12-19 07:23:55 INFO     search done in 2.421 s + 0.244 s, total 161285 results, end threshold 14.4867
2022-12-19 07:23:58 INFO     Got 96303 unique video pairs.
2022-12-19 07:23:58 INFO     Limiting to 2075 highest score pairs.
2022-12-19 07:23:58 INFO     Storing candidates to subset_rankings.csv
+ echo '... finished'
... finished
+ echo '... finished'
... finished
+ echo 'Running similarity search to generate rankings for scoring...'
Running similarity search to generate rankings for scoring...
+ conda run --no-capture-output -n condaenv python /opt/descriptor_eval.py --query_features query_descriptors.npz --ref_features reference_descriptors.npz --candidates_output full_rankings.csv
2022-12-19 07:24:01 INFO     Starting Descriptor level eval
2022-12-19 07:24:01 INFO     Loaded 8295 query features
2022-12-19 07:24:02 INFO     Loaded 40318 ref features
2022-12-19 07:24:02 INFO     Performing search for 9954000 nearest vectors
2022-12-19 07:24:05 INFO     too many results 36767616 > 19908000, scaling back radius
2022-12-19 07:24:09 INFO     too many results 23277797 > 19908000, scaling back radius
2022-12-19 07:24:11 INFO     too many results 20995243 > 19908000, scaling back radius
2022-12-19 07:24:17 INFO     too many results 39909330 > 19908000, scaling back radius
2022-12-19 07:24:20 INFO     too many results 20196043 > 19908000, scaling back radius
2022-12-19 07:24:24 INFO     too many results 19990746 > 19908000, scaling back radius
2022-12-19 07:24:47 INFO     too many results 39617011 > 19908000, scaling back radius
2022-12-19 07:25:16 INFO     too many results 20004222 > 19908000, scaling back radius
2022-12-19 07:25:30 INFO     search done in 79.957 s + 7.992 s, total 12054860 results, end threshold 14.8545
/tmp/tmp76jdk3kh: line 3:    73 Killed                  python /opt/descriptor_eval.py --query_features query_descriptors.npz --ref_features reference_descriptors.npz --candidates_output full_rankings.csv
ERROR conda.cli.main_run:execute(49): `conda run python /opt/descriptor_eval.py --query_features query_descriptors.npz --ref_features reference_descriptors.npz --candidates_output full_rankings.csv` failed. (See above for error)

It looks like the subset is processed smoothly. But processing the full set goes wrong. I can’t see the detail error log, could you please take a look?

2

Hi @Zihao,

We’re looking into this. In the meantime, can you let us know whether your submitted descriptors are sorted by video ID? Another participant had an issue with the evaluation script that we’re working to fix, and you could be facing the same issue.

Thanks,
Mike

3

I have checked, it is sorted, since I only ran the demo script provided in your git. In all the four steps, I only failed at the last one.
ps: I observed that after printing “search done in 79.957 s + 7.992 s, total 12054860 results, end threshold 14.8545”, there is a long time that log doesn’t print anything. Then I got error. Could it be a time-out problem？

4

It might be a hardware issue - what are the specs of the machine you are running on? Can you try monitoring with htop to see whether you might be running out of memory?

5

cool! It’s the OOM problem, htop shows that half of memory (16G in total) is occupied by some unkilled process. So I restart my machine, It turns out make test-submission needs 11G for running.

Thanks!

2 Likes