# 20k+ null reference\_id in public\_ground\_truth?

**URL:** <https://community.drivendata.org/t/20k-null-reference-id-in-public-ground-truth/6229>\
**Category:** Image Similarity Challenge\
**Created:** [June 29, 2021, 11:01am UTC](https://community.drivendata.org/t/20k-null-reference-id-in-public-ground-truth/6229 "2021-06-29T11:01:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![henrique](https://avatars.discourse-cdn.com/v4/letter/h/ecb155/32.png) [@henrique](https://community.drivendata.org/u/henrique)\
**Post date:** [June 29, 2021, 11:01am UTC](https://community.drivendata.org/t/20k-null-reference-id-in-public-ground-truth/6229/1 "2021-06-29T11:01:32Z")

</div>

Most query images in public\_ground\_truth.csv don’t have a `reference_id`  
Is the same expected to the test set (the other 25k images)?  
Thank you

---

<div class="post-metadata">

**Author:** ![mike-dd](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mike-dd](https://community.drivendata.org/u/mike-dd)\
**Post date:** [June 29, 2021, 7:03pm UTC](https://community.drivendata.org/t/20k-null-reference-id-in-public-ground-truth/6229/2 "2021-06-29T19:03:18Z")

</div>

Hi henrique! Welcome to the challenge, and thanks for the question.

You are correct in noting that most query images in the public ground truth do not have a reference id. These are the “distractor” queries.

There are a total of 50K query images, of which 10K have matches in the reference set, while the remaining 40K do not. See below a relevant section of the “2021 Image Similarity Dataset and Challenge” paper:

> Development query image set (Phase I): 10,000 images from the reference set mixed with 40,000 distractor images that are not part of the reference set, that have been edited in various ways. Distractor image queries have no matching counterpart in the set of 1 million reference images.

I hope this helps - good luck!