I’m a bit confused that can I train on the reference set ? Since we need to compute the similarity between the reference and query images pair, it is necessary to learn the embeddings of the reference set.
Hi gradient.machine. Thanks for the question.
Use of the reference images for training is discouraged but not prohibited. We would instead strongly encourage you to use the provided training set for training purposes. Check out the “Getting started” blog post for some more information about how to use the training set.
Hope this helps. Good luck!
I don’t understand.
The rule says that “Submitted individual predictions may not take into account more than one query image or more than one reference image at a time.”
Does the model trained with reference images is able to take into account only one reference image at each prediction? The model already saw all reference images and learned from them.
2 Likes
Hi separate! Thanks for the question.
The key point here is that predictions need to be independent of one another in the sense that they are not affected by the presence of other images. See the following section of the Rules:
The goal of the competition is to reflect real world settings where new images are continuously added to the reference set, or old images get removed from it. Therefore, the used approach should have an equivalent result if all of the reference images are provided at once, or if they are provided in chunks over time. We restrict the focus here, and only allow approaches that treat each new image of the reference set independently and without any interaction with other reference images.
You should also check out the “Acceptable methods” section (6.5) of the competition paper, which covers the same topic.
Please let us know if anything remains unclear.
I do NOT understand why you argue that training on the reference dataset is not prohibited. In the early reply, Is legal for training on the reference dataset? and the rule Competition: Facebook AI Image Similarity Challenge: Matching Track have already pointed out "Use of augmented reference images for any other reason, including model training, is prohibited ".
1 Like
Hi wenhaowang.
Note in the sentence you have quoted that augmenting reference images for training purposes is prohibited. From the Rules section you can see that we define augmenting as applying a transformation to an image to generate a new image, such as the manner in which the query set images were derived from the reference set images.
Applying these transformations to the reference images and then training is prohibited. Training using the reference images themselves without augmentation is not prohibited; but again, participants are encouraged to use the training set for these purposes.
1 Like
Ok, thanks for your reply.
By the way, is the resize operation, such as resizing an image to 512x512, is regarded as a transformation?
1 Like
Or, whether RandomCrop is regarded as augmentation? Because for training, images with different shapes are difficult to process.
Thanks for the question, wenhaowang.
Resizing images once as part of preprocessing to get consistent image sizes is allowed, since this is a basic requirement for many CV techniques.
Any form of augmentation is allowed during inference. For example, you could run inference on multiple augmentations of a single image and aggregate the results for that image independently of other images.
Augmentation (including resizing or cropping) for training purposes is only allowed on the training set, not the reference or query sets.
I hope this helps.
From the competition rules on augmenting:
- “Augmenting” images refers to applying a transformation to an image to generate a new image, such as the manner in which the query set images were derived from the reference set images.
- Augmenting reference images is permitted in the inference process of generating embeddings and matching scores, so long as each reference image is used independently without any interaction with other reference images. Use of augmented reference images for any other reason, including model training, is prohibited.
Thanks for your reply. It helps.
Hi,
Are we allowed to normalized reference images for training?
For example, we first divide the image by 255, and then submit mean and divide std of each channel. If we want to train on reference dataset, are we allowed to do this normalization please?
“Phase III: verification. (November) The organizers verify that the code submitted in advance reproduces the reported results. The participants are expected to reply to requests from the organizers to assist in this validation.
The results will be announced at the NeurIPS competition workshop.”
But the organizer cannot catch collaborating between teams. For example, there are three people working together, but they registered the competition sperately as three teams. Thus they would have 3x submitting chances. The final rank would be inaccurate due to this. If they achieved rank-3 in phase-II, then rank-3 to rank-6 are occupied, the original rank-4 would be rank-7. Is this behavior allowed?
I suggest the organizer @mike-dd check the duplication of the final report and codes of the competition. I think in this way, the mentioned behavior could be avoided.
Hi @mike-dd,
Resizing images once as part of preprocessing to get consistent image sizes is allowed, since this is a basic requirement for many CV techniques.
Augmentation (including resizing or cropping) for training purposes is only allowed on the training set, not the reference or query sets.
Above two statements seem to contradict each other. Just to be clear, is ‘Resizing images once on the reference sets for training purposes’ allowed?
And another question is about the statement from the paper,
This means that even if the dataset had a single query image and a single reference image, the
score of the image pair would be the same.
For one query-reference image pair A-B, one model that trained on all reference images and another model that trained only on one reference image(B) will have different score, which seems to break the above statement.
So, can I understand that above statement(same score for different dataset size) is applied in ‘inference’ stage? And in ‘training’ stage, we are just okay to train using all reference images?
Hi separate,
Yes to all of your questions.
Resizing images once on the reference sets for training purposes is allowed.
The requirement that an image score be unchanged by the dataset size is applicable at the inference step. It is ok to train on all reference images, but as you know participants are encouraged to use the training set for training purposes.
I hope this helps.
1 Like
Thanks for answering repeated questions! Really appreciate it.
Hi wenhaowang,
Resizing images, if applied once to an image as part of preprocessing, is not considered augmentation. Augmentation for training purposes is only allowed on the training set, not the reference or query sets.
From the competition rules:
“Augmenting” images refers to applying a transformation to an image to generate a new image, such as the manner in which the query set images were derived from the reference set images.
Again, participants are encouraged to use the training set for training purposes, and to use the ground truth labels as a test/validation set.