I want to know in the video similarity challenge, whether the queries in the phase 2 test set have been made and fixed? That means whether you may design some queries according to the submissions. For example, including the queries that top submission cannot deal well with? Or design some methods against the top submissions? Will you check the submitted code before the end of Phase 2?
If so, I think it may be unfair to some extend. Thanks.
Hi Wenhao,
We will not design the phase 2 query set based on phase 1 submissions. We do make some changes to the transforms between phases, but changes are modest and the main parameters of the phase 2 test set were designed before the challenge started.
The purpose of phase 2 is to ensure that submissions from phase 1 generalize. Challenge rules require using the same method on phases 1 and 2, and we won’t design a test set that would require changes. We just want to ensure that methods aren’t significantly overfitting to one set of transforms.
As for when we will check submitted code: I expect we will only check submitted code after phase 2 has completed.
Ed
Thanks very much. That helps lot!