More information about hidden test set

Hateful Memes
1

I believe having more information about the hidden test set would be helpful. I have some questions but maybe you can think of more:

  • Does the hidden test set comes from the same distribution as the public train/dev/test data? Let’s say 20k images were collected for the challenge, and they were randomly split between train, dev, public test and hidden test…
  • Does the hidden test set comes from the same timeline as the public train/dev/test data? Maybe the hidden test set will only have memes created in the future, for example in next September…
  • How many images does it have? The more the images the less uncertainty on the score.

Thanks

1 Like
2

Hi @ironbar. The hidden test test––what an interesting topic!

I can let you know––from us here at DrivenData and our partners at Facebook––that more information will provided in due course. But for now you’ll just need to wait!

I recommend going ahead and keeping this thread alive to accumulate questions that you and the community can investigate when the time is right.

1 Like