We just noticed in the data download page (for Red Teams) that there is a new dataset for Track A (not sure since when it has been there). The old dataset is still there and we have noticed that their characteristics are quite different (in ways I won’t mention here). Do attacks have to be developed with respect to the new dataset or are attacks with respect to the old dataset also valid?
Apologies—we did not post an announcement for red teams regarding the inclusion of the new dataset. I have added information to the announcements page: U.S. link | U.K. link.
For the attack period, we will provide red teams access to the held-out evaluation dataset that blue team solutions were trained on. As the announcement mentions, that held-out evaluation data is more closely similar to the new dataset.