The ID 36804 is labeled as 1, meanwhile the same meme is 0 at 34816 (and multiple other instances). There are more examples like that.
Is that an error, or just different annotators shown a slightly different version?
Thanks for pointing this out @slawekbiel - there are indeed a few of these in the dataset, as a result of using different annotators. For now you can just treat them as given - we will update them in a future release.
Hi there, has the future release happened already? We’re seeing a fair amount of these issues. In some cases like slawekbiel points to, the “meanwhile at the isis strip club” meme appears 44 times, with most annotators showing agreement. However there are other cases like the “burning things that don’t mean anything to you” meme where there are only two training examples that are labeled differently and it’s not clear what the “true” label should be. What do you recommend here?