# Is the split between train, test and dev random?

**URL:** <https://community.drivendata.org/t/is-the-split-between-train-test-and-dev-random/4487>\
**Category:** Hateful Memes\
**Created:** [May 21, 2020, 3:37pm UTC](https://community.drivendata.org/t/is-the-split-between-train-test-and-dev-random/4487 "2020-05-21T15:37:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\
**Post date:** [May 21, 2020, 3:37pm UTC](https://community.drivendata.org/t/is-the-split-between-train-test-and-dev-random/4487/1 "2020-05-21T15:37:40Z")

</div>

Is it random or some heuristic was followed to do the split?

---

<div class="post-metadata">

**Author:** ![caseyalan](https://avatars.discourse-cdn.com/v4/letter/c/edb3f5/32.png) [@caseyalan](https://community.drivendata.org/u/caseyalan)\
**Post date:** [May 21, 2020, 3:53pm UTC](https://community.drivendata.org/t/is-the-split-between-train-test-and-dev-random/4487/2 "2020-05-21T15:53:08Z")

</div>

You can find information about the splits in section 2.4 of the [publication](https://arxiv.org/pdf/2005.04790.pdf) about the dataset.

---

<div class="post-metadata">

**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\
**Post date:** [May 21, 2020, 4:16pm UTC](https://community.drivendata.org/t/is-the-split-between-train-test-and-dev-random/4487/3 "2020-05-21T16:16:45Z")

</div>

Thanks, I did not have time yet to read the paper.

> Using the various phases outlined above and after further filtering to remove low-quality examples,  
> we end up with a dataset totalling exactly 10k memes. The dataset comprises five different types  
> of memes: multimodal hate, where benign confounders were found for both modalities, unimodal  
> hate where one or both modalities were already hateful on their own, benign image and benign text  
> confounders and finally random not-hateful examples.

> We construct a dev and test set from 5% and 10% of the data respectively, and set aside the rest  
> to serve as fine-tuning training data. The dev and test set are fully balanced, and are comprised of  
> memes using the following percentages: 40% multimodal hate, 10% unimodal hate, 20% benign text confounder, 20% benign image confounder, 10% random non-hateful.

So dev and test are balanced with that distribution, and train has an unknown distribution.