# Pseudo labeling **URL:** \ **Category:** Hateful Memes\ **Created:** [October 27, 2020, 10:34pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412 "2020-10-27T22:34:17Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 27, 2020, 10:34pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/1 "2020-10-27T22:34:17Z")
Is pseudo labeling is allowed, or does it violate the competition rules? I would expect it to be allowed, but just want to be sure of it. Thank you, Nuno ---
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 28, 2020, 7:21am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/2 "2020-10-28T07:21:43Z")
@glipstein AS the time for the end is running out, can you let me know if the competition rules allow for the use of pseudo labeling? Thanks ---
**Author:** ![glipstein](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/glipstein/32/920_2.png) [@glipstein](https://community.drivendata.org/u/glipstein)\ **Post date:** [October 28, 2020, 8:07pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/3 "2020-10-28T20:07:06Z")
Hi @ngcferreira - Thanks for reaching out. It’s not clear exactly what you’re intending, but we’d encourage you to check out [this thread](http://community.drivendata.org/t/external-data-annotation/4447) and the [Dataset License Agreement](https://www.drivendata.org/competitions/70/hateful-memes-phase-2/page/269/) which addresses annotating the dataset. ---
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 28, 2020, 9:46pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/4 "2020-10-28T21:46:54Z")
hi @glipstein, Thank you for your reply, but that thread doesn’t really cover this particular case. I wanted to know if it is allowed to use my model’s predictions on the test set as training data. So in fact the test data gets labeled automatically by my model (I use my best submission as training data). No manual process is involved. This is a semi-supervised machine learning technique, which can improve the performance of a model. From the rules it isn’t clear if this is allowed or not, so I would appreciate some guidance. Thank you, Nuno ---
**Author:** ![burebista](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@burebista](https://community.drivendata.org/u/burebista)\ **Post date:** [October 28, 2020, 10:25pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/5 "2020-10-28T22:25:19Z")
Uploading the predictions and submitting to the leaderboard in the first place involves a manual process. Plus I think leaderboard-supervised sounds way cooler than semi-supervised 😃 ---
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 28, 2020, 10:37pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/6 "2020-10-28T22:37:48Z")
It’s a standard ML technique, used when you don’t have enough data, which is the case here. I just want to know if that’s allowed or not, to decide what to do for my 3rd submission. Don’t want to waste it on something, that might not be allowed 😉 Still have a few other ideas on how to improve. ---
**Author:** ![bahushruth](https://avatars.discourse-cdn.com/v4/letter/b/dec6dc/32.png) [@bahushruth](https://community.drivendata.org/u/bahushruth)\ **Post date:** [October 28, 2020, 11:14pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/7 "2020-10-28T23:14:49Z")
Achieving the state of the art result by training the model on the test set labelled by outsourced annotators sounds cooler in my opinion 🤔 ---
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 29, 2020, 8:31am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/8 "2020-10-29T08:31:39Z")
@bahushruth Here is an article where google achieves state of the art results in a different problem with a mix of pseudo labeling an other techniques: [https://arxiv.org/pdf/2001.07685.pdf](https://arxiv.org/pdf/2001.07685.pdf) And note that independently on knowing the private score of the training set, this technique is still useful. Anyway as there isn’t an answer from the organizers @glipstein , I will stick to my current approach, which does not use pseudo labeling. 🙂 ---
**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\ **Post date:** [October 29, 2020, 5:30pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/9 "2020-10-29T17:30:22Z")
I think a clear response is needed here @glipstein instead of pointing to other threads. Pseudolabelling is a standard ML technique as @ngcferreira said. Taking the predictions of a model on unlabelled data and using them for training is not a modification of the dataset from my point of view. For example if I use data augmentation and flip the images during training I’m also modifying the dataset? ---
**Author:** ![glipstein](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/glipstein/32/920_2.png) [@glipstein](https://community.drivendata.org/u/glipstein)\ **Post date:** [October 30, 2020, 12:04am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/10 "2020-10-30T00:04:40Z")
@ngcferreira Thanks for your patience. Labeling the test data through semi-supervised learning is still not in compliance with the [Dataset License Agreement](https://www.drivendata.org/competitions/70/hateful-memes-phase-2/page/269/). You may perform semi-supervised learning on other data (including train and dev), but not the test set, which should really be treated as “unseen” new data. ---
**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\ **Post date:** [October 30, 2020, 7:00am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/11 "2020-10-30T07:00:43Z")
That’s sad because we have been training the whole month using pseudolabel on test\_seen, I encourage to make rules clearer the next time. In the other hand it’s better to know now, so thanks @ngcferreira for asking. ---
**Author:** ![ngcferreira](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@ngcferreira](https://community.drivendata.org/u/ngcferreira)\ **Post date:** [October 30, 2020, 9:11am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/12 "2020-10-30T09:11:57Z")
@glipstein Thank for clarifying this, which was not clear in the rules. Please next time make the rules more clear. @ironbar I’m sorry for the time you have put into pseudo-labeling. By the way was it working for you? I did a fast try, and it was only improving slightly my results on the validation set. Of course I do not know how much it would improve the test set results, but from my experience so far, the difference between validation and test score is not that big. ---
**Author:** ![james005](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@james005](https://community.drivendata.org/u/james005)\ **Post date:** [October 30, 2020, 5:17pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/13 "2020-10-30T17:17:07Z")
@glipstein and how about data augmentation on the train and dev sets, is it allowed or not? ---
**Author:** ![glipstein](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/glipstein/32/920_2.png) [@glipstein](https://community.drivendata.org/u/glipstein)\ **Post date:** [October 31, 2020, 3:42pm UTC](https://community.drivendata.org/t/pseudo-labeling/5412/14 "2020-10-31T15:42:56Z")
@james005 Please do not modify the test set, but you may use the training and dev set. Good luck! ---
**Author:** ![bahushruth](https://avatars.discourse-cdn.com/v4/letter/b/dec6dc/32.png) [@bahushruth](https://community.drivendata.org/u/bahushruth)\ **Post date:** [November 1, 2020, 12:02am UTC](https://community.drivendata.org/t/pseudo-labeling/5412/15 "2020-11-01T00:02:35Z")
> [@glipstein](#): > > do not modify the test set, but you may use the training a Hope this goes for test seen as well.