Small feedback for the organizers

Power Laws
1

Thanks for the challenging tasks and here are some thoughts on how to improve in the future:

  1. Re all competitions: please make submissions accessible by links. It is very inconvenient not to be able to download old submissions (and even do not have access to the filenames). Comments are great but some researchers prefer to make them via filenames. And current system just doubles the amount of work.
  2. Forecast. Lets put aside the craziness of the task with all the different timestamps and periods (basically, it was three competitions in one). I don’t think that there was enough data for 15 min timeframe for the accurate prediction. Also, it was really hard to come up with a validation strategy which correlated with LB (at least I failed). Good luck checking winners solutions for the leaks from future!
  3. Anomalies. It is clear that designing unsupervised learning challenge is hard and I did like your report track idea a lot. But everything else was just a waste of time, i.e trying to guess your algorithm with 2 attempts per day. It would’ve made sense to make it supervised in case you wanted to automate you anomaly detection. Unless it is already automated. As a result, the competition turned into LB probing and became supervised with a very slow convergence.
  4. Demand. Great task (reminded me algotrading), cool but slow simulator, everything else was great except for the computation restrictions - was it really necessary to perform it on a calculator? One core and no GPU in 2018, seriously?
5 Likes
2

I completely agree with Pavel.

Also, I would like to add that cheaters could take information from future in Prediction competition and type any score they want in Demand competition (also, I don’t think that Anomalies is competition at all). Thus, we could not rely on leaderboard score and should compete blindly.

1 Like
3

Last but not the least (probably even the most important thing): it turns out that selecting submissions for private LB is not implemented in this platform. Which basically means that there is no meaning in private LB since every submission counts and public overfitting is not penalized. I would really like to hear the reasoning behind that decision.

3 Likes