# Why submission\_format.csv and test\_2h.csv have different lengh?

**URL:** <https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433>\
**Category:** From Fog Nets to Neural Nets\
**Created:** [March 2, 2016, 11:20am UTC](https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433 "2016-03-02T11:20:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Littus](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/littus/32/290_2.png) [@Littus](https://community.drivendata.org/u/Littus)\
**Post date:** [March 2, 2016, 11:20am UTC](https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433/1 "2016-03-02T11:20:35Z")

</div>

And time stamps, actually.

Could you explain, please?

Edit: Ok, I get it — no micro data.

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [March 2, 2016, 3:30pm UTC](https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433/2 "2016-03-02T15:30:14Z")

</div>

Yep, that’s right–sorry if it is a little confusing.

For other folks who may be wondering the same thing, the easiest way to deal with this is a left outer join on the micro data. If you’re using Python, you can see an example of how to do this in `pandas` at the [end of our benchmark blogpost](http://blog.drivendata.org/2016/03/01/fogwater-benchmark/#what-should-our-submission-look-like-).

---

<div class="post-metadata">

**Author:** ![lucasribeiroabreu](https://avatars.discourse-cdn.com/v4/letter/l/779978/32.png) [@lucasribeiroabreu](https://community.drivendata.org/u/lucasribeiroabreu)\
**Post date:** [March 2, 2016, 10:38pm UTC](https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433/3 "2016-03-02T22:38:17Z")

</div>

OK, so you are predicting a bunch of NaNs?

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [March 2, 2016, 10:45pm UTC](https://community.drivendata.org/t/why-submission-format-csv-and-test-2h-csv-have-different-lengh/433/4 "2016-03-02T22:45:26Z")

</div>

In the blog post model, those NaNs get imputed using a [simple imputation strategy](http://blog.drivendata.org/2016/03/01/fogwater-benchmark/#building-a-model-pipeline). Competitors have a number of options for imputing those `NaN`s:

- Basic strategies like the example (e.g., mean imputation)
- Create forecasts using the existing microclimate data
- Use the available macroclimate data during those time periods
- Any other strategy you can think of

Part of the challenge is to answer the question: how well can we predict water yield if we don’t have a weather station right next to the nets. Or, to put it another way, can we predict water yield accurately from understanding the larger meteorological picture?