# Just Making Sure Data Is Correct

**URL:** <https://community.drivendata.org/t/just-making-sure-data-is-correct/2624>\
**Category:** General Discussion\
**Created:** [September 5, 2018, 4:07pm UTC](https://community.drivendata.org/t/just-making-sure-data-is-correct/2624 "2018-09-05T16:07:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![AmirH](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AmirH](https://community.drivendata.org/u/AmirH)\
**Post date:** [September 5, 2018, 4:07pm UTC](https://community.drivendata.org/t/just-making-sure-data-is-correct/2624/1 "2018-09-05T16:07:42Z")

</div>

Hi!  
Don’t want to be annoying or anything, just making sure the data is correct.  
I spotted some places in which a specific hour has 10X the consumption of all other hours of that specific day + series.  
e.g:  
RowID 108926  
All hours except 2PM of 18/8/2016 series 103634 have mean comsumption of 45k  
2PM has a VERY high consumption of 440K  
which is ~10X more than all other hours.

sometimes competitions have some kind of data issue that is found too late and a lot of competitor’s hours are gone.  
Could you please just verify and let us know it’s OK and not some kind of glitch in the matrix?

Thanks!

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [September 5, 2018, 6:10pm UTC](https://community.drivendata.org/t/just-making-sure-data-is-correct/2624/2 "2018-09-05T18:10:08Z")

</div>

Hi @AmirH, you might want to check your preprocessing. The values you cite for that series look as expected:

 ![35%20AM](https://canada1.discourse-cdn.com/flex028/uploads/drivendata1/original/1X/f8e04615b4b82342df3fa3255845f7ade70d0ee4.png)

---

<div class="post-metadata">

**Author:** ![AmirH](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AmirH](https://community.drivendata.org/u/AmirH)\
**Post date:** [September 5, 2018, 7:03pm UTC](https://community.drivendata.org/t/just-making-sure-data-is-correct/2624/3 "2018-09-05T19:03:27Z")

</div>

Hi Bull,  
Thanks for your response.

The picture actually does show the issue i was talking about, the first part out of 4 of your plot shows the very low values with one single peak, that’s the data i meant.  
 ![image](https://canada1.discourse-cdn.com/flex028/uploads/drivendata1/original/1X/504590abdc16bbed2e40e16993ccbeb7f69c68cd.png)

It just looked very odd that one specific hour of the day would have 10X more than all other hours

---

<div class="post-metadata">

**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\
**Post date:** [September 5, 2018, 7:27pm UTC](https://community.drivendata.org/t/just-making-sure-data-is-correct/2624/4 "2018-09-05T19:27:54Z")

</div>

These spikes are reflections of the data as collected. As with all real data, it is possible that some outliers are anomalies and some are real data. Effective methods will have to take this into consideration. 👍