# Question About NMAE metric, numbers seem off? **URL:** \ **Category:** Cold Start Energy Forecasting\ **Created:** [September 11, 2018, 5:28pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652 "2018-09-11T17:28:40Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![AmirH](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AmirH](https://community.drivendata.org/u/AmirH)\ **Post date:** [September 11, 2018, 5:28pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/1 "2018-09-11T17:28:40Z")
Hi, I’m not sure i understood how the metric works. E.G: Hourly True value: 50 Prediction: 40 nmae = abs(40-50)\*((24/24)/50) Which results in 0.2 This seems reasonable If I multiply these values and treat them as Daily values: Daily True Value: 1200 Prediction: 960 nmae = abs(960-1200)\*((24/7)/1200) Which results in 0.6857143 is this correct? The daily error seems off to me. ---
**Author:** ![c3josh](https://avatars.discourse-cdn.com/v4/letter/c/c89c15/32.png) [@c3josh](https://community.drivendata.org/u/c3josh)\ **Post date:** [September 11, 2018, 8:56pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/2 "2018-09-11T20:56:37Z")
This is how I understand the metric to work. It become even more odd when you look at weekly model errors. My errors are in the order of 0.9 - 2.5 for many of the series in my test sets. Thus, we are essentially making week errors 12 worse than they should be!! To me, it would have made more sense to weight these in a way such that the weekly errors are 12x more influential, not 12x bigger. This could have been done by taking an equal number of samples of hourly, daily and weekly, 1x, 24/7x and 12x respectively. It would be interesting to know why this method was not used, in favour of the current NMAE. ---
**Author:** ![AmirH](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AmirH](https://community.drivendata.org/u/AmirH)\ **Post date:** [September 11, 2018, 10:02pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/3 "2018-09-11T22:02:37Z")
I think this somehow works different, It’s quite weird that the NMAE in LB is around 0.4, when indeed those weekly errors are penalized so harshly. I’m pretty sure I got something wrong in the way i understood it. ---
**Author:** ![adilism](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@adilism](https://community.drivendata.org/u/adilism)\ **Post date:** [September 12, 2018, 12:52am UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/4 "2018-09-12T00:52:04Z")
I think the idea is to make each series equally important. In your example, taking into account that there are 24 predictions for hourly and 7 for daily, `mae1= 24*10*(24/24)/50=4.8` and `mae2=7*(24*10)*(24/7)/(24*50)=4.8` ---
**Author:** ![AmirH](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AmirH](https://community.drivendata.org/u/AmirH)\ **Post date:** [September 12, 2018, 7:27am UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/5 "2018-09-12T07:27:34Z")
Hi @adilism, I’m not sure that is correct. 1 - the value you got is 4.8, where the “correct” metric in this case should be 0.2 (40/50 is 20% error) 2 - in your calculation you simply multiplied both sides by 24 or 7, simply removing these gets you to the same result. 3 - simply calculating (abs(true-pred) / true) which is a regular MAE, gives this dataset 0.2 MAE, which seems to me the logical result. @bull Could you please give some insight on this? What is the expected NMAE on this set of data? ![image](https://canada1.discourse-cdn.com/flex028/uploads/drivendata1/original/1X/153a92b780cc3fa0650f4c5839cc034729c02992.png) Thanks!! ---
**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\ **Post date:** [September 12, 2018, 3:52pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/6 "2018-09-12T15:52:37Z")
As far as I understand you don’t use the true hourly value. Instead you get the average value of all the hours in the prediction (the 24 hours) ---
**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\ **Post date:** [September 12, 2018, 3:55pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/7 "2018-09-12T15:55:11Z")
Probably the best way to clarify the doubts is releasing the code implementation of the metric. ---
**Author:** ![ironbar](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/ironbar/32/671_2.png) [@ironbar](https://community.drivendata.org/u/ironbar)\ **Post date:** [September 19, 2018, 2:36pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/8 "2018-09-19T14:36:50Z")
Please @bull the metric needs to be clarified, it is essential. ---
**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\ **Post date:** [September 19, 2018, 3:25pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/9 "2018-09-19T15:25:35Z")
The calculation is made directly on the same format as the submission format for the competition. This means that hourly is compared to hourly, daily to daily, and weekly to weekly. The aggregated values for consumption in daily and weekly predictions are sums, not averages. The weights, as mentioned by @adilism are used to make sure each series is equally important when we take the overall mean across series, even if there are fewer predictions made for that particular series. ---
**Author:** ![c3josh](https://avatars.discourse-cdn.com/v4/letter/c/c89c15/32.png) [@c3josh](https://community.drivendata.org/u/c3josh)\ **Post date:** [October 8, 2018, 3:41pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/10 "2018-10-08T15:41:42Z")
What happens if the true mean consumption is zero? Can we assume that this will not happen for any of the series\_ids in the test sets? There are potentially training examples with zero means over a whole day. This was just bad luck on my part selecting this day to test on and div/0. ---
**Author:** ![c3josh](https://avatars.discourse-cdn.com/v4/letter/c/c89c15/32.png) [@c3josh](https://community.drivendata.org/u/c3josh)\ **Post date:** [October 9, 2018, 4:17pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/11 "2018-10-09T16:17:41Z")
@bull - can you confirm this please - just to be super clear ---
**Author:** ![bull](https://yyz2.discourse-cdn.com/flex028/user_avatar/community.drivendata.org/bull/32/8_2.png) [@bull](https://community.drivendata.org/u/bull)\ **Post date:** [October 9, 2018, 6:03pm UTC](https://community.drivendata.org/t/question-about-nmae-metric-numbers-seem-off/2652/12 "2018-10-09T18:03:07Z")
If actual mean is zero, it will be clipped to a very small number instead of dividing by zero. However, there are no examples where the value is exactly zero in the test set.