Error in formula AMT

Power Laws
1

AMT = $\frac{1}{N} \sum_{n=1}^{N} \frac{M_{battery}}{M_{no battery}}

It seems to me that this formula is wrong, when the price is negative, that is to say, there is some energy
sold.
it’s better to take \sum_{n=1}^{N}M_{battery} / \sum_{n=1}^{N}M_{no_battery}.
Do you agree ?
@bull

2

The metric works as intended.

In this formula, N is the number of simulation runs. (The total money spent for a simulation is summed during the evaluation at each timestep). Different runs may have different lengths and be worth more or less money. However, the goal is to weight each simulation run equally regardless of the length of time simulated. For each simulation, we take the ratio of the battery scenario to the scenario if there is no battery. A value less than one means less money is spent with the battery. A value greater than one means that more money is spent when using the battery.

While technically possible, it’s very unlikely to be able to net make money energy across any given simulation (and therefore have a negative ratio). However, the outcome with the metric is still as intended since the goal is to minimize the metric.

Hope that clarifies the metric!

3

Thanks for your answer.

If negative, the ratio should be the inverse, but as you mention, this does not happen.

4

Hi, There are negative costs on site_id 7 even when not using the battery.

So in those cases with the current metric the more the cost saving the worst the metric. This should be corrected.

5

Yes, I also agree with @ironbar. In site id 7 period 1 and 2 there is a negative cost, which means a profit. When using a battery we can increase the negative value (increase the profit), that results a number higher than 1 in evaluation metric. Here using a battery, profit was increased but it interprets in a wrong way.

1 Like
6

Yes, this is why I asked to inverse the ratio, when the cost is negative, for instance.

I had not seen the site 7 yet when saying “as you mention, this does not happen” in my previous post.
But it happens and the current metric makes no sense when electricity is sold.

Or you could take 1 - (cost_with - cost_without)/cost_without when cost_ are negative.

7

Moreover with the current metric it’s possible to get absurd scores. Just look at the leaderboard and enjoy XD

1 Like
8

Thanks for the discussion here @ironbar @DelphineM @Y4jk

We’ll be releasing an updated metric for this competition in the next few days along with a new submission format.

9

Hi @ironbar @DelphineM @Y4jk,

Thanks again for the discussion here. We’ve updated the metric and removed the old scores. Feel free to resubmit after re-running based on the latest code in the repo.

Here’s the official announcement:
https://www.drivendata.org/competitions/53/optimize-photovoltaic-battery/announcements/

10

Thank you for your new formula.