Definition of Log Loss

Warm Up: Machine Learning with a Heart
1

I’m using Azure ML Studio to work on the data set. This provides two different log loss metrics for tuning hyperparameters - Average Log Loss and Train Log Loss. Which of these should I choose to match to the measure of Log Loss used in the submission assessment?

I’ve noticed that if I optimize for Average Log Loss, I get better results on submission than for Train Log Loss, but that the values I get for Average Log Loss based on the training data are an order of magnitude smaller than the log loss I get in the submission grade, and those I get for Train Log Loss are an order of magnitude larger, so I’m confused.

Any views?

2

Per the docs at: https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/evaluate-model

  • Average log loss is a single score used to express the penalty for wrong results. It is calculated as the difference between two probability distributions – the true one, and the one in the model.
  • Training log loss is a single score that represents the advantage of the classifier over a random prediction. The log loss measures the uncertainty of your model by comparing the probabilities it outputs to the known values (ground truth) in the labels. You want to minimize log loss for the model as a whole.