Now that we’ve built multinomial logistic regression (a model that outputs a predicted class distribution $p_\theta(\mathbf x) \in \Delta^{c-1}$), we’ll now answer question of which loss function to use.
Why Not Accuracy as a Loss?
We use accuracy to evaluate our model. The natural question is whether we can also use it to train our model, that is, set $-1*(\text{acc})$ as the loss function and optimize the weights such that this loss is minimized (and equivalently, the accuracy is maximized).
Most optimization algorithms require that the loss function be differentiable (we’ll later see why). Accuracy is locally constant almost everywhere. On a finite dataset, the predicted class $\operatorname{argmax} p_\theta(\mathbf{x})$ changes only when the ordering of the class probabilities changes, which means that as we continuously vary $\theta$ to find a minimum, we don’t have a good response by the loss. Optimizers make no progress under these conditions since it’s hard to see which way we should tune $\theta$ to minimize the loss .
More fundamentally, accuracy is blind to the predicted probabilities. A model that assigns probability 0.51 to the correct class looks identical to one that assigns 0.99, both get credit since they classified the correct class as the majority; however, a good loss function should also be sensitive to how confident the model is and not just whether it’s right.
Loss: Negative Log Likelihood
We want a loss that rewards high probability on the correct class and penalizes low probability. A natural starting point from statistics is the likelihood of the dataset under our model, that is, the probability that our model would assign to observing the labels in the training set given the input examples.
Assuming the training examples are independent, the likelihood of the entire training set is the product of the probabilities assigned to the correct label for each example:
$$ \mathbf P(\mathscr D_\text{train} \mid p_\theta) = \prod_{i \in I_\text{train}} p_\theta(\mathbf x_i)_{y_i} $$
Here, $p_\theta(\mathbf x_i)_{y_i}$ denotes the probability that the model assigns to the true class $y_i$ for input $\mathbf x_i$. A model that consistently assigns high probability to the correct class will therefore have a high likelihood.
Our goal is to maximize this. Since $\log$ is strictly monotone, maximizing the likelihood is equivalent to maximizing the log likelihood:
$$ \mathrm{LL}(\mathscr D_\text{train}; \theta) = \sum_{i \in I_\text{train}} \log p_\theta(\mathbf x_i)_{y_i} $$
Since we want a minimization problem (to be consistent with the rest of our framework), we can negate it. Averaging over the dataset in the usual way, we arrive at the Negative Log Likelihood (NLL) function:
$$ \ell_\mathrm{NLL}(\mathscr D_\text{train}; \theta) = -\frac{1}{n_\text{train}} \sum_{i \in I_\text{train}} \log p_\theta(\mathbf x_i)_{y_i} $$
The per-example loss is simply:
$$ \ell_\mathrm{NLL}(\mathbf x_i, y_i; \theta) = -\log p_\theta(\mathbf x_i)_{y_i} $$
We can see that it penalizes the model in proportion to the surprise of the true label under the predicted distribution. If the model is confident and correct, $p_\theta(\mathbf x_i)_{y_i} \approx 1$ and the loss is near zero. If the model is confident and wrong, $p_\theta(\mathbf x_i)_{y_i} \approx 0$ and the loss blows up.
Note that $\ell_\mathrm{NLL}$ is always non-negative and equals zero only if the model assigns probability 1 to the correct class for every example.
In information theory, this formula computes the cross-entropy between the true label distribution and the predicted distribution. See Christopher Olah’s Visual Information Theory if you are interested.