We shall now consider the case where the target space $\mathscr Y$ is finite. Recall from the introduction that this is a classification problem. Unless stated otherwise, we let $\mathscr Y = \{0, \dotsc, c-1\} =: [c]$, where $c$ is the number of classes.
Metric: Accuracy
Before building the model, we need to decide how to measure its performance. The natural choice for a classification is accuracy: the proportion of examples for which the modelâs predicted class matches the true label.
$$ \operatorname{acc}(f,\mathscr D) = \frac{1}{N} \sum_{i=1}^{N} \begin{cases} 1, & \text{if } f(\mathbf{x}_i)=y_i,\\ 0, & \text{otherwise.} \end{cases} $$
Why Model Class Distributions?
If accuracy only cares about the predicted class, why not have our model directly output a class $j \in [c]$ for each input $\mathbf x$? Why are we going through the trouble of outputting probabilities?
There are two reasons.
- Richer output. Predicted class probabilities carry more information than a hard prediction. Knowing that a model assigns probability 0.51 to class 0 versus 0.99 is quite telling. In a way, probabilities can encode confidence as well 1.
- Differentiability. The target space $[c]$ is discrete. A model $f: \mathscr X \to [c]$ cannot be continuously differentiated, which rules out gradient-based optimization (which youâll see later is the backbone of essentially all of deep learning). By instead having our model output a distribution over $[c]$, we have something that is continuous and differentiable, which is much more useful to us.
Thus, we should model $\mathbf P(Y = j \mid X = \mathbf x)$ for each class $j$. Weâll also see later that accuracy alone also canât serve as a loss function.
The Probability Simplex
Since $\mathscr Y = [c]$ is a finite set, a probability distribution over it is a categorical distribution, specified by $c$ values $p_0, \dotsc, p_{c-1}$ satisfying:
- $0 \le p_j \le 1$ for all $j$, and
- $\sum_{j=0}^{c-1} p_j = 1$.
Geometrically, the set of valid probability vectors forms the standard $(c-1)$-simplex:
$$ \Delta^{c-1} = \left\{(p_0, \dotsc, p_{c-1}) \in \mathbb{R}^c_{\ge 0} : \sum_{j=0}^{c-1} p_j = 1\right\}. $$
We call this the probability $(c-1)$-simplex and write $\mathscr P([c]) \cong \Delta^{c-1}$.
The Softmax Function
We want our model to be an affine transformation. Since the output should be a distribution over $c$ classes, the model has a $c$-dimensional output. The parameters are:
- a weight matrix $W \in \mathbb{R}^{d \times c}$, and
- a bias vector $\mathbf b \in \mathbb{R}^c$.
The map $\mathbf x \mapsto \mathbf x^T W + \mathbf b$ sends inputs to arbitrary vectors in $\mathbb{R}^c$, not necessarily valid probability distributions. To project into $\Delta^{c-1}$, we apply the softmax function:
$$ \operatorname{softmax}(\mathbf z)_j = \frac{\exp(z_j)}{\sum_{j'=0}^{c-1} \exp(z_{j'})}, \qquad j \in [c]. $$
All outputs are positive and sum to 1, so softmax always produces a valid distribution. The intermediate values $z_j$ fed into softmax are called logits.2
Multinomial Logistic Regression
If we compose the affine transformation with softmax, we get whatâs called multinomial logistic regression:
$$ \mathbb{R}^d \xrightarrow{\mathbf x \mapsto \mathbf x^T W + \mathbf b} \mathbb{R}^c \xrightarrow{\operatorname{softmax}} \Delta^{c-1}. $$
We can write the output as $p_\theta(\mathbf x)$, with parameters $\theta = (W, \mathbf b)$, to emphasize that it is a predicted probability distribution. Concretely, $p_\theta(\mathbf x)_j$ estimates $\mathbf P(Y = j \mid X = \mathbf x)$.
But what loss function should we use? And why canât we use accuracy here? Weâll see the answers to these questions in the next note.
In Practice
The same train-test split discipline from linear regression applies here without modification. We need to fit $\theta$ on $\mathscr D_\text{train}$, and report accuracy on $\mathscr D_\text{test}$.
We have one new consideration though: if the classes in $\mathscr D$ are not equally represented, a model that always predicts the majority class can achieve high accuracy without learning anything. This means that before training, itâs important to always check the class distribution of your dataset, since if itâs skewed, raw accuracy is a very misleading metric.3
Numerical Stability
Implementing mathematical expression in code with floating point arithmetic may seem simple, but we canât follow theory directly. We need to worry about maintaining called numerical stability. Floating point numbers have finite precision and a finite representable range, so certain mathematically valid computations may be inaccurate and can simply overflow to inf or underflow to zero.
The softmax function is a classic example. For large $z_j$, $\exp(z_j)$ overflows to inf before the normalization can bring it back down. For very negative $z_j$, $\exp(z_j)$ underflows to 0, so small but nonzero probabilities vanish entirely.
The fix is to subtract a constant $m$ from all logits before exponentiating doesnât change the output.
Expand to see why this works
Let $m \in \mathbb{R}$ be any constant. Then:
$$ \frac{\exp(z_j - m)}{\sum_{j'=0}^{c-1}\exp(z_{j'} - m)} = \frac{\exp(z_j)\cdot\exp(-m)}{\sum_{j'=0}^{c-1}\exp(z_{j'})\cdot\exp(-m)} = \frac{\exp(z_j)}{\sum_{j'=0}^{c-1}\exp(z_{j'})}. $$
The $\exp(-m)$ factors cancel. So softmax is invariant to translation of the logits.
The standard choice is $m = \max_{j} z_j$. With this, the largest exponentiated value is $\exp(0) = 1$, and all others are in $(0, 1]$. This prevents overflow entirely, and keeps underflow confined to genuinely negligible probabilities. The numerically stable softmax is therefore:
$$ \operatorname{softmax}(\mathbf z)_j = \frac{\exp(z_j - \max_{j'} z_{j'})}{\sum_{j''=0}^{c-1}\exp(z_{j''} - \max_{j'} z_{j'})}. $$
The problem of finding an equivalent form of a computation that avoids numerical hazards is a typical deep learning obstacle.
Fortunately though, you wouldnât usually implement softmax by hand; people have already created libraries that solve this particular problem internally (for example, PyTorchâs F.softmax function), but understanding the issue is important when you do write numerical code from scratch, and for interpreting the occasional nan or inf that appears during training.
-
Here, I donât mean confidence in the statistical sense, but rather in the colloquial sense. ↩
-
The term logit comes from the binary classification setting, where the pre-sigmoid value is the log-odds of the positive class. Weâll return to this when we cover binary logistic regression. ↩
-
More careful metrics for imbalanced settings will be discussed in a later section. ↩