We now turn to the special case $c = 2$: binary classification. The two classes are $\mathscr Y = \{0, 1\}$, typically called the negative and positive class. Rather than a whole new model, binary logistic regression is just multinomial logistic regression specialized to $c = 2$, and this simplification will make it easier to see what’s going on.
We’ll also see the true meaning of logits and why the pre-softmax values get that name.
From Softmax to Sigmoid
With $c = 2$, the softmax output is a vector $(p_0, p_1) \in \Delta^1$. Since $p_0 + p_1 = 1$, the entire distribution can be determined by $p_1$ alone, and thus, it suffices to model just the probability of the positive class.
Let $\mathbf z = \mathbf x^T W + \mathbf b \in \mathbb{R}^2$ be the logit vector with components $z_0$ and $z_1$. Applying softmax and dividing numerator and denominator by $\exp(z_1)$:
$$ p_1 = \frac{\exp(z_1)}{\exp(z_0) + \exp(z_1)} = \frac{1}{1 + \exp(z_0 - z_1)}. $$
Only the difference $z_0 - z_1$ matters.
Why only the difference matters
Softmax is translation-invariant, so subtracting any constant $m$ from all logits leaves the output unchanged, since the $\exp(-m)$ factors cancel in numerator and denominator. Recall that we showed this explicitly in the numerical stability discussion in the previous page. With two logits, this means one degree of freedom is always redundant, that is, only their difference carries information.
We can therefore replace both logits with a single scalar $z := z_1 - z_0$, giving:
$$ p_1 = \frac{1}{1 + \exp(-z)} =: \sigma(z). $$
The function $\sigma: \mathbb{R} \to (0,1)$ is the sigmoid function:
$$ \sigma(z) = \frac{1}{1 + \exp(-z)}. $$
It maps any real number to a probability. As $z \to +\infty$, $\sigma(z) \to 1$; as $z \to -\infty$, $\sigma(z) \to 0$; and $\sigma(0) = \tfrac{1}{2}$.
The Model
Since the full distribution is determined by $p_1 = \sigma(z)$, our parameter space gets much simpler. Instead of a weight matrix $W \in \mathbb{R}^{d \times 2}$ and bias $\mathbf b \in \mathbb{R}^2$, we only need:
- a weight vector $\mathbf w \in \mathbb{R}^d$, and
- a bias scalar $b \in \mathbb{R}$.
The model is:
$$ p_\theta(\mathbf x) = \sigma(\mathbf x^T \mathbf w + b), $$
where $\theta = (\mathbf w, b)$ are the only parameters and the output is the predicted probability of the positive class.
Logits and Log-Odds
We can finally explain the term logit. The odds of the positive class under our model are:
$$ \frac{p_1}{p_0} = \frac{p_1}{1 - p_1}. $$
Taking the log and substituting $p_1 = \sigma(z)$:
$$ \log \frac{p_1}{1 - p_1} = \log \frac{\sigma(z)}{1 - \sigma(z)} = z. $$
Expand to see why this equals z
We have $1 - \sigma(z) = \sigma(-z) = \frac{\exp(-z)}{1 + \exp(-z)}$, so:
$$ \frac{\sigma(z)}{1 - \sigma(z)} = \frac{\frac{1}{1+\exp(-z)}}{\frac{\exp(-z)}{1+\exp(-z)}} = \frac{1}{\exp(-z)} = \exp(z). $$
Therefore:
$$ \log \frac{p_1}{1 - p_1} = \log \exp(z) = z. $$
So $z = \mathbf x^T \mathbf w + b$ is exactly the log-odds of the positive class (the logarithm of the ratio of probabilities). The function that maps a probability $p \in (0,1)$ to its log-odds $\log \frac{p}{1-p}$ is called the logit function, and $\sigma$ is its inverse. This is where the term logit comes from: the pre-sigmoid value is the logit of the predicted probability.1
Loss: Binary Cross-Entropy
With $c = 2$, NLL also simplifies. The per-example loss is:
$$ \ell_\mathrm{NLL}(\mathbf x_i, y_i; \theta) = -\log p_\theta(\mathbf x_i)_{y_i}. $$
Since $y_i \in \{0, 1\}$ and $p_0 = 1 - p_1 = 1 - \sigma(z_i)$, we can write this without cases:
$$ \ell_\mathrm{BCE}(\mathbf x_i, y_i; \theta) = -y_i \log \sigma(z_i) - (1 - y_i)\log(1 - \sigma(z_i)), $$
where $z_i = \mathbf x_i^T \mathbf w + b$.
This is the binary cross-entropy (BCE) loss. When $y_i = 1$ only the first term survives; when $y_i = 0$ only the second does. Averaged over the dataset:
$$ \ell_\mathrm{BCE}(\mathscr D; \theta) = -\frac{1}{N} \sum_{i \in I} \left[ y_i \log \sigma(z_i) + (1 - y_i)\log(1 - \sigma(z_i)) \right]. $$
In Practice
Binary logistic regression introduces no new practical considerations beyond what we’ve already covered: train-test split, class imbalance checks, and numerical stability. The BCE loss is implemented in PyTorch as torch.nn.BCEWithLogitsLoss, which takes raw logits $z_i$ rather than sigmoid outputs and is numerically stable.
-
In the multinomial setting, the pre-softmax values $z_j$ generalize this idea: they are log-probabilities up to a normalizing constant, which is why the name logits carries over. ↩