Logistic Regression

Logistic regression models the probability that $Y$ belongs to a particular category

We are attempting to model the relationship between $p (X) = P r (Y = 1 | X)$ and $X$
Logistic Function:

p (X) = \frac{e^{β_{0} + β_{1} X}}{1 + e^{β_{0} + β_{1} X}}

Fit the model using maximum likelihood

likelihood function:

l (β_{0}, β_{1}) = \prod_{i : y_{i} = 1} p (x_{i}) \prod_{i^{'} : y_{i^{'}} = 0} (1 - p (x_{i^{'}}))

Estimates for $\hat{β_{0}}$ and $\hat{β_{1}}$ are chosen to maximize the likelihood function

p (X) = \frac{e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}}{1 + e^{β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p}}}

Use maximum likelihood to estimate $β_{0}, β_{1}, \dots, β_{p}$

Extending two-class logistic regression to $K > 2$ classes
One class is selected as the baseline $K$ , then the model becomes:

P r (Y = k | X = x) = \frac{e^{β_{k 0} + β_{k 1} x_{1} + \dots + β_{k p} x_{p}}}{1 + \sum_{l = 1}^{K - 1} e^{β_{l 0} + β_{l 1} x_{1} + \dots + β_{l p} x_{p}}}

for $k = 1, \dots, K - 1$ and

P r (Y = K | X = x) = \frac{1}{1 + \sum_{l = 1}^{K - 1} e^{β_{l 0} + β_{l 1} x_{1} + \dots + β_{l p} x_{p}}}

Alternatively, there's softmax coding:

P r (Y = k | X = x) = \frac{e^{β_{k 0} + β_{k 1} x_{1} + \dots + β_{k p} x_{p}}}{\sum_{l = 1}^{K} e^{β_{l 0} + β_{l 1} x_{1} + \dots + β_{l p} x_{p}}}

Sources: 1