A Comparison of Classification Methods

Mathematically: LDA vs QDA vs naive Bayes vs logistic regression

Setting $K$ as the baseline class, we assign an observation to the class for $k = 1, \dots, K$ that maximizes

log (\frac{P r (Y = k | X = x)}{P r (Y = K | X = x)})

For LDA:

log (\frac{P r (Y = k | X = x)}{P r (Y = K | X = x)}) = a_{k} + \sum_{j = 1}^{p} b_{k j} x_{j}

where a_{k} = log (\frac{π_{k}}{π_{K}}) - \frac{1}{2} (μ_{k} + μ_{K})^{T} Σ^{- 1} (μ_{k} - μ_{K})

and b_{k j} is the j^{t h} component of Σ^{- 1} (μ_{k} - μ_{K})

For QDA:

log (\frac{P r (Y = k | X = x)}{P r (Y = K | X = x)}) = a_{k} + \sum_{j = 1}^{p} b_{k j} x_{j} + \sum_{j = 1}^{p} \sum_{l = 1}^{p} c_{k j l} x_{j} x_{l}

where $a_{k}$ , $b_{k j}$ , and $c_{k j l}$ are functions of $π_{k}$ , $π_{K}$ , $μ_{k}$ , $μ_{K}$ , $Σ_{k}$ and $Σ_{K}$
For naive Bayes:

log (\frac{P r (Y = k | X = x)}{P r (Y = K | X = x)}) = a_{k} + \sum_{j = 1}^{p} g_{k j} (x_{j})

where a_{k} = log (\frac{π_{k}}{π_{K}}) and g_{k j} (x_{j}) = log (\frac{f_{k j} (x_{j})}{f_{K j} (x_{j})})

For logistic regression:

log (\frac{P r (Y = k | X = x)}{P r (Y = K | X = x)}) = β_{k 0} + \sum_{j = 1}^{p} β_{k j} x_{j}

where $β$ coefficients are chosen to maximize the likelihood function

Compared to KNN, which is a non-parametric approach:

dominate LDA and logistic regression when the decision boundary is highly non-linear, provided that $n$ is very large and $p$ is small
KNN requires $n$ to be much larger than $p$
In settings where the decision boundary is non-linear but $n$ is only modest, or $p$ is not very small, then QDA may be preferred to KNN.
Unlike logistic regression, KNN does not tell us which predictors are important

Practically: LDA vs QDA vs naive Bayes vs logistic regression vs KNN

Which is better? It depends on your use case! Know how to test and compare each model's accuracy.

Sources: 1