Generative Models for Classification

When there is substantial separation between the classes, parameter estimates for logistic regression are unstable
If the distribution of the predictors $X$ is approximately normal in each of the classes and the sample size is small, then these approaches may be more accurate than logistic regression
These methods can be naturally extended to the case of more than two response classes

Let $π_{k}$ represent the overall or prior probability that a randomly chosen observation comes from the $k^{t h}$ class. Let $f_{k} (X) \equiv P r (X | Y = k)$ denote the density function of $X$ or an observation that comes from the $k^{t h}$ class. Then the posterior probability that an observation $X = x$ belongs to the $k^{t h}$ class is:

P r (Y = k | X = x) = \frac{π_{k} f_{k} (x)}{\sum_{l = 1}^{K} π_{l} f_{l} (x)}

Estimate $π_{k}$ by computing the fraction of the training observations that belong to the $k^{t h}$ class using a random sample from the population. To estimate $f_{k} (x)$ , simplifying assumptions is typical as it's more challenging to do. Three classifiers that use different estimates of $f_{k} (x)$ : Linear Discriminant Analysis for p eq1, Linear Discriminant Analysis for p greater than 1, Quadratic Discriminant Analysis, and Naive Bayes.

Sources: 1

Generative Models for Classification

Connect With Me!