Linear Discriminant Analysis for $p>1$

Assume that $X=(X_1,X_2,\dots ,X_p)$ is drawn from a multivariate Gaussian (or multivariate normal) distribution, with a class-specific mean vector and a common covariance matrix.

The multivariate Gaussian distribution assumes that each individual predictor follows a one-dimensional normal distribution with some correlation between each pair of predictors.

To indicate that a $p$-dimensional random variable $X$ has a multivariate Gaussian distribution, we write $X\sim N(\mu ,\mathrm{\Sigma })$. Here $E(X)=\mu$ is the mean of $X$ (a vector with $p$ components), and $Cov(X)=\mathrm{\Sigma }$ is the $p\times p$ covariance matrix of $X$. Formally, the multivariate Gaussian density is defined as:

In the case of $p>1$ predictors, the LDA classifier assumes that the observations in the $k^{th}$ class are drawn from a multivariate Gaussian distribution $N({\mu }_k,\mathrm{\Sigma })$, where ${\mu }_k$ is a class-specific mean vector, and $\mathrm{\Sigma }$ is a covariance matrix that is common to all $K$ classes.

The Bayes classifier assigns an observation $X=x$ to the class for which

is largest.

The Bayes decision boundaries are the values $x$ for which ${\delta }_k(x)={\delta }_l(x)$:

for $k\ne 1$ and ${\pi }_k$ is the same for each class. The Bayes classifier will classify an observation according to the region in which it is located.

LDA method uses estimates for the Bayes classifier the same as the $p=1$ case.

Check for errors using a Confusion Matrix and accuracy with a ROC Curve

Sources: 1