Linear Discriminant Analysis for p eq1

Linear Discriminant Analysis for $p = 1$

The Linear Discriminant Analysis (LDA) classifier results from assuming that the observations within each class come from a normal distribution with a class specific mean and a common variance $σ^{2}$ , and plugging estimates for these parameters into the Bayes classifier.

Assume that $f_{k} (x)$ is normal or Gaussian:

f_{k} (x) = \frac{1}{\sqrt{2 π} σ_{k}} exp (- \frac{1}{2 σ_{k}^{2}} (x - μ_{k})^{2})

Where $μ_{k}$ and $σ_{k}^{2}$ are the mean and variance for the $k^{t h}$ class.
Assume that $σ_{1}^{2} = \dots = σ_{K}^{2}$ denoted by $σ^{2}$ :

p_{k} (x) = \frac{π_{k} \frac{1}{\sqrt{2 π} σ} exp (- \frac{1}{2 σ^{2}} (x - μ_{k})^{2})}{\sum_{l = 1}^{K} π_{l} \frac{1}{\sqrt{2 π} σ} exp (- \frac{1}{2 σ^{2}} (x - μ_{l})^{2})}

The Bayes classifier assigns an observation $X = x$ to the class for which

δ_{k} (x) = x * \frac{μ_{k}}{σ^{2}} - \frac{μ_{k}^{2}}{2 σ^{2}} + log (π_{k})

is largest.

If $K = 2$ and $π_{1} = π_{2}$ then the Bayes decision boundary is:

x = \frac{μ_{1}^{2} - μ_{2}^{2}}{2 (μ_{1} - μ_{2})} = \frac{μ_{1} + μ_{2}}{2}

LDA method uses estimates for the Bayes classifier:

{\hat{μ}}_{k} = \frac{1}{n_{k}} \sum_{i : y_{i} = k} x_{i}

{\hat{σ}}^{2} = \frac{1}{n - K} \sum_{k = 1}^{K} \sum_{i : y_{i} = k} (x_{i} - {\hat{μ}}_{k})^{2}

Where $n$ is the total number of training estimates and $n_{k}$ is the number of training observations in the $k^{t h}$ class.

In the absence of any additional information, LDA estimates $π_{k}$ using:

{\hat{π}}_{k} = \frac{n_{k}}{n}

The LDA classifier assign an observation $X = x$ to the class for which

{\hat{δ}}_{k} (x) = x * \frac{{\hat{μ}}_{k}}{{\hat{σ}}^{2}} - \frac{{\hat{μ}}_{k}^{2}}{2 {\hat{σ}}^{2}} + log ({\hat{π}}_{k})

is the largest.

Sources: 1

Linear Discriminant Analysis for p=1

Connect With Me!

Linear Discriminant Analysis for $p = 1$