Naive Bayes

Naive Bayes estimates $f_1(x),\dots ,f_K(x)$ with the assumption that within the $k^{th}$ class, the $p$ predictors are independent. Mathematically, for $k=1,\dots ,K$:

where $f_{kj}$ is the density function of the $j^{th}$ predictor among observations in the $k^{th}$ class.

Naive Bayes is a good estimator where $n$ is not large enough relative to $p$ for us to effectively estimate the joint distribution of the predictors within each class.

The posterior probability of naive Bayes for $k=1,\dots ,K$ is:

Estimate $f_{kj}$ using training data $x_{1j},\dots ,x_{nj}$ using one of the following methods:

• If $X_j$ is quantitative, then we can assume that $X_j|Y=k\sim N({\mu }_{jk},{\sigma }_{jk}^2)$
• If $X_j$ is quantitative, use a non-parametric estimate for $f_{kj}$. Think of a histogram or a kernel density estimator (a smoothed version of a histogram)
• If $X_j$ is qualitative, count the proportion of training observations for the $j^{th}$ predictor corresponding to each class.

Sources: 1