Leave-One-Out Cross-Validation

aka LOOCV
Similar to validation set approach, but instead leaves only one observation out for testing. MSE on one value provides an approximately unbiased estimate for the test error, but is a poor estimate because it is highly variable. The procedure is repeated $n$ times. The LOOCV estimate for the test MSE is the average of these $n$ test error estimates:

{CV}_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {MSE}_{i}

Advantages over validation set approach:

far less bias
tends to not overestimate test error rate as much
performing LOOCV multiple times will always yield the same results: there is no randomness in the training/validation set splits
There's a shortcut when implementing least squares linear or polynomial regression:

{CV}_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {(\frac{y_{i} - {\hat{y}}_{i}}{1 - h_{i}})}^{2}

where ${\hat{y}}_{i}$ is the $i^{t h}$ fitted value from the original least squares fit and $h_{i}$ is the leverage defined here

LOOCV is a very general method and can be applied to any kind of predictive modeling

Sources: 1

Leave-One-Out Cross-Validation

Connect With Me!