k-Fold Cross Validation

alternative to LOOCV
Randomly divides observations into $k$ groups, or folds, of approximately equal size. Similar to LOOCV by holding out one fold at a time and calculating the MSE for each fold. LOOCV is a special case of $k$-fold where $k=n$. The $k$-fold CV estimate is computed by:

$k$-fold has a computational advantage over LOOCV

Bias-Variance Trade-Off for k-Fold Cross-Validation

$k$-fold CV often gives more accurate estimates of the test error rate than LOOCV. This has to do with a bias-variance trade-of.

• LOOCV is better than $k$-fold for bias reduction
• $k$-fold is better than LOOCV for variance reduction
  To summarize, there is a bias-variance trade-of associated with the choice of $k$ in $k$-fold cross-validation. Typically, given these considerations, one performs $k$-fold cross-validation using $k$ = 5 or $k$ = 10, as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance.

Sources: 1