The Validation Set Approach

Randomly divides data into a training and test set. The resulting validation set error rate - typically assessed using MSE in the case of a quantitative response - provides an estimate of the test error rate.

Drawbacks:

• the validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
• only a subset of the observations are used to fit the model. Statistical models tend to perform worse when trained with fewer observations, thus the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.

Sources: 1