Leave-One-Out Cross-Validation (LOOCV)
As the name implies, LOOCV will leave one observation out as a test set, then fit the model to the rest of the data.
In our mtcars dataset, it will work like this
In the first iteration, the first observation is the test dataset; the model is fit on the other observations, then MSE or other stats are calculated.
In the second iteration, the second observation is the test dataset; the model is fit on the other observations, then MSE or other stats are calculated.
The step will be repeated for 32 times; then the calculated stats will be averaged.
The advantage of LOOCV over Random Selection is zero randomness. Besides, the bias will also be lower as the model is trained on the entire dataset, which consequently will not overestimate the test error rate. But its disadvantage is the computational time.
We can easily use caret package to perform LOOCV.
|
1 2 |
control <- trainControl(method="LOOCV") lm_model <- train(mpg ~ . , data=mtcars, trControl=control, method="lm") |
|
1 |
lm_model |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
> print(model) Linear Regression 32 samples 10 predictors No pre-processing Resampling: Leave-One-Out Cross-Validation Summary of sample sizes: 31, 31, 31, 31, 31, 31, ... Resampling results: RMSE Rsquared 3.490209 0.6818575 |