Cross-validation

Data mining

Computer science

Artificial intelligence

Programming language

Fold (higher-order function)

IEEE Transactions on Knowledge and Data Engineering

It is popular to evaluate the performance of classification algorithms by null k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform null k -fold cross validation. Most of them did not consider the correlation among the replications of null k -fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether null k -fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of null k -fold cross validation are first analyzed for null k -nearest neighbors with null null $k= 1$ null null null null k null = null 1 null null null null null . Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of null k -fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of null k -fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The null k -fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.

Reliable Accuracy Estimates from k-Fold Cross Validation