Machine learning

Mathematical analysis

Generalization

Task (project management)

Convolutional neural network

Overfitting

Data mining

Bearing (navigation)

Computer science

Random forest

Management

Geography

Mathematics

Artificial neural network

Hyperparameter

Support vector machine

Geodesy

Benchmark (surveying)

Artificial intelligence

Economics

Expert Systems with Applications

This paper presents a systematic procedure to fairly compare experimental performance scores for machine learning methods for fault diagnosis based on vibration signals. In the vast majority of related scientific publications, the estimated accuracy and similar performance criteria are the sole quality parameter presented. However, the experimental design giving rise to these results is mostly biased, based on unacceptably simple validation methods and on recycling identical patterns in test data sets, previously used for training. Moreover, the methods in general overfit their hyperparameters, introducing additional overoptimistic results. In order to remedy this defect, we critically analyse the usual training-validation-test division and propose an algorithmic guideline in the form of a validation framework. This allows a well defined comparison of experimental results. In order to illustrate the ideas of the paper, the Case Western Reserve University Bearing Data benchmark is used as a case study. Four di...

An experimental methodology to evaluate machine learning methods for fault diagnosis based on vibration signals