Pathology

Sociology

Metric (unit)

Consistency (knowledge bases)

Population

Data mining

Machine learning

Demography

Raw data

Artificial intelligence

Synthetic data

Economics

Fidelity

Programming language

Alternative medicine

Operations management

Data sharing

Computer science

Bivariate analysis

Data consistency

Medicine

Telecommunications

Database

IEEE Access

Synthetic datasets are gradually emerging as solutions for data sharing. Multiple synthetic data generators have been introduced in the last decade fueled by advancement in machine learning and by the increased demand for fast and inclusive data sharing, yet their utility is not well understood. Prior research tried to compare the utility of synthetic data generators using different evaluation metrics. These metrics have been found to generate conflicting conclusions making direct comparison of synthetic data generators very difficult. This paper identifies four criteria (or dimensions) for masked data evaluation by classifying available utility metrics into different categories based on the measure they attempt to preserve: attribute fidelity, bivariate fidelity, population fidelity, and application fidelity. A representative metric from each category is chosen based on popularity and consistency, and the four metrics are used to compare the overall utility of four recent data synthesizers across 19 datasets of different sizes and feature counts. The paper also examines correlations between the selected metrics in an attempt to streamline synthetic data utility.

A Multi-Dimensional Evaluation of Synthetic Data Generators