Physics

Sociology

Pure mathematics

Cover (algebra)

Mathematics

Social science

Optics

Data collection

Machine learning

Engineering

Artificial intelligence

Focus (optics)

Data science

Face (sociological concept)

Computer science

Field (mathematics)

Mechanical engineering

Qualitative property

Discipline

Patterns

In this work, we survey a breadth of literature that has revealed the limitations of predominant practices for dataset collection and use in the field of machine learning. We cover studies that critically review the design and development of datasets with a focus on negative societal impacts and poor outcomes for system performance. We also cover approaches to filtering and augmenting data and modeling techniques aimed at mitigating the impact of bias in datasets. Finally, we discuss works that have studied data practices, cultures, and disciplinary norms and discuss implications for the legal, ethical, and functional challenges the field continues to face. Based on these findings, we advocate for the use of both qualitative and quantitative approaches to more carefully document and analyze datasets during the creation and usage phases.

Data and its (dis)contents: A survey of dataset development and use in machine learning research