Data mining

Computer science

Gene

Engineering

Sensitivity (control systems)

Medicine

Scalability

Biochemistry

Nursing

Search engine indexing

Health informatics

Population

Artificial intelligence

Linkage (software)

Environmental health

Record linkage

Chemistry

Electronic engineering

Database

Public health

BMC Medical Informatics and Decision Making

Background Record linkage is the process of identifying and combining records about the same individual from two or more different datasets. While there are many open source and commercial data linkage tools, the volume and complexity of currently available datasets for linkage pose a huge challenge; hence, designing an efficient linkage tool with reasonable accuracy and scalability is required. Methods We developed CIDACS-RL (Centre for Data and Knowledge Integration for Health – Record Linkage), a novel iterative deterministic record linkage algorithm based on a combination of indexing search and scoring algorithms (provided by Apache Lucene). We described how the algorithm works and compared its performance with four open source linkage tools (AtyImo, Febrl, FRIL and RecLink) in terms of sensitivity and positive predictive value using gold standard dataset. We also evaluated its accuracy and scalability using a case-study and its scalability and execution time using a simulated cohort in serial (single cor...

CIDACS-RL: a novel indexing search and scoring-based record linkage system for huge datasets with high accuracy and scalability