user experience

usability evaluation

user satisfaction

data extraction

Proceedings of the Association for Information Science and Technology

This study investigated performance, usability, and user experience with Rayyan, a systematic review (SR) tool. A remote usability inspection was conducted via the institutional Zoom platform. Each participant was randomly assigned to two groups and completed two citation screening sessions independently. Data were collected through pre‐ and post‐test questionnaires, screening sessions, and Systems Usability Scale (SUS) surveys. Thirteen participants completed the study. We found significant accuracy difference between Web Rayyan and its mobile app (t[12] = 1.877 p = .042), but no statistically sequential effect on screening performance. The average SUS scores of both Rayyan versions were below 68. More participants (77%) preferred Web Rayyan to the mobile app. The mobile Rayyan has potential if its user interface design can prevent errors or allow easy action reversal. As the first controlled experiment investigating both Web and mobile Rayyan, this study provides empirical evidence to guide the design, development, and selection of SR tools.

Performance, Usability, and User Experience of Rayyan for Systematic Reviews