Crowdsourcing in health and medical research: A systematic review

Cheng Wang, Larry Han, Gabriella Stein, Suzanne Day, Cedric Bien-Gund, Allison Mathews, Jason J. Ong, Pei Zhen Zhao, Shu Fang Wei, Jennifer Walker, Roger Chou, Amy Lee, Angela Chen, Barry Bayus, Joseph D. Tucker

Research output: Contribution to journalReview articlepeer-review

45 Scopus citations


Background: Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health. Methods: We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach. Results: We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled. Conclusions: Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Original languageEnglish (US)
Article number8
JournalInfectious Diseases of Poverty
Issue number1
StatePublished - Jan 20 2020


  • Challenge contest
  • Crowdsourcing
  • Health
  • Innovation
  • Medicine
  • Systematic review

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health
  • Infectious Diseases


Dive into the research topics of 'Crowdsourcing in health and medical research: A systematic review'. Together they form a unique fingerprint.

Cite this