Comparison of US panel vendors for online surveys.

Published on Nov 29, 2013in Journal of Medical Internet Research5.034
· DOI :10.2196/JMIR.2903
Benjamin M. Craig24
Estimated H-index: 24
(USF: University of South Florida),
Ron D. Hays152
Estimated H-index: 152
+ 3 AuthorsBryce B. Reeve74
Estimated H-index: 74
Sources
Abstract
Background: Despite the increasing use of panel surveys, little is known about the differences in data quality across panels. Objective: The aim of this study was to characterize panel survey companies and their respondents based on (1) the timeliness of response by panelists, (2) the reliability of the demographic information they self-report, and (3) the generalizability of the characteristics of panelists to the US general population. A secondary objective was to highlight several issues to consider when selecting a panel vendor. Methods: We recruited a sample of US adults from 7 panel vendors using identical quotas and online surveys. All vendors met prespecified inclusion criteria. Panels were compared on the basis of how long the respondents took to complete the survey from time of initial invitation. To validate respondent identity, this study examined the proportion of consented respondents who failed to meet the technical criteria, failed to complete the screener questions, and provided discordant responses. Finally, characteristics of the respondents were compared to US census data and to the characteristics of other panels. Results: Across the 7 panel vendors, 2% to 9% of panelists responded within 2 days of invitation; however, approximately 20% of the respondents failed the screener, largely because of the discordance between self-reported birth date and the birth date in panel entry data. Although geographic characteristics largely agreed with US Census estimates, each sample underrepresented adults who did not graduate from high school and/or had annual incomes less than US $15,000. Except for 1 vendor, panel vendor samples overlapped one another by approximately 20% (ie, 1 in 5 respondents participated through 2 or more panel vendors). Conclusions: The results of this head-to-head comparison provide potential benchmarks in panel quality. The issues to consider when selecting panel vendors include responsiveness, failure to maintain sociodemographic diversity and validated data, and potential overlap between panels. [J Med Internet Res 2013;15(11):e260]
📖 Papers frequently viewed together
162 Citations
504 Citations
36 Citations
References17
Newest
Background: Satisfactory psychometric properties in offline questionnaires do not guarantee the same outcome in Web-based versions. Any construct that is measured online should be compared to a paper-based assessment so that the appropriateness of online questionnaire data can be tested. Little research has been done in this area regarding Attention-Deficit/Hyperactivity Disorder (ADHD) in adults. Objective: The objective was to simultaneously collect paper-based and Web-based ADHD questionnaire...
17 CitationsSource
Background: The majority of Internet-mediated studies use measures developed as paper-and-pencil measures or face-to-face-delivered material. Previous research suggests that the equivalence between online and offline measures must be demonstrated rather than assumed. Objective: The objective of this study was to explore the equivalence 4 measures completed in an online or offline setting. Methods: A sample of students (n = 1969) was randomly assigned to complete 4 popular scales (the SF-12v2, th...
52 CitationsSource
#1J. Michael Brick (UMD: University of Maryland, College Park)H-Index: 22
The twentieth century saw a dramatic change in the way infor- mation was generated as probability sampling replaced full enumeration. We examine some key events of the past and issues being addressed today to gain perspective on the requirements that might steer changes in the di- rection of survey sampling in the future. The potential for online Web pan- els and other methods that do not use probability sampling are considered. Finally, we conclude with some thoughts on how the future of survey...
82 CitationsSource
#1David S. YeagerH-Index: 35
#2Jon A. Krosnick (Stanford University)H-Index: 92
Last. Rui Wang (Stanford University)H-Index: 2
view all 7 authors...
This study assessed the accuracy of telephone and Internet surveys of probability samples and Internet surveys of non-probability samples of American adults by comparing aggregate survey results against benchmarks. The probability sample surveys were consistently more accurate than the non-probability sample surveys, even after post-stratification with demographics. The non-probability sample survey measurements were much more variable in their accuracy, both across measures within a single surv...
504 CitationsSource
#1Hiroaki Sugiura (Nara Medical University)H-Index: 4
#2Yasushi OhkusaH-Index: 18
Last. Tomoaki ImamuraH-Index: 23
view all 6 authors...
Background: Early detection of symptoms arising from exposure to pathogens, harmful substances, or environmental changes is required for timely intervention. The administration of Web-based questionnaires is a potential method for collecting information from a sample population. Objective: The objective of our study was to develop a Web-based daily questionnaire for health (WDQH) for symptomatic surveillance. Methods: We adopted two different survey methods to develop the WDQH: an Internet panel...
16 CitationsSource
#4Jie Shen (UCLA: University of California, Los Angeles)
Abstract Objectives To evaluate the Patient-Reported Outcomes Measurement Information System (PROMIS), which collected data from an Internet polling panel, and to compare PROMIS with national norms. Study Design and Setting We compared demographics and self-rated health of the PROMIS general Internet sample (N = 11,796) and one of its subsamples (n = 2,196) selected to approximate the joint distribution of demographics from the 2000 U.S. Census, with three national surveys and U.S. Census data. ...
292 CitationsSource
#1Kumar Nagaraja RaoH-Index: 5
#2Olena KaminskaH-Index: 11
Last. Allan L. McCutcheonH-Index: 13
view all 3 authors...
Survey response rates have been declining over the past several decades, particularly for random-digit-dialing (RDD) telephone surveys (see de Leeuw and de Heer 2002; Steeh 1981). This trend af- fects research panels such as the Gallup Panel, which uses RDD methodology to recruit its members. If significant improvements in pan- el recruitment response rates are to be achieved, new approaches must be considered. This paper presents the findings of a mail and telephone mode experiment conducted by...
28 CitationsSource
#1Burke D. GrandjeanH-Index: 7
#2Nanette M. NelsonH-Index: 4
Last. Patricia A. TaylorH-Index: 8
view all 3 authors...
Recently, the U.S. Environmental Protection Agency sought to determine whether a recruited Internet panel of respondents can be used to produce reliable estimates of willingness to pay for goods or services for which no formal market exists. Here we report the results of a national mode test comparing results from an Internet panel survey on air quality in national parks with the same questionnaire administered by mail and by phone. Sampling frame was held nearly constant by using national Rando...
8 Citations
As more researchers use online panels for studies, the need for standardized rates to evaluate these studies becomes paramount. There are currently many different ways and conflicting terminology used to compute various metrics for online panels. This paper discusses the sparse literature on how to compute response, refusal, and other rates and proposes a set of formulas and a standardized terminology that can be used to calculate and interpret these metrics for online panel studies. A descripti...
359 CitationsSource
ABSTRACT Online polls and surveys using volunteer panels have grabbed a large slice of the research data collection market largely because they offer fast and cost-effective research solutions. Concerns over the level of internet penetration and the use of volunteer panels have in many cases been set aside, as has research evidence showing sharp differences between the results of online and conventional telephone and face-to-face surveys. We set out to find effective weighting systems that would...
25 CitationsSource
Cited By58
Newest
#1Spencer L. James (BYU: Brigham Young University)H-Index: 9
#2Jeremy B. Yorgason (BYU: Brigham Young University)H-Index: 22
Last. Dean M. Busby (BYU: Brigham Young University)H-Index: 26
view all 5 authors...
Source
#1Hans Oh (SC: University of Southern California)H-Index: 18
#2Banawa R (UCLA: University of California, Los Angeles)
Last. Jimi Huh (SC: University of Southern California)H-Index: 25
view all 5 authors...
Abstract null null Aim null While cigarette and marijuana use has been linked to psychotic experiences, few empirical studies have examined the relation between vaping and psychotic experiences. null null null Methods null We analyzed data from the Healthy Minds Survey (September 2020 – December 2020; N = 29,232 students from 36 universities), and used multiple logistic regression models to examine the associations between vaping over the past 30 days and psychotic experiences over the past 12 m...
Source
#1Beth BourdeauH-Index: 11
#2Brenda A. MillerH-Index: 33
Last. Joel W. GrubeH-Index: 62
view all 6 authors...
Source
#1Joanna Hockenhull (Guy's and St Thomas' NHS Foundation Trust)H-Index: 4
#2Joshua C. Black (Hospital Authority)H-Index: 3
Last. David M. Wood ('KCL': King's College London)H-Index: 59
view all 7 authors...
AIM To estimate prevalence of last 12-month non-medical use (NMU) of benzodiazepines and Z-drugs (the non-benzodiazepine hypnotics zaleplon, zolpidem, and zopiclone) in the United Kingdom. METHODS Data were collected using the Non-Medical Use of Prescription Drugs survey with post-stratification weighting applied to be representative of the UK population (≥16 years). Participants were questioned about whether they had non-medically used benzodiazepines and/or Z-drugs in the last 12-months and fr...
3 CitationsSource
#1Venera Tomaselli (University of Catania)H-Index: 6
#2Sebastiano Battiato (University of Catania)H-Index: 31
Last. Riccardo Polosa (University of Catania)H-Index: 53
view all 6 authors...
This article reviews contemporary issues in survey research, connecting established methods to innovative tools and technologies like real-time sensors and computer vision. This link takes into acc...
1 CitationsSource
Source
#1John P. Barile (U.H.: University of Hawaii at Manoa)H-Index: 15
#2Rebecca J. Guerin (NIOSH: National Institute for Occupational Safety and Health)H-Index: 8
Last. Christine E. PrueH-Index: 15
view all 10 authors...
BACKGROUND: Investigating antecedents of behaviors, such as wearing face coverings, is critical for developing strategies to prevent SARS-CoV-2 transmission. PURPOSE: The purpose of this study was to determine associations between theory-based behavioral predictors of intention to wear a face covering and actual wearing of a face covering in public. METHODS: Data from a cross-sectional panel survey of U.S. adults conducted in May and June 2020 (N = 1,004) were used to test a theory-based behavio...
9 CitationsSource
#1Sungwon Kim (St. John's University)H-Index: 1
#2Daniel P. Connaughton (UF: University of Florida)H-Index: 17
Abstract Purpose Parents are important figures in properly managing youth sport concussions. Although media attention has predominantly centered on concussions in contact/collision sports, evidence suggests that the concussion rate in soccer is comparable to those found in contact/collision sports. Given the high rate of concussions in youth soccer, this study aimed to examine parents of youth soccer athletes' attitudes and perceptions about concussions and associated factors. Methods A cross-se...
Source
Facebook ads are increasingly used by political scientists as a method of survey recruitment. A key advantage is said to be the ability to recruit targeted audiences defined by demographics, political beliefs, location, and numerous other attributes. The same feature has been decried by non-researchers concerned about potential racial discrimination and foreign influence in elections. The extent to which these ads actually reach their targets, however, is unknown. Using a series of six surveys a...
3 CitationsSource
#1Radhika Gharpure (CDC: Centers for Disease Control and Prevention)H-Index: 12
#2Gabrielle F. Miller (CDC: Centers for Disease Control and Prevention)H-Index: 7
Last. Amanda G. Garcia-Williams (CDC: Centers for Disease Control and Prevention)H-Index: 9
view all 6 authors...
Cleaning and disinfection of frequently touched surfaces and frequent hand hygiene are recommended measures to prevent transmission of SARS-CoV-2, the virus that causes COVID-19. Since the onset of the COVID-19 pandemic, poison center calls regarding exposures to cleaners, disinfectants, and hand sanitizers have increased as compared with prior years, indicating a need to evaluate household safety precautions. An opt-in Internet panel survey of 502 U.S. adults was conducted in May 2020. Survey i...
2 CitationsSource