1. bookVolume 38 (2022): Issue 3 (September 2022)
Journal Details
License
Format
Journal
eISSN
2001-7367
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English
Open Access

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Published Online: 12 Sep 2022
Volume & Issue: Volume 38 (2022) - Issue 3 (September 2022)
Page range: 793 - 822
Received: 01 Jan 2021
Accepted: 01 Mar 2022
Journal Details
License
Format
Journal
eISSN
2001-7367
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English

Aranberri, N., G. Labaka, A. Díaz de Ilarraza, and K. Sarasola. 2017. “Ebaluatoia: Crowd Evaluation for English–Basque Machine Translation.” Language Resources and Evaluation 51(4): 1053–1084. DOI: https://doi.org/10.1007/s10579-016-9335-x. Search in Google Scholar

Barreto, M., L. Frasure-Yokley, E. Vargas, and J. Wong. 2018. “Best Practices in Collecting Online Data with Asian, Black, Latino, and White Respondents: Evidence from the 2016 Collaborative Multiracial Post-Election Survey.” Politics, Groups, and Identities 6(1): 171–180. DOI: https://doi.org/10.1080/21565503.2017.1419433. Search in Google Scholar

Behr, D., L. Kaczmirek, W. Bandilla, and M. Braun. 2012. “Asking Probing Questions in Web Surveys: Which Factors Have an Impact on the Quality of Responses?” Social Science Computer Review 30(4): 487–498. DOI: https://doi.org/10.1177/0894439311435305. Search in Google Scholar

Behr, D., K. Meitinger, M. Braun, and L. Kaczmirek. 2017. “Web Probing-Implementing Probing Techniques from Cognitive Interviewing in Web Surveys with the Goal to Assess the Validity of Survey Question.” Social Science Open Access Repository: 18. DOI: https://doi.org/10.15465/gesis-sg_en_023 Search in Google Scholar

Brown, A., G. López, and M. Hugo Lopez. 2016. Digital Divide Narrows for Latinos as More Spanish Speakers and Immigrants Go Online. Washington, DC: Pew Research Center. Available at: https://www.pewresearch.org/hispanic/2016/07/20/digital-divide-narrows-for-latinos-as-more-spanish-speakers-and-immigrants-go-online/ (accessed July 2021). Search in Google Scholar

Buchanan, E. and J. Scofield. 2018. “Methods to Detect Low Quality Data and Its Implication for Psychological Research.” Behavior Research Methods 50(6): 2586–2596. DOI: https://doi.org/10.3758/s13428-018-1035-6.29542063 Search in Google Scholar

Callegaro, M., A. Villar, D.S. Yeager, and J.A. Krosnick. 2014. “A Critical Review of Studies Investigating the Quality of Data Obtained with Online Panels Based on Probability and Nonprobability Samples.” In Online Panel Research: A Data Quality Perspective, edited by M. Callegaro, R.P. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas: 23–53. Sussex, UK: Wiley.10.1002/9781118763520.ch2 Search in Google Scholar

Chandler, J. and G. Paolacci. 2017. “Lie for a Dime: When Most Prescreening Responses are Honest but Most Study Participants are Impostors.” Social Psychological and Personality Science 8(5): 500–508. DOI: https://doi.org/10.1177/1948550617698203. Search in Google Scholar

Chandler, J., C. Rosenzweig, A. Moss, J. Robinson, and L. Litman. 2019. “Online Panels in Social Science Research: Expanding Sampling Methods Beyond Mechanical Turk.” Behavior Research Methods 51(5): 2022–2038. DOI: https://doi.org/10.3758/s13428-019-01273-7.679769931512174 Search in Google Scholar

Chmielewski, M., and S. Kucker. 2019. “An MTurk Crisis? Shifts in Data Quality and The Impact on Study Results.” Social Psychological and Personality Science 11(4): 464–473. DOI: https://doi.org/10.1177/1948550619875149. Search in Google Scholar

De Heer, W. and E. de Leeuw. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey Nonresponse, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little: 41–54. New York: Wiley. Search in Google Scholar

Edgar, J. 2013. “Self-Administered Cognitive Interviewing.” In 68th American Association for Public Opinion Research Conference, May 2013. Boston, MA. Available at: http://www.aapor.org/AAPOR_Main/media/AnnualMeetingProceedings/2013/Session_A-1-3-Edgar.pdf (accessed July 2021). Search in Google Scholar

Edgar, J., J. Murphy, and M. Keating. 2016. “Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions.” Sage Open 6(4): 1–14. DOI: https://doi.org/10.1177/2158244016671770. Search in Google Scholar

Goerman, P., L. Fernandez, and R. Quiroz. 2018. “Translation of Country-Specific Programs and Survey Error: Measuring the Education Level of Immigrants.” Translation & Interpreting 10(2): 21–33. DOI: https://doi.org/10.1177/2158244016671770. Search in Google Scholar

Hanson, M. and L. Espinosa. 2016. “Culture, Ethnicity, and Linguistic Diversity: Implications for Early Childhood Special Education.” In Handbook of Early Childhood Special Education, edited by B. Reichow, Boyd, E. Barton, and S. Odom: 455–471. Springer, Cham.10.1007/978-3-319-28492-7_24 Search in Google Scholar

Hillygus, S., N. Jackson, and M. Young. 2014. “Professional respondents in nonprobability online panels.” In Online Panel Research: A Data Quality Perspective, edited by M. Callegaro, R.P. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas: 219–237. Sussex, UK: Wiley.10.1002/9781118763520.ch10 Search in Google Scholar

Kennedy, C., N. Hatley, A. Lau, A. Mercer, S. Keeter, J. Ferno, and D. Asare-Marfo. 2020. Assessing the Risks to Online Polls from Bogus Respondents. Washington, DC: Pew Research Center. Available at: https://www.pewresearch.org/methods/wp-content/uploads/sites/10/2020/02/PM_02.18.20_dataquality_FULL.REPORT.pdf (accessed July 2021). Search in Google Scholar

Kennedy, C., N. Hatley, A. Lau, A. Mercer, S. Keeter, J. Ferno, and D. Asare-Marfo. 2021, “Strategies for Detecting Insincere Respondents in Online Polling.” Public Opinion Quarterly 85(4): 1050–1075. DOI: https://doi.org/10.1093/poq/nfab057. Search in Google Scholar

Kennedy, R., S. Clifford, T. Burleigh, P. Waggoner, R. Jewell, and N. Winter. 2020. “The Shape of and Solutions to the MTurk Quality Crisis.” Political Science Research and Methods 8(4): 614–629. DOI: https://doi.org/10.1093/poq/nfab057 Search in Google Scholar

Lenzner, T. and C. Neuert. 2017. “Pretesting Survey Questions Via Web Probing–Does it Produce Similar Results to Face-to-Face Cognitive Interviewing?” Survey Practice 10(4): 2768. DOI: https://doi.org/10.29115/SP-2017-0020. Search in Google Scholar

Luiten, A., J. Hox, and E. de Leeuw. 2020. “Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of an International Study Across Countries and Surveys.” Journal of Official Statistics 36(3): 469–487. DOI: https://doi.org/10.2478/jos-2020-0025. Search in Google Scholar

Matthijsse, S., E. de Leeuw, and J. Hox. 2015. “Internet Panels, Professional Respondents, and Data Quality.” Methodology 11(3). DOI: https://doi.org/10.1027/1614-2241/a000094. Search in Google Scholar

Meitinger, K. and D. Behr. 2016. “Comparing Cognitive Interviewing and Online Probing: Do They Find Similar Results?” Field Methods 28(4): 363–380. DOI: https://doi.org/10.1177/1525822X15625866. Search in Google Scholar

Meitinger, K., D. Behr, and M. Braun. 2019. “Using Apples and Oranges to Judge Quality?: Selection of Appropriate Cross-National Indicators of Response Quality in Open-Ended Questions.” Social Science Computer Review 39(3): 1–22. DOI: https://doi.org/10.1177/0894439319859848. Search in Google Scholar

Murphy, J., Edgar, J., and Keating, M. 2014. “Crowdsourcing in the Cognitive Interviewing Process.” In the Annual Meeting of the American Association for Public Opinion Research, May, 2014. Anaheim, CA. Available at: http://dc-aapor.org/2014%20conference%20slides/EdgarMurphyKeating.pdf (accessed January 2021). Search in Google Scholar

Neuert, C., and T. Lenzner. 2019. “Effects of the Number of Open-Ended Probing Questions on Response Quality in Cognitive Online Pretests.” Social Science Computer Review 39(3): 456–468. DOI: https://doi.org/10.1177/0894439319866397. Search in Google Scholar

Ortega-Santos, Iván. 2019. “Crowdsourcing for Hispanic Linguistics: Amazon’s Mechanical Turk as a source of Spanish data.” Borealis – An International Journal of Hispanic Linguistics 8(1): 187–215. DOI: https://doi.org/10.7557/1.8.1.4670. Search in Google Scholar

Porter, C.O., R. Outlaw, J.P. Gale, and T.S. Cho. 2019. “The Use of Online Panel Data in Management Research: A Review and Recommendations”. Journal of Management 45(1): 319–344. DOI: https://doi.org/10.1177/0149206318811569. Search in Google Scholar

Prince, K.R., A.R. Litovsky, and D.G. Friedman-Wheeler. 2012. “Internet-Mediated Research: Beware of Bots.” The Behavior Therapist, 35(5): 85–88. Available at: https://services.abct.org/i4a/doclibrary/index.cfm?category_id=15 Search in Google Scholar

Rumbaut, R.G., and D.S. Massey. 2013. “Immigration & Language Diversity in the United States.” Daedalus 142(3): 141–154. DOI: https://doi.org/10.1162/DAED_a_00224.409200825018562 Search in Google Scholar

Shepperd, J.A., G. Pogge, J.M. Hunleth, S. Ruiz, and E.A. Waters. 2021. “Guidelines for Conducting Virtual Cognitive Interviews During a Pandemic.” Journal of Medical Internet Research 23(3): 25173e. DOI: https://doi.org/10.2196/25173.795410933577464 Search in Google Scholar

Simone, M. 2019a. “How to Battle the Bots Wrecking Your Online Study,” Behavioral Scientist. Available at: https://behavioralscientist.org/how-to-battle-the-bots-wrecking-your-online-study/. (accessed January 2021). Search in Google Scholar

Simone, M. 2019b. “Bots started sabotaging my online research. I fought back,” STAT. Available at: https://www.statnews.com/2019/11/21/bots-started-sabotaging-my-online-research-i-fought-back/. (accessed January 2021). Search in Google Scholar

Storozuk, A., M. Ashley, V. Delage, and E. Maloney. 2020. “Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks.” Quantitative Methods for Psychology 16(5): 472–481. DOI: https://doi.org/10.20982/tqmp.16.5.p472. Search in Google Scholar

Team Y&R. 2020. Census Tracking Survey. Ann Arbor, MI: Inter-university Consortium for Political and Social Research. Available at: https://doi.org/10.3886/E125761V3. Search in Google Scholar

Trejo, Y.A.G., and A. Schoua-Glusberg. 2017. “Device and Internet Use among Spanish-Dominant Hispanics: Implications for Web Survey Design and Testing.” Survey Practice 10(3). DOI: https://doi.org/10.29115/SP-2017-0016. Search in Google Scholar

University of Maryland. 2014. “Institutional Review Board Investigator Handbook.” College Park, MD: University of Maryland. Available at: https://research.umd.edu/-sites/default/files/documents/irb-forms/Investigator%20Handbook%20FINAL%20112114.pdf (accessed July 2021). Search in Google Scholar

Upadhyay, U.D., and H. Lipkovich. 2020. “Using Online Technologies to Improve Diversity and Inclusion in Cognitive Interviews with Young People.” BMC Medical Research Methodology 20(1): 1–10. DOI: https://doi.org/10.1186/s12874-020-01024-9.729569032539726 Search in Google Scholar

Willis, G.B. 2005. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications. DOI: https://dx.doi.org/10.4135/9781412983655.10.4135/9781412983655 Search in Google Scholar

Yarrish, C., L. Groshon, J. Mitchell, A. Appelbaum, S. Klock, T. Winternitz, and D. Friedman-Wheeler. 2019. “Finding the Signal in the Noise: Minimizing Responses from Bots and Inattentive Humans in Online Research.” The Behavior Therapist 42(7): 235–242. Available at: https://services.abct.org/i4a/doclibrary/index.cfm?category_id=22 Search in Google Scholar

Yu, E., A. Fobia, J. Graber, J. Holzberg, R. Kaplan, B. Kopp, K. Kubzdela, B. Mockovak, R. Morrison, P. Scanlon. 2019. Experiences Using Online Testing to Support Survey-Methods Research and Pre-Testing in the Federal Government. Research and Methodology Directorate, Center for Behavioral Science Methods Research Report Series (Survey Methodology 2019-06). Washington, DC: U.S. Census Bureau. Available at: http://www.census.gov/content/dam/Census/library/working-papers/2019/adrm/rsm2019-06.pdf (accessed January 2021). Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo