<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>In this article, we evaluate how the analysis of open-ended probes in an online cognitive interview can serve as a metric to identify cases that should be excluded due to disingenuous responses by ineligible respondents. We analyze data collected in 2019 via an online opt-in panel in English and Spanish to pretest a public opinion questionnaire (n = 265 in English and 199 in Spanish). We find that analyzing open-ended probes allowed us to flag cases completed by respondents who demonstrated problematic behaviors (e.g., answering many probes with repetitive textual patterns, by typing random characters, etc.), as well as to identify cases completed by ineligible respondents posing as eligible respondents (i.e., non-Spanish-speakers posing as Spanish-speakers). These findings indicate that data collected for multilingual pretesting research using online opt-in panels likely require additional evaluations of data quality. We find that open-ended probes can help determine which cases should be replaced when conducting pretesting using opt-in panels. We argue that open-ended probes in online cognitive interviews, while more time consuming and expensive to analyze than close-ended questions, serve as a valuable method of verifying response quality and respondent eligibility, particularly for researchers conducting multilingual surveys with online opt-in panels.</p>
</abstract>

In this article, we evaluate how the analysis of open-ended probes in an online cognitive interview can serve as a metric to identify cases that should be excluded due to disingenuous responses by ineligible respondents. We analyze data collected in 2019 via an online opt-in panel in English and Spanish to pretest a public opinion questionnaire (n = 265 in English and 199 in Spanish). We find that analyzing open-ended probes allowed us to flag cases completed by respondents who demonstrated problematic behaviors (e.g., answering many probes with repetitive textual patterns, by typing random characters, etc.), as well as to identify cases completed by ineligible respondents posing as eligible respondents (i.e., non-Spanish-speakers posing as Spanish-speakers). These findings indicate that data collected for multilingual pretesting research using online opt-in panels likely require additional evaluations of data quality. We find that open-ended probes can help determine which cases should be replaced when conducting pretesting using opt-in panels. We argue that open-ended probes in online cognitive interviews, while more time consuming and expensive to analyze than close-ended questions, serve as a valuable method of verifying response quality and respondent eligibility, particularly for researchers conducting multilingual surveys with online opt-in panels.

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Journal of Official Statistics

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

In this article, we evaluate how the analysis of open-ended probes in an online cognitive interview can serve as a metric to identify cases that should be...

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Published Online: Sep 12, 2022

Page range: 793 - 822

Received: Jan 01, 2021

Accepted: Mar 01, 2022

DOI: https://doi.org/10.2478/jos-2022-0035

Keywords
Data quality, cognitive interviews, open-ended probes, online opt-in panels

© 2022 Yazmín García Trejo et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Published Online: Sep 12, 2022

Page range: 793 - 822

Received: Jan 01, 2021

Accepted: Mar 01, 2022

DOI: https://doi.org/10.2478/jos-2022-0035

KeywordsData quality, cognitive interviews, open-ended probes, online opt-in panels

© 2022 Yazmín García Trejo et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
Data quality, cognitive interviews, open-ended probes, online opt-in panels