<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns whether web questionnaires lead to mode-specific measurement bias. We argue that the magnitude of measurement bias strongly depends on the content of a variable. Based on the Luxembourgish Labour Force Survey, we investigate differences between web and telephone data in terms of objective (i.e., Employment Status) and subjective (i.e., Wage Adequacy and Job Satisfaction) variables. To assess whether differences in outcome variables are caused by sample composition or mode-specific measurement bias, we apply a coarsened exact matching that approximates randomized experiments by reducing dissimilarities between web and telephone samples. We select matching variables with a combination of automatic variable selection via random forest and a literature-driven selection. The results show that objective variables are not affected by mode-specific measurement bias, but web participants report lower satisfaction-levels on subjective variables than telephone participants. Extensive supplementary analyses confirm our results. The present study supports the view that the impact of survey mode depends on the content of a survey and its variables.</p></abstract>

Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns whether web questionnaires lead to mode-specific measurement bias. We argue that the magnitude of measurement bias strongly depends on the content of a variable. Based on the Luxembourgish Labour Force Survey, we investigate differences between web and telephone data in terms of objective (i.e., Employment Status) and subjective (i.e., Wage Adequacy and Job Satisfaction) variables. To assess whether differences in outcome variables are caused by sample composition or mode-specific measurement bias, we apply a coarsened exact matching that approximates randomized experiments by reducing dissimilarities between web and telephone samples. We select matching variables with a combination of automatic variable selection via random forest and a literature-driven selection. The results show that objective variables are not affected by mode-specific measurement bias, but web participants report lower satisfaction-levels on subjective variables than telephone participants. Extensive supplementary analyses confirm our results. The present study supports the view that the impact of survey mode depends on the content of a survey and its variables.

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Journal of Official Statistics

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns...

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Publié en ligne: 13 mars 2021

Pages: 213 - 237

Reçu: 01 avr. 2019

Accepté: 01 sept. 2020

DOI: https://doi.org/10.2478/jos-2021-0009

Mots clés
Web survey, telephone survey, mode effects, coarsened exact matching, measurement bias

© 2020 Joachim Schork et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Publié en ligne: 13 mars 2021

Pages: 213 - 237

Reçu: 01 avr. 2019

Accepté: 01 sept. 2020

DOI: https://doi.org/10.2478/jos-2021-0009

Mots clésWeb survey, telephone survey, mode effects, coarsened exact matching, measurement bias

© 2020 Joachim Schork et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Mots clés
Web survey, telephone survey, mode effects, coarsened exact matching, measurement bias