<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns whether web questionnaires lead to mode-specific measurement bias. We argue that the magnitude of measurement bias strongly depends on the content of a variable. Based on the Luxembourgish Labour Force Survey, we investigate differences between web and telephone data in terms of objective (i.e., Employment Status) and subjective (i.e., Wage Adequacy and Job Satisfaction) variables. To assess whether differences in outcome variables are caused by sample composition or mode-specific measurement bias, we apply a coarsened exact matching that approximates randomized experiments by reducing dissimilarities between web and telephone samples. We select matching variables with a combination of automatic variable selection via random forest and a literature-driven selection. The results show that objective variables are not affected by mode-specific measurement bias, but web participants report lower satisfaction-levels on subjective variables than telephone participants. Extensive supplementary analyses confirm our results. The present study supports the view that the impact of survey mode depends on the content of a survey and its variables.</p></abstract>

Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns whether web questionnaires lead to mode-specific measurement bias. We argue that the magnitude of measurement bias strongly depends on the content of a variable. Based on the Luxembourgish Labour Force Survey, we investigate differences between web and telephone data in terms of objective (i.e., Employment Status) and subjective (i.e., Wage Adequacy and Job Satisfaction) variables. To assess whether differences in outcome variables are caused by sample composition or mode-specific measurement bias, we apply a coarsened exact matching that approximates randomized experiments by reducing dissimilarities between web and telephone samples. We select matching variables with a combination of automatic variable selection via random forest and a literature-driven selection. The results show that objective variables are not affected by mode-specific measurement bias, but web participants report lower satisfaction-levels on subjective variables than telephone participants. Extensive supplementary analyses confirm our results. The present study supports the view that the impact of survey mode depends on the content of a survey and its variables.

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Journal of Official Statistics

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Web questionnaires are increasingly used to complement traditional data collection in mixed mode surveys. However, the utilization of web data raises concerns...

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Published Online: Mar 13, 2021

Page range: 213 - 237

Received: Apr 01, 2019

Accepted: Sep 01, 2020

DOI: https://doi.org/10.2478/jos-2021-0009

Keywords
Web survey, telephone survey, mode effects, coarsened exact matching, measurement bias

© 2020 Joachim Schork et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Survey Mode Effects on Objective and Subjective Questions: Evidence from the Labour Force Survey

Published Online: Mar 13, 2021

Page range: 213 - 237

Received: Apr 01, 2019

Accepted: Sep 01, 2020

DOI: https://doi.org/10.2478/jos-2021-0009

KeywordsWeb survey, telephone survey, mode effects, coarsened exact matching, measurement bias

© 2020 Joachim Schork et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
Web survey, telephone survey, mode effects, coarsened exact matching, measurement bias