<abstract xmlns="http://www.w3.org/1999/xhtml"><p> A source of survey processing error that has received insufficient study to date is the misclassification of open-ended responses. We report on efforts to understand the misclassification of open occupation descriptions in the Current Population Survey (CPS). We analyzed double-coded CPS descriptions to identify which features vary with intercoder reliability. One factor strongly related to reliability was the length of the occupation description: longer descriptions were less reliably coded than shorter ones. This effect was stronger for particular occupation terms. We then carried out an experiment to examine the joint effects of description length and classification “difficulty” of particular occupation terms. For easy occupation terms longer descriptions were less reliably coded, but for difficult occupation terms longer descriptions were slightly more reliably coded than short descriptions. Finally, we observed as coders provided verbal reports on their decision making. One practice, evident in coders’ verbal reports, is their use of informal coding rules based on superficial features of the description. Such rules are likely to promote reliability, though not necessarily validity, of coding. To the extent that coders use informal rules for long descriptions involving difficult terms, this could help explain the observed relationship between description length and difficulty of coding particular terms.</p></abstract>

A source of survey processing error that has received insufficient study to date is the misclassification of open-ended responses. We report on efforts to understand the misclassification of open occupation descriptions in the Current Population Survey (CPS). We analyzed double-coded CPS descriptions to identify which features vary with intercoder reliability. One factor strongly related to reliability was the length of the occupation description: longer descriptions were less reliably coded than shorter ones. This effect was stronger for particular occupation terms. We then carried out an experiment to examine the joint effects of description length and classification “difficulty” of particular occupation terms. For easy occupation terms longer descriptions were less reliably coded, but for difficult occupation terms longer descriptions were slightly more reliably coded than short descriptions. Finally, we observed as coders provided verbal reports on their decision making. One practice, evident in coders’ verbal reports, is their use of informal coding rules based on superficial features of the description. Such rules are likely to promote reliability, though not necessarily validity, of coding. To the extent that coders use informal rules for long descriptions involving difficult terms, this could help explain the observed relationship between description length and difficulty of coding particular terms.

Classifying Open-Ended Reports: Factors Affecting the Reliability of Occupation Codes

Survey Research Center, Institute for Social Research, University of Michigan, PO Box 1248, Ann Arbor, MI 48106, U.S.A., and Joint Program in Survey Methodology, 1218 LeFrak Hall, University of Maryland, College Park, MD 20742, U.S.A.

Cathie Marsh Institute for Social Research, University of Manchester, Humanities Bridgeford Street Building, Manchester M13 9PL, U.K., and Institute for Employment Research, Regensburger Str. 104, Nuremberg 90478, Germany.

Journal of Official Statistics

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

A source of survey processing error that has received insufficient study to date is the misclassification of open-ended responses. We report on efforts to...

Classifying Open-Ended Reports: Factors Affecting the Reliability of Occupation Codes

Published Online: Mar 10, 2016

Page range: 75 - 92

Received: Feb 01, 2014

Accepted: Oct 01, 2015

DOI: https://doi.org/10.1515/jos-2016-0003

Keywords
Survey processing error, coding error, occupational classification

© by Frederick G. Conrad

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Classifying Open-Ended Reports: Factors Affecting the Reliability of Occupation Codes

Published Online: Mar 10, 2016

Page range: 75 - 92

Received: Feb 01, 2014

Accepted: Oct 01, 2015

DOI: https://doi.org/10.1515/jos-2016-0003

KeywordsSurvey processing error, coding error, occupational classification

© by Frederick G. Conrad

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Keywords
Survey processing error, coding error, occupational classification