<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Record linkage addresses the problem of identifying pairs of records coming from different sources and referred to the same unit of interest. Fellegi and Sunter propose an optimal statistical test in order to assign the match status to the candidate pairs, in which the needed parameters are obtained through EM algorithm directly applied to the set of candidate pairs, without recourse to training data. However, this procedure has a quadratic complexity as the two lists to be matched grow. In addition, a large bias of EM-estimated parameters is also produced in this case, so that the problem is tackled by reducing the set of candidate pairs through filtering methods such as blocking. Unfortunately, the probability that excluded pairs would be actually true-matches cannot be assessed through such methods.</p><p>The present work proposes an efficient approach in which the comparison of records between lists are minimised while the EM estimates are modified by modelling tables with structural zeros in order to obtain unbiased estimates of the parameters. Improvement achieved by the suggested method is shown by means of simulations and an application based on real data.</p></abstract>

Record linkage addresses the problem of identifying pairs of records coming from different sources and referred to the same unit of interest. Fellegi and Sunter propose an optimal statistical test in order to assign the match status to the candidate pairs, in which the needed parameters are obtained through EM algorithm directly applied to the set of candidate pairs, without recourse to training data. However, this procedure has a quadratic complexity as the two lists to be matched grow. In addition, a large bias of EM-estimated parameters is also produced in this case, so that the problem is tackled by reducing the set of candidate pairs through filtering methods such as blocking. Unfortunately, the probability that excluded pairs would be actually true-matches cannot be assessed through such methods.The present work proposes an efficient approach in which the comparison of records between lists are minimised while the EM estimates are modified by modelling tables with structural zeros in order to obtain unbiased estimates of the parameters. Improvement achieved by the suggested method is shown by means of simulations and an application based on real data.

An Improved Fellegi-Sunter Framework for Probabilistic Record Linkage Between Large Data Sets

Journal of Official Statistics

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Record linkage addresses the problem of identifying pairs of records coming from different sources and referred to the same unit of interest. Fellegi and...

An Improved Fellegi-Sunter Framework for Probabilistic Record Linkage Between Large Data Sets

Published Online: Dec 10, 2020

Page range: 803 - 825

Received: Jul 01, 2018

Accepted: Jul 01, 2020

DOI: https://doi.org/10.2478/jos-2020-0039

Keywords
Structural zeros, robustness, EM algorithm, blocking

© 2020 Marco Fortini, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

An Improved Fellegi-Sunter Framework for Probabilistic Record Linkage Between Large Data Sets

Published Online: Dec 10, 2020

Page range: 803 - 825

Received: Jul 01, 2018

Accepted: Jul 01, 2020

DOI: https://doi.org/10.2478/jos-2020-0039

KeywordsStructural zeros, robustness, EM algorithm, blocking

© 2020 Marco Fortini, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
Structural zeros, robustness, EM algorithm, blocking