Published Online: May 24, 2023
Page range: 43 - 51
DOI: https://doi.org/10.2478/ijanmc-2022-0026
Keywords
© 2022 Yao Feng et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
In the information age, with the continuous development of Internet technology, information data occupies every field of contemporary society. The development of the big data age makes these data more and more prominent. While users read the information they need from these massive data, data quality has also become a concern of users. A large number of data are preprocessed before data analysis, such as some duplicate values, missing values deal with inaccurate and other abnormal data, and filter the data through the data cleaning system to improve the standardization of the data, so as to improve the analysis efficiency of the data, reduce some unnecessary expenses, and save time and effort. The data cleaning system in this paper is implemented based on flash framework. Taking Python as the main language for data cleaning, technical cleaning and standard integration are carried out for some structural problems, duplication problems and missing problems of some different source data. Through the processing of abnormal data, the data quality and data analysis efficiency are greatly improved.