Detection and correction of data quality problems with machine learning

Significant progress has been made in recent years in the design of tools to automate the evaluation, monitoring and improvement of data quality, thanks in particular to technological advances in Artificial Intelligence, and in particular machine learning (ML – Machine Learning). Machine learning techniques have been made operational on a large scale, and are now widely deployed in all sectors of activity, to automate prediction and classification tasks in decision support for numerous fields of application (health, finance, marketing, etc.). However, the reliability of these methods' results remains highly dependent on the quality of the input data for the learning models. Data is often imperfect, and optimal data quality is rarely achieved. Thus, two complementary approaches are commonly proposed: one from the data management research community, aimed at correcting data upstream of analysis chains (by cleaning or repairing data), and the other from the community of learning researchers and practitioners (data scientists), aimed at developing models that are more robust to noise and more efficient, with greater emphasis on transforming and preparing data for a particular predictive task.

For decades, for the data management community, data cleansing has consisted in correcting and transforming data using declarative ETL (Extraction-Transformation-Loading) approaches , detecting inconsistencies in relational databases in the form of constraint violations, to "repair" them and to propose solutions, often theoretical, enabling reasoning from inconsistent data, querying it, verifying and satisfying integrity constraints , discovering functional dependencies or business rules...