Data quality

Problems with the quality of data stored in databases and data warehouses are endemic to all types of data (structured or unstructured) and in all fields of application: Web data, government, commercial, industrial or scientific data. These include data errors, duplications, inconsistencies, missing, incomplete, uncertain, outdated, aberrant or unreliable values. The consequences of non-quality (or poor quality) data on decision-making, and the financial costs involved, are considerable: the Gartner Institute estimates that over 25% of critical data in the world's largest companies is inaccurate. Data quality problems cost the global economy millions every year. As a result, in most development projects focused on data use and analysis, data cleansing accounts for between 30% and 80% of the overall budget and development time dedicated to improving data quality, rather than to building the system or software .

In the era of "Big Data", the sources of information available are multiplying, and the volumes of data potentially accessible are increasing exponentially. Data quality and, more broadly, the quality of information and its veracity have taken on major importance, not only within companies and in the academic world, but also for the general public, the primary consumer and producer of online information. It has become essential to know the quality of the data produced and used, both to adapt its use and to prevent it from deteriorating.

Continuously assessing the quality of data stored in information systems, databases and data warehouses, or on the Web, has become crucial, as it is :

• provide users with objective measures and critical expertise of data quality that can be used for decision-making;

• enable them to put the trust they might place in data into perspective, so that they use or analyze it with care.

If analysis and decision-making can be based on data that is inaccurate, incomplete, ambiguous and of poor quality, then we can question the meaning to be given to these results, and rightly question the quality of the knowledge thus "elaborated".

This article presents a range of solutions to the many problems associated with data quality. The aim of this summary is...