Introduction to Big Data : storage, big analytics and data mining

The aim of this article is to define the term Big Data, to explain the economic and societal issues involved, and to introduce the various methods and techniques involved. Although the origin of the term Big Data is controversial, it is believed to have appeared in 1997, and its recommended official French translation is "mégadonnées", although it is sometimes referred to as "données massives". Recently, the spread of approaches and applications exploiting neural networks has tended to replace this term in the collective imagination with those of Data science, Artificial Intelligence or Machine Learning, and even if each refers to different fields, the engineering and research questions common to them are numerous: they concern the storage, management, processing, analysis and exploitation (uses) of very large quantities of data, and the opportunities and risks associated with them.

For at least the last thirty years, the amount of data generated has only increased. Currently, we produce an estimated 74 zettabytes of data per year, equivalent to more than 1 GB per hour per inhabitant of the planet. By 2025, this amount is set to more than double . This increase in data affects all sectors, whether scientific, cultural, industrial or financial.

The global average annual growth rate of the market for technology and services around Big Data was estimated at over 30% over the period 2011-2016, and has remained at around 20% ever since. According to an IDC study, this market reached $23.8 billion in 2016 and $90 billion in 2021 for BigData software and other cloud services alone. IDC also estimates that spending on Big Data and business analytics (BDA) solutions in Europe will reach $50 billion by 2022.

In this article, we focus on two major issues associated with megadata: on the one hand, its storage, and on the other, its analysis and exploitation using statistical and machine learning approaches, while identifying the limitations of traditional and historical approaches. Megadata is mainly accompanied by the development of analytical or predictive applications, which process data to make sense of it, classify, search or filter it, or make estimates of future states or values. These analyses are generally referred to as "Big Analytics", and rely on distributed and parallel computing methods, which are often costly in terms of computing time and energy, and...