Understanding Big Data Processing and Analytics: The Top 6 Questions Answered
Big Data changes our way of thinking – it has already begun – the Big Data also transforms the way we live. It also affects our work habits. Decision-making is democratized in the workplace and many self-proclaimed “experts” see their expertise challenged in the light of the Big Data whose prospects and predictions are far more accurate.
How is Big Data a revolution?
Big data is a revolution in the way that we humans understand and give meaning to the reality that surrounds us. In the past, due to the constraints of data collection and analysis, we had to study small subsets, samples of reality, and then, extrapolate the set to “everything”, hoping that this “everything” works identically to the sample. But it was limited to the use of a number of methodologies, very careful sampling, the need to know what question we wanted to answer before collecting the data, and did not allow us to focus on details.
In addition, we focused on the quality of the data rather than its size. Our ability to collect and analyze a large number of data evolves and we are able to discern reality “on a scale”, in all its complexity and dynamism. In a sense, this movement resembles the Newtonians’ passage to the Einsteinian world of gravity – a transformation in the way we give meaning to the world around us. This revolution affects all aspects of life, society and all sectors of our economy: politics and citizenship via Open Data platforms; Urbanization, Energy through various connected sensors; e-business through access to a growing number of consumer data.
Can we predict the future with these data?
To some extent, we can talk about predictions. We have done this before, we do today by extrapolating the past or present to the future. It works well if things do not change radically. Even if these expectations of data tend to tell us the opposite, the individual does not change so easily.
Is there a limit to the amount of data that can be created?
No, for now, there is no limit in sight. Of course, there is a physical barrier as to the amount of information that can be stored in the universe. But this barrier represents an order of magnitude that is so much greater than what we are currently collecting and analyzing, that it offers a very important growth perspective. Moreover, the drop in collection and storage costs associated with better analytical tools makes the Big Data less and less expensive and more and more simple to use.
What is an “algorithmist” and a data scientist?
The data scientists make quantitative analysis. “Algorithmists” are quants who have joined a specialized profession and act as independent audits of Big Data analysis. The relationship between data scientists and “algorithmists” is a bit like having studied law and being a lawyer or notary.
What new skills does Big Data require?
Big data requires capacities for data collection and sharing (for example, through new sensors, retrieving them over the Internet or targeting the value in data owned by others but licensable), skills in analyzing data (well beyond simple statistics) and understanding the limits of these analysis. These new skills are already being taught throughout the world in leading-edge academic programs that also include training for quantitative analysts.
What is the value of these data?
Previously, the value of the data was limited to the purpose for which it was collected. In the future, this value will reside in the uses and re-uses that may be made of these data, even if this was not apparent at the time of their collection. The value of the data thus increases with our ability to extract this value.
