Exploration of End-to-End Big Data Engineering and Analytics

Abstract

The emergence of Big Data represents a timely opportunity for industries and service
organizations to generate value and meaningful knowledge or insights from capturing, storing,
managing, and analyzing massive datasets. Most of the commercial value is expected to add to the
data economy in the next years. In recent years, the advent of new sources of data from sensors, social
networks, point of sale, logs, and interactions became an overwhelming opportunity for exploring the
knowledge contained in these data. Traditional processing platforms and Big Data Management
systems were not able to deal with this ever-increasing volume of data. Defining a system with the
capability of handling large and complex data sets has been a challenge of civil engineering,
computer science, business and economics, and many other disciplines.
Big Data is a concept that describes the data and its reflections in business, services, and
management, usually associated with the three properties of high volume, high variety, and high
velocity. The analysis of the Web by tools capable of handling large datasets for analyzing link-based
social networks and web services became an important theme of research and development. Earlier
definitions of these systems stated the four Vs (volume, variety, velocity, and veracity), and research
has also extended this degree of relevance. There is also a definition that stems from the four Vs and
can be stated as: “the Big Data is a data set so large and complex that its creation, processing and
analysis have a low return on investment with the tools and techniques that are on the market”.