Big Data – Evolution & Introduction

Introduction: What is Big Data

So what is Big Data? Well, there are several definitions but the most prominent one, one according to https://www.newdata.ai/, is that Big Data is the notion that the world has reached a point where there is TONNE of data of available. More data than the average computer can process. The advent of new technology allows us to collect, store and analyse these vast amounts of data. This combination - the availability of massive amounts of data & the ability to process it is called “Big Data”.

What’s the point? Well, the analysis of all this data allows businesses to improve their profit margins, decrease their risk portfolio , increase employee productivity and more. Big Data is changing how businesses operate.

Evolution of Big Data

Traditionally, society has always been data-hungry. But the amount of data that could be stored and processed was limited by technology & cost. The cost of storing 1GB of Data in 1981 was ~700,000$ .  Today, that number is closer to 0.02$.  

(For more on the implications on this, I recommend reading “Free - Chris Andersen”)

So, as you can see, that upper-limit due to cost of storage is disappearing. Data Storage is cheaper than ever. This is where things get interesting. On one hand we have cheaper-than-ever data, and on the other hand we have more data than ever before. Data is collected from social media (facebook, instagram, tinder etc), mobile phones (gps data, network coverage data), ecommerce platforms, streaming platforms etc. Data is available everywhere.

Data abundance isn’t all that new. But it’s synergy with affordable data storage is what makes it usable. Before, organizations were forced to store only the most important data and throw away the rest. The problem though was that you could unknowingly throw out really important data.

The human mind is fascinating - but it has its limitations. We can see patterns and make sense out of complex data sets. But when the data set gets too large - we struggle.

Now this is where technology like Hadoop, Machine Learning, in-memory Analytics and come into play This new technology allows us to take massives amounts data - that would have otherwise been gibberish to us - and turn it into valuable information. 

​

Resources Referenced

    Lean Analytics - Oreilly Media
   Big Data: Principles and best practices of scalable realtime data systems