Big Data Vs Small Data – Where Does the Gold Lie?

Data is the new oil of the 21st century. As an increasing number of organisations realise the benefits of taking a data-driven business approach, data is fast becoming one of the most coveted assets of an organisation. As technologies like the Internet of Things take off, the data deluge that we are experiencing today will grow 10 fold by circa 2020. We are looking at data volumes that will amount to 44 zettabytes! This data, called Big Data, can deliver transformational insights to organisations by providing massive quantities of information that can deliver tremendous business value. However, data by itself can hardly deliver the potential benefits it promises. For data to deliver value organisations have to make significant IT infrastructure and IT investments to store and analyze this data. This is not something that every organisation can afford. So, does this mean that those organisations who cannot make these Big Data investments cannot reap the rewards that data promises? Given the technologically advanced world that we are living in, it would be ludicrous to think so.

So what is Small Data?

It’s easy to say in a data-obsessed world all innovations are driven by Big Data. It is only when you peel the onion that you realize that it is the smaller data pieces, data that is readily available and generated every day that can provide timely and meaningful insights to deliver something that can have a profound business impact. With Small Data, organisations can identify causation and get the solutions for unrecognised consumer or business needs. It is hardly a surprise to see that almost all innovations and breakthrough ideas stemmed from studying seemingly innocuous behavior trapped in Small Data that helped in creating revolutionary brands (Think Facebook, GoPro, SnapChat and many more).

The good thing about Small Data is that it is all around us and can be found in social conversations, digital interactions, customer information in CRM systems etc. These interactions hold valuable insights that organisations can leverage to identify and improve performance barriers, measure conversion rates, streamline processes, improve profitability and drive ROI. Small Data looks at utilising everyday data resources efficiently to make timely and competent business decisions that ultimately lead to better profitability by improving efficiencies.

Big Data and Small Data – what’s the difference?

Essentially the difference between Big and Small Data lies in the 3 V’s of data – Volume, Variety, and Velocity. Big Data contains huge volumes of structured and unstructured data and holds the key to uncovering hidden patterns that provide a business benefit by evaluating past performance. Big Data aims to discover correlations in these huge data volumes to uncover patterns that can help businesses make profitable decisions for the future. So, you want to predict the results of the next elections, find out if there is a risk of a pandemic or epidemic, assess where your company should allocate more spends or predict the results of the next Super Bowl or Olympics etc., then you need to look at Big Data. In fact, the Rio Olympics held this year has been touted as the most data-driven Olympics so far.

However, to leverage Big Data, organisations need to invest in server storage and use sophisticated analytics and data mining applications to search for usable data that can be accumulated from different sources such as demographic data, user actions etc. Then a set of complex algorithms has to be applied to the data to process it and display it. The process, needless to say, is detailed, exhaustive and time-consuming.

Small data, on the other hand, requires fewer resources and less robust data mining strategies. Small data provides real-time information and gives organisations the opportunity to adjust their strategies on a more proactive basis for a positive business impact. Small data is more connected to the customer and is all about the end-user. It determines what they need, why they need it, and when they need it. Small data also allows businesses to build collaborative efficiency around the distributed data environment. Small data is all about data democratisation and about building distributed models instead of centralized ones and focusing on small data packages and not monoliths and centralised silos.

Rufus Pollock, Founder, and Co-Director of the Open Knowledge Foundation, states that ‘This (the data) story isn’t about large organisations running parallel software on tens of thousands of servers, but about more people than ever being able to collaborate effectively around a distributed ecosystem of information, an ecosystem of small data”.

The idea of Big Data is compelling and will continue to remain so. It will continue to be a business differentiator and will answer questions that Small Data does not have the power to answer. For example, it is Big Data that is helping Target identify their pregnant customers and Walmart by auto-generating shopping lists for their customers. While Big Data is effectively removing guess work from an organisational conversation, what it cannot do is replace the conversation itself. It is with the help of Small Data that brands can build conversations with their target audience and build tactile interactions that will deliver business value.

As the world becomes more data driven and data dependent, the lines differentiating Big and Small data will have to blur and in order to be ahead of the curve, businesses will have to leverage both efficiently.