Our clients occasionally ask us what big data is and how we can use it. We asked our data guy Shaun to explain this for us.

What is big data?

Big data usually means a large data sample that needs advanced statistical analysis techniques, and computing power to process and provide deep analytical insights. The bigger the data, the more computing power needed.

We use big data techniques to analyse customer interactions, everything from phone calls, POS transactions, website visits, and potentially data mining public profiles to build profiles of customers that can be used to optimise marketing, and target only customers who will respond positively.

But it is a fluid term because the size of data sets vary, and with that, the tools that we use vary, and what they tell us varies. The easiest categorisation is to look at Small Data, Big Data, and Huge Data.

Small data

We all work with small data all the time, but generally it is data sets that can be managed and manipulated in Microsoft Excel. Excel has a limit of 1,000,000 (1M) rows, and has limited equations and functions that it can run. The work that we do with Google Analytics would usually be small data, but could be big.

We would use small data when we are looking at transactions and orders for an online store. For a store with hundreds of transactions a day, a few tens of thousands of transactions a year with a few thousand customers. With some simple manipulations, we would be able to show graphs explaining insights such as average order values, demographic information, return customer volume and other details. This is usually sufficient to explain to decision makers what is going on with their business.

If you would like to get these in your inbox weekly, plus our extra commentary, please subscribe on our connect page.
Connect Now!

Big data.

Analysis of big data requires you to use complex tools to leverage the full computing power of a PC (or mac) to complete analyses, and predict customer behaviour. The biggest opportunity we see initially is to help our clients identify common behaviour among users prior to taking a certain action, and then prompting the user to take a certain action.

As an example, we may be able to identify loyal customers that are most likely to leave in the next couple of months. This would allow you to take action (call the customer or email) to re-engage them. For example, the data may show that 30-35 year old Males are unlikely to shop if they don’t receive an email every two weeks. With this information we would intervene to keep them as customers. This would still probably be small data, we want to go further; if we can analyse how each customer uses a site, gain access to customer service phone data, individual data on whether someone opened an email, we can collate this and predict all sorts of future behaviour. We could run data, and predict that a particular group of people would respond best to a promo, and just target it at them. This is often called predictive analytics.

Running this data requires the full processing power of a computer, and can take from minutes to hours to process.

Huge Data

It’s the same as big data, but the processing power needed is more than a home computer or desktop can handle in a reasonable period of time. In these cases, we can hire processing power in large servers to process the data for us.

Let us know if you have any big data projects in the works, want to make the most of the small data you have available.