Practical Examples of Big Data

13th March, 2014

Whilst Big Data continues to dominate the minds of the market research industry, I thought it would be interesting in a follow up to our previous posting to look at some of the practical applications of big data to ascertain where you may already be encountering its uses.

First of all, we should be clear. Big data is and will continue shaping our lives through analysis of large data sets that hopefully conclude with streamlining of important services and day to day tasks. One of the biggest fears for big data is removal of personal liberty from decision making; machines will be able to predict our movements, thoughts and actions before we do – and much the better in some cases. When these decisions concern traffic flow, disease management and hospital efficiency few could argue against utilising data to improve the public environment. And as we will see, big data can have brilliant but unintended effects.

So we can see that IBM has taken big data and analytic into disease management. Utilising data in this manner is not entirely new, even back to the Broad Street Cholera Outbreak of 1854 with John Snow plotting his data set over a map to discern the location of the infected well, famously breaking the handle from the water pump to see an immediate reduction in cholera rates in the area. IBM is taking up the mantle by using big data to potentially predict outbreaks of deadly malaria and dengue fever. As outbreaks of dengue fever spread due to increased numbers of global mosquitos, and malaria continues to kill million every year, IBM’s Spatio Temporal Epidemiological Modeller (STEM) allow for quick combinations of incoming data to be correlated with existing disease data, constantly creating and maintaining an important disease database.

And schemes such as this could be further enhanced through data-sharing with search engines such as Google, who can somewhat accurately, based on search term volume, begin to accurately project outbreaks in specific locations. So where in the past it could have taken years to correlate specific climatic or social changes for disease management it can now be quickly managed to hopefully save some lives. It is important to continue to recognise the growing importance of public facing big data analysis and improving on these already enhanced offerings.

Big data has always maintained an integral part of stock market dealings. As the system becomes increasingly automated and as its algorithms learn profitable patterns through continual data analysis we must consider if signing over this much power to a global computer network is wise. For as the system can be used to make extremely profitable deals it is also sensitive to human interaction and actively monitors social network feeds for indicators of coming boom or bust moments. Such was the case when the Associated Press Twitter feed was hacked in April, 2013, informing us that President Obama had been injured in two White House explosions. The offending Tweet was quickly deleted and various press meetings announced that no harm had come to Mr Obama.

But the stock market network quickly responded and began to sell shares causing an immediate 0.9% slump in the S&P 500 – enough to wipe out $120bn in company values. Unsurprisingly the market quickly recovered but it did enough to highlight problematic dependencies on big data stock analysis. Without a human intervention, global share prices will remain susceptible to volatile market actions picked up by high frequency trading software designed to comb SEC reports, social networks, industrial filings and many other data sources to complete automated deals. The ease with which the network could crash is startling.

And where the market could crash, there is something that constantly affects all of us: banking fraud. It costs all of us, including those at the top so it is no wonder that large organisations are taking note of big data analysis. It may seem surprising that institutions that hold mountains of data concerning individuals worldwide are not already utilising big data analytics but it seems there is some uncertainty about how to best approach the issue, as well as what data can be used. Fraud prevention methods could be greatly improved by monitoring established behavioural patterns and learning to sight any deviation from the norm, improving response times to hacks and external attacks.

As well as this, individual customer protection could be improved by monitoring the geographic locations of online banking logins, whether by desktop or mobile, by times of day – any number of distinguishing factors that could enable a closer grip on online banking security through continuous big data analysis. The banking sector, however, must realise that to fully realise the benefits of big data, the industry must become more data-focused and follow that path without detracting or removing data that does not fit the bill or give them the answer they wanted. Unfortunately, banks are regularly subject to large scale data breach attempts, so safeguarding individual’s data would proceed to the top of the protection pile.

Slightly less world changing or market crashing is the offering from online and mobile music service Shazam which analyses the millions of searches made each day on its network in an attempt to predict the ‘next big thing’ in music. Shazam will utilise its incoming data to guide its musical output as well as assist in the production and promotion of these artists. In previous years the service accurately predicted that Lana del Rey and French Montana would come to the fore in 2012 and 2013 respectively and they have compiled a new list for the coming year (N.B. don’t worry if you haven’t heard of any of them – they will be making the rounds later this year and early next. I have only heard of Action Bronson, and that’s as someone who indulges in hip hop) which is likely to again prove true. This form of big data analysis follows the more traditional route we see, where consumer behaviour is monitored by the data collector to maintain their ever present market share and in most cases, boost profitability.

Clearly big data has a big future and it should go without saying that the applications are vast and whilst not all directly affect customer behaviour, all require it to begin moving. What is clear is that as multitudes of companies begin to utilise the information at their disposal a greater number of talented big data analysts will be required to process the information into vivid fact for those to act upon. And as the sheer volume of date expands at roughly 2.5 exabytes per day, the world is going to need a mass of individuals to battle the tide.

Because we can have all the data we want, but without someone to analyse it, that’s all it is. Data.