Generalise at your own risk

26th July, 2010

I enjoyed this blog by Conversition Strategies in response to the New York Times article, “Tracking the National Mood Through Twitter.”

It challenges how the views of the Twitterati are not necessarily a proxy for the view of all Internet users as a whole. *

The NYT article stated that “by gauging the mood of messages on Twitter, a group of researchers from the Northeastern University College of Computer and Information Sciences, along with researchers from Harvard Medical School, set out to determine how happy or sad Americans are at different times of the day and week.”

An interesting and extensive exercise examining 300 million tweets over 3 years, and to all intent and purposes is an exercise done well.

à la ‘peer review’, after examining thousands of Twitter-based comments versus other Internet comments relating to the iPad and gauging each on a scale of ‘happiness’, Conversition Strategies reasonably conclude that Twitter users do not necessarily represent all Internet users let alone represent a whole population.  Whilst it is reasonable that moods of people could be predicted by computer algorithms, Twitter as a data source is not simply and instantly predictive of a general population.

It seems to me that there are potentially so many leaps of faith in trying to express the mood of a nation by examining Twitter, it’s like Chinese Whispers for scientists.

In market research there are always margins of error.  It’s why need to understand how representative our samples are, and why we quote confidence intervals, test significance of small differences and often find ourselves delivering prose with disclaimers as long as the M1.

Sometimes the results are just not all that we think they are, or as profound as one would hope.  Sometimes we need to reign in the conclusions our clients want to draw in their quest to suggest differentiators as they strategise.  Sometimes we just have to point out the shades of grey and that we need to look elsewhere.

* P.S.  Thanks to @lovestats (Annie Pettit) who by re-tweeting the blog brought my attention to it… if you want to know what’s happening in the market research world and be happier for it, then it’s obvious that you must be on Twitter and follow hashtags: #mrx #marketresearch #esomar…