Big data will drown you if you let it

You hear a lot about Big Data these days: using powerful computers to collect and analyse information on a massive scale.

There’s no doubt that Big Data can be a very good thing. It’s made our weather forecasts much more accurate; it’s helping scientists to model and predict natural disasters, such as earthquakes and tsunamis; and it’s giving brands the opportunity to understand and connect with customers in a more helpful and personal way.

But too much data can also be a bad thing. It can paralyse decision-making, distract attention and confer an unwarranted legitimacy on bad ideas.

IBM recently estimated that some 2.5 quintillion bytes of information are created every day. That doesn’t even sound like a real number, so perhaps I should tell you that it’s 2,500,000,000,000,000,000 bytes. Or, to put it another way, 350Mb of data for every man, woman and child on the planet. Every day.

I don’t know how long it would take you to fill up 350Mb of spreadsheet, but that sounds like quite a lot of data to me.  And, since I’m pretty sure there are some toddlers and people in the Amazon rainforest who aren’t using up their daily data quota, that means the real number the rest of us are getting through is even larger.

We generate vast amounts of information in almost everything we do every day: from the items on our supermarket checkout bill to web cookies to the CCTV cameras that follow us around town. And, increasingly, that information is being used as the backdrop and justification for nearly everything we do in business.

So, is it helping us to make better decisions? Or is it just creating confusion?

The reality is that, in many cases, the sheer scale of information available to us is overwhelming. We don’t know where to start looking – or where to stop. We often carve out data at random. And, worse, we often draw entirely the wrong conclusions from it.

I once saw the same data used in the same business to tell two totally conflicting stories. Not just opposed, but 100% diametrically opposed. What’s odd about this is that the people on both sides were acting in good faith. They didn’t hide anything. They didn’t make anything up. It’s just that they were both interpreting the data according to what they wanted to hear.

Twenty years ago, information was power. Now we’re drowning in the stuff.

The real power lies in being able to explain what it means.