When Big Data Isn’t Enough for Small Minds

I’ve been a Business Intelligence practitioner for a long time. Sometimes, the endless deliberating of ‘new’ technology or big data strategies can simply mask the simple fact that people are vacillating over making a decision. I see this thread when participating in conversations about ‘big data’, for example; as if having more data, is going to make things better.

This premise might be fine, if it wasn’t the case that organisations can sometimes ignore the data that is under their noses. This is evident when you see that the ‘Cassandra’ types of people in the organisation are ignored when they point to new evidence, which flies in the face of current understanding.

I’ve often wondered why existing data can be ignored, as if having more data or new technology will make the business problems go away. I’ve concluded that there are various reasons. Please humour me whilst I give you an example, which illustrates the point.

In the 1950s, a fantastic Doctor called Alice Stewart, from Yorkshire in England, researched childhood cancers, and it was noted that the children were from affluent families rather than poorer backgrounds. Dr. Stewart had one attempt at questioning the bereaved parents of the children who had died, and she collected a large amount of data from them about anything and everything that might give a clue to the cause of the childhood cancers.

When Dr. Stewart got the data back – the results were staggering, and showed an insight which many doctors could only dream of experiencing. The results showed that, with a ratio of 2:1, the mothers had had an X-Ray when they were expecting the child, who had gone on to suffer cancer and die at an early age. Dr. Stewart published her preliminary findings and her data in 1956. During the 1970s, until the point that the use of X-rays during pregnancy was curtailed, one child a week died in the United States of America and in the UK, in this horrible way.

It is sobering to realise that, despite the fact that Stewart’s data was freely and openly available, it was over two decades before the British and American Medical Associations accepted her findings. Her data was directly oppositional to their experience that X-Rays can – and do – save lives.

For me, this sad case demonstrates that simply having data available, openly, freely, and in volumes, isn’t enough. Big Data isn’t going to be enough to get over closed minds, or even small minds, who cannot see or accept the evidence in front of them. This is why I’m passionate about Data Visualisation – I want people to ‘see’ the results, to ‘see’ their data, and to think better and more clearly about the evidence before them.

I hope that Data Visualisation can help the ‘Cassandra’ types in the work place. The Cassandras will tell you the truth, based on the data, given the tools to do so. They are often found in the middle tiers, and are close enough to the data to see the patterns and the trends.

They can help you to ‘see’ your data, and the evidence which disrupts your organisation – regardless of its size.

This blog is a preview note of my SQLPass presentations at the Summit in Seattle, in November 2013. I hope to see you there!