Feeding the Elephant

Recently I participated in a Big Data conference which boasted of speakers of all shapes and sizes (literally too) from government, global multinationals, large enterprises, to vendors and academicians rounding off the tail. The audience filled the room to the brim with expectations of gaining insights from the deliberations and debate. After all, according to IT research analysts, Big Data is one of the key technology trends on everyone’s agenda and priority list. It’s like if you are not doing it, then you are Jurassic.

The agenda comprised of speakers from all mentioned above; some had done it, some were selling wares with titles containing “Big Data”, a couple of consultants and service providers who offered their “expertise” on the subject, and finally a CIO to provide an enterprise perspective of how are corporates looking at it. All in all it was an eclectic mix which promised to give value for time invested to the organizers and participants. I took up a corner perched at the edge of my seat and watched the proceedings.

Setting the foundation the keynote speaker talked about the concept, progress made by IT companies, known deployments of Big Data by a few FMCG, internet companies, and government agencies. A case study of a potential big data application at a government initiative demonstrated the dimensions of Big Data, i.e. Volume, Variety, Velocity and Value. Everything was going well thus far with the audience – a mix of technology staff, IT students, and some service providers – lapping it up all.

Then events took a turn that changed the atmosphere in the room; everyone sat up awoken from their stupor and peaceful existence in the cushioned chairs. Like falling off a cliff was how a participant described it later; the turmoil changed the agenda and the utterings of future speakers who were cautious in their exultations of Big Data. The speaker exceeded his time; no one interrupted his thought train. He challenged everyone to challenge his hypothesis; none did. He was the CIO talking about relevance to the corporate.

Who needs Big Data ? Where does it fit into the maturity curve of an enterprise using Business Intelligence or Analytics ? How do you partner with business who is still swamped by reports or dashboards at best ? Actionable insights ? When does a data warehouse become inadequate and Big Data become necessary ? Is it about unstructured data only or volume of data or complexity of analysis ? Is analysis of social media tags or text Big Data even when volume is low ? So what is Big Data ?

Consultants and IT companies have developed models and tools respectively to hypothetically help companies mine the sea of data. They have been talking about uses and value across industries based on some assumptions. A few pilots with companies have not empirically demonstrated a correlation between the Big Data analytics and the benefit. Internet companies have used scalable models of their earlier working solutions as they grew; e.g. recommendation engines, product associations, etc. These are not new.

Is it just hype or a technology solution created for specific purposes now being touted as nirvana for all kinds of data problems or analytics that have historically belonged to the data mart or data warehouse ? The CIO challenged the audience to clear their vision, heads, and minds and think rationally on what is the business problem they want to solve before deciding on the tools and technology. The yellow elephant in the room cannot be ignored; its relevance however needs to be established before feeding it.

At the end of the session which led into the lunch break, the CIO was hounded for his contrarian views; everyone wanted a piece of advice and some wanted to debate their conflicts in private. The poor fellow was deprived of lunch with the next session being ringed in. I believe Big Data like any new technology trend needs evaluation in the context of the enterprise’s reality. Is there benefit to customers or employees ? If not why do it ? Like my old CFO friend said “If it makes cents, only then it makes sense !”