Monday, February 04, 2013

Feeding the Elephant

Recently I participated in a Big Data conference which boasted of speakers of all shapes and sizes (literally too) from government, global multinationals, large enterprises, to vendors and academicians rounding off the tail. The audience filled the room to the brim with expectations of gaining insights from the deliberations and debate. After all, according to IT research analysts, Big Data is one of the key technology trends on everyone’s agenda and priority list. It’s like if you are not doing it, then you are Jurassic.

The agenda comprised of speakers from all mentioned above; some had done it, some were selling wares with titles containing “Big Data”, a couple of consultants and service providers who offered their “expertise” on the subject, and finally a CIO to provide an enterprise perspective of how are corporates looking at it. All in all it was an eclectic mix which promised to give value for time invested to the organizers and participants. I took up a corner perched at the edge of my seat and watched the proceedings.

Setting the foundation the keynote speaker talked about the concept, progress made by IT companies, known deployments of Big Data by a few FMCG, internet companies, and government agencies. A case study of a potential big data application at a government initiative demonstrated the dimensions of Big Data, i.e. Volume, Variety, Velocity and Value. Everything was going well thus far with the audience – a mix of technology staff, IT students, and some service providers – lapping it up all.

Then events took a turn that changed the atmosphere in the room; everyone sat up awoken from their stupor and peaceful existence in the cushioned chairs. Like falling off a cliff was how a participant described it later; the turmoil changed the agenda and the utterings of future speakers who were cautious in their exultations of Big Data. The speaker exceeded his time; no one interrupted his thought train. He challenged everyone to challenge his hypothesis; none did. He was the CIO talking about relevance to the corporate.

Who needs Big Data ? Where does it fit into the maturity curve of an enterprise using Business Intelligence or Analytics ? How do you partner with business who is still swamped by reports or dashboards at best ? Actionable insights ? When does a data warehouse become inadequate and Big Data become necessary ? Is it about unstructured data only or volume of data or complexity of analysis ? Is analysis of social media tags or text Big Data even when volume is low ? So what is Big Data ?

Consultants and IT companies have developed models and tools respectively to hypothetically help companies mine the sea of data. They have been talking about uses and value across industries based on some assumptions. A few pilots with companies have not empirically demonstrated a correlation between the Big Data analytics and the benefit. Internet companies have used scalable models of their earlier working solutions as they grew; e.g. recommendation engines, product associations, etc. These are not new.

Is it just hype or a technology solution created for specific purposes now being touted as nirvana for all kinds of data problems or analytics that have historically belonged to the data mart or data warehouse ? The CIO challenged the audience to clear their vision, heads, and minds and think rationally on what is the business problem they want to solve before deciding on the tools and technology. The yellow elephant in the room cannot be ignored; its relevance however needs to be established before feeding it.

At the end of the session which led into the lunch break, the CIO was hounded for his contrarian views; everyone wanted a piece of advice and some wanted to debate their conflicts in private. The poor fellow was deprived of lunch with the next session being ringed in. I believe Big Data like any new technology trend needs evaluation in the context of the enterprise’s reality. Is there benefit to customers or employees ? If not why do it ? Like my old CFO friend said “If it makes cents, only then it makes sense !”


  1. yes, right. Big data has to be in context of needs and benefits of an organization, and its context will change based on objectives and needs of organization.
    concept and clarity will evolve along with technical platforms to support it and make this analytic easier and feasible.
    it is about large size of data, but certainly not only about that. it is about compiling structured, non structured, media and any other data that organization needs or generates, consolidating, quantifying, contextualizing and aggregating this data sources to help organizations with actionable insights. it will involve mix of textual / context analysis, various feeders to various data sources, running and generating statistical and complex algorithms, ETL, data warehousing and visualization / front end analytics. Mix of various technologies and tools, based on what is required.
    How it can be done will be based on what needs to be done, and as you rightly said above, what needs to be done depends on organization objectives, needs and priorities.....

  2. Very relevant article and posting a para worth of comment wont do justice to the topic but I am making an attempt.
    Big Data is coming into picture because of the Volume of data and the velocity with which it is being created and ofcourse the variety of it -- all together creates complexity for business in terms of adding value to the customer. Big Data mean different thing to different vendors. It is about business intelligence, it is about data sprawl, data management, data processing/access, data storage, monitoring the performance of application creating these data, data security etc -- there are various aspects of it.
    However one must have a business imperative to take this journey rather than making it a technology pursuit.

    This is an interesting read which I had read some time back:

    For example: I want to filter all the comments on FaceBook and Twitter that has a keyword "Cipla", coorelate and categorize them to find opportunity for innovation in Customer Service and Custsat Index. If this is a business imperative then it could be a Big Data project because it has volume, velocity as well as variety.

  3. Interesting, observations but the fact remains the fear of being left behind guides the Complex IT Organizations. Rather, than understanding of Business Relevance. One of the company that I own is into Bi, BW, DWH but find it really surprising that clients or techie's have ignored the basic tenets of BI. That is is not a singular reality but it is driven by Multi Variables. I have yet to see a Multivariate technology or Adaptive Analytics working for Organization, that had the capability in making the difference.
    Now, you have Big Data or Hadoop but pertinent question is what is in it for the Organization to look at it or just some techie whims and fancies loading the Organization additional cost.

  4. Anonymous4:17 PM

    More technology , but with little clear benefit , how many companies have obtained any benefit from BI in general , after multi million investments - ROI on most BI projects close to zero , why will big data be any better , sure there are some specific usages such as in R&D in Pharma but in general lets get a return on BI before blowing big bucks on yet another hunch