Can Big Data deliver Big Insights ?

The other day I met a CIO friend who wanted to discuss a tricky situation in which he had landed; he worked in an industry which was in the thick of being projected as one of the industries that will benefit from investments in Big Data. His CEO wanted him to build a data warehouse to rival some of their global competitors, at least one of which was prominently talked about as the poster boy of Big Data analytics. He was thus under pressure to invest while the rest of his IT budget was under pressure.

Having a keen understanding of technology, his company and the industry, he was a non-believer in the Big Data story; according to him the hype around some of the Big Data insights were not commensurate to the investments made in the overall project. And there was nothing new since the first story broke out of one of the companies having found a use case that conventional technologies would not have delivered. He had many data warehouses and Business Intelligence successes in the past for which he was well known too.

By definition Big Data was all about big data sets that earlier available technologies could not bind together within tolerated elapsed time and budgets. Volume, Variety and Velocity defined Big Data; (Business) Value was added later. The availability of high compute resources and ability to store large volumes of data had made solving some problems easier, faster and cheaper; that is not necessarily success from the capitalized Big Data. It is just that larger data sets were analyzed as compared to the past.

The question at hand that needed an answer was whether he should let go and invest as directed by his CEO or he should help the business with a scalable data warehouse which would deliver immediate value. Is it possible to get started small with Big Data (an oxymoron if there was one) and then work with the business to find the needle (if they wanted to find the needle or a pin) in the haystack; after all Big Data is expected to throw up unknown possibilities by random correlations that human minds are not able to pick.

Big data works on “found” data, i.e. data that you have and complex algorithms which can provide some statistical probabilities. Analysts predict the value that different industries can gain from investments; no one is talking about the real value derived. Governments have been making investments with equal zeal as are large enterprises; the providers and consultants are happy to make hay not just while the sun shines but until by accident they discover a needle in the haystack and make a case study out of it putting pressure on the rest of the gold diggers.

What about the data that you don’t have ? Can you draw negative inferences from Big Data ? For that you have to know what you don’t have ! Can what you have tell you what you don’t ? The answer to that is still to be found; available data in a Big Data repository cannot indicate to what is missing. The concept of “found” data predicates that available data set is the whole universe from which correlations are to be created. And that is where many Big Data implementations are unable to deliver any meaningful insights.

The veracity (the 5^th V) of information in a Big Data store can throw up many false positives which have been the bane of many projects. Data will never be clean unlike conventional data warehouses and the velocity will keep you challenged to move with agility. The ability to come out of the clean and complete data mindset is the beginning of what Big Data may enable. From here to get to Value is a long journey with no near-term goals; if you hit something, consider yourself lucky and celebrate.

My suggestion to my friend was to get started the way he believed he will be able to deliver what the business wanted. Forget the discussion on technology and focus on what matters, insights driven by data. If he can get traction from some CXOs based on the results, no one will grudge whether they came from Big Data or Small Data. The business leader in him understood while the technologist wanted to fight; for his benefit, I hope the business guy prevails.

Data, data everywhere

“They want 78 new reports from the EDW and Big Data which has taken more than a year and a major part of my budget to build ! They already have hundreds from the transactional systems which are printed on reams of paper which no one reads. All the Excel sheets that they were churning out from data dumps from various systems and a bit of external data get into management meetings where everyone has a different number”. Looking at the CIO I sympathized with his predicament, it was a familiar story.

We all have at some time or the other been frustrated with endless requirements of reports and data dumps from all and sundry; lot of effort is spent in analyzing the past and validating hypothesis on what worked or what did not. Requests flow like rainwater on a slope, never ending stream many similar to others from neighbors at workplace not talking to each other. Reports get built for a casual question in a meeting never to be used again; when another one pops up from new quarters, the effort is repeated.

We hear of associations, correlations and insights not possible in the past as we did not know how to combine an apple and pineapple to get a watermelon. Structured data was easy until we started going to multiple sources with limited commonality. Even then with statistical models diving through seas of data, the proverbial needle could be found in the haystack. People buying napkins buy beer, not vice versa; owners of red cars have a higher propensity to be rash drivers, and so on. You could correlate anything to sunspots !

Not too long ago the need to explore unstructured data began and with social media explosion the dimensions for analysis changed. Thus Big Data began its journey to challenge conventional way of looking at data and information. Jumping on the bandwagon the term was hyped by one and all to include variants that stretched imagination. Came along new skills everyone thought were important for the future: Data Scientists and Chief Digital Officer to name a couple; did such a species exist or it was glamorized plain old profiles ?

Moving from hundreds of GB of data to thousands of GB does not make it Big Data. The amount of data being created and added to corporate storage is growing exponentially. Data types are also expanding with technology offering ways to mine it. Dashboards and cubes work well in selected situations, their action-ability is still wishful thinking. Enterprise manager thinking has yet to evolve beyond reports from transactional systems; thus the data scientist continues to remain a glorified report writer.

The CIO narrated his woes which started with the Company Board approving a really large budget and unrealistic expectations from the project they called BBF (Bigger Better Faster). With much fanfare the project was kicked off, many people inducted into the team and a few pretentious youngsters hired to lead them to gaining insights thus far unknown, from this prestigious first of a kind in the industry Big Data project. The CIO kept his reservations to himself knowing his meanderings would not be given a kind ear.

The project team got bigger faster than anyone thought possible; the technology they bought was deemed better than what they had. Everyone loved the progress they made in the initial months. Then started the reality check with the target audience (managers) putting across what they wanted to run their business better, to grow bigger and reach out to customers faster than their competitors. Challenges with technology and data consistency appeared small compared to the change required in the mind set within.

Activity Reports on social media, portal registration, access reports, keyword searches and some more were the peak of expectation. There was no marriage between the old and the new as if they lived in separate worlds. What could have been remained buried somewhere while everyone wanted better and faster transactional or tactical reports. The rich stream of data that could have been big for the business was diverted and converted into wasted effort. In the corporate world, I believe that the overwhelming data deluge is far from being tamed.

Do you know different ?

Feeding the Elephant

Recently I participated in a Big Data conference which boasted of speakers of all shapes and sizes (literally too) from government, global multinationals, large enterprises, to vendors and academicians rounding off the tail. The audience filled the room to the brim with expectations of gaining insights from the deliberations and debate. After all, according to IT research analysts, Big Data is one of the key technology trends on everyone’s agenda and priority list. It’s like if you are not doing it, then you are Jurassic.

The agenda comprised of speakers from all mentioned above; some had done it, some were selling wares with titles containing “Big Data”, a couple of consultants and service providers who offered their “expertise” on the subject, and finally a CIO to provide an enterprise perspective of how are corporates looking at it. All in all it was an eclectic mix which promised to give value for time invested to the organizers and participants. I took up a corner perched at the edge of my seat and watched the proceedings.

Setting the foundation the keynote speaker talked about the concept, progress made by IT companies, known deployments of Big Data by a few FMCG, internet companies, and government agencies. A case study of a potential big data application at a government initiative demonstrated the dimensions of Big Data, i.e. Volume, Variety, Velocity and Value. Everything was going well thus far with the audience – a mix of technology staff, IT students, and some service providers – lapping it up all.

Then events took a turn that changed the atmosphere in the room; everyone sat up awoken from their stupor and peaceful existence in the cushioned chairs. Like falling off a cliff was how a participant described it later; the turmoil changed the agenda and the utterings of future speakers who were cautious in their exultations of Big Data. The speaker exceeded his time; no one interrupted his thought train. He challenged everyone to challenge his hypothesis; none did. He was the CIO talking about relevance to the corporate.

Who needs Big Data ? Where does it fit into the maturity curve of an enterprise using Business Intelligence or Analytics ? How do you partner with business who is still swamped by reports or dashboards at best ? Actionable insights ? When does a data warehouse become inadequate and Big Data become necessary ? Is it about unstructured data only or volume of data or complexity of analysis ? Is analysis of social media tags or text Big Data even when volume is low ? So what is Big Data ?

Consultants and IT companies have developed models and tools respectively to hypothetically help companies mine the sea of data. They have been talking about uses and value across industries based on some assumptions. A few pilots with companies have not empirically demonstrated a correlation between the Big Data analytics and the benefit. Internet companies have used scalable models of their earlier working solutions as they grew; e.g. recommendation engines, product associations, etc. These are not new.

Is it just hype or a technology solution created for specific purposes now being touted as nirvana for all kinds of data problems or analytics that have historically belonged to the data mart or data warehouse ? The CIO challenged the audience to clear their vision, heads, and minds and think rationally on what is the business problem they want to solve before deciding on the tools and technology. The yellow elephant in the room cannot be ignored; its relevance however needs to be established before feeding it.

At the end of the session which led into the lunch break, the CIO was hounded for his contrarian views; everyone wanted a piece of advice and some wanted to debate their conflicts in private. The poor fellow was deprived of lunch with the next session being ringed in. I believe Big Data like any new technology trend needs evaluation in the context of the enterprise’s reality. Is there benefit to customers or employees ? If not why do it ? Like my old CFO friend said “If it makes cents, only then it makes sense !”