Sick IT, Healthy Life

I have been sick for most of the week; on Monday there was a niggling pain which I hoped would go away like it did most times. By mid-week it had aggravated considering I ignored not just the pain but also the cause which I was not consciously aware of. Once the connection was made I aggressively tried to remedy the situation, except that it was out of control now and needed expert attention. On Thursday the Doctor looked gravely, a long prescription and pronounced that it required a specialist to treat it.

The specialist sympathized and made it look innocuously simple to fix; for him it was a routine escalation to manage. On Friday he did what he was good at and fixed the root cause with knowledge that the rest will fix itself if I followed the defined SOP. The prescription was not too much, the lingering pain and inaction that restricted me to a bed with limited ambulation was. Daily checkup visits add to the agony as I am now in a state of mind, when will this ordeal end and I declared fit and healthy again to get back to work.

When you have a lot of time to do nothing, run out of music to listen or have no inclination to read with half the senses dulled due to heavy dose of pain killers and antibiotics, you start thinking; Doctors also refer to some of the extreme thoughts as hallucinations. To me my chain of thought was lucid and it created many correlations in the swarm of random disconnected millions of thoughts. Analogous to associations created in a data warehouse by a skilled analyst, I picked associated groups which made a lot of sense.

Early pain = project not on track; you know something is not working, you think it will get better, it doesn’t. Falling sick = project misses milestone; you get the vendor consultant to help, he refers to subject matter experts who have been there done that. SME educates on root cause, defines road ahead and KPIs to keep the project healthy. Recovery = always slow and painful; getting back on track takes a lot of effort, following the prescription, no shortcuts. Most of the time you do get back on track with no further slips; you have lost time, money and momentum.

I realize how we can create correlations between totally disconnected facts and make them look like similar data sets or for that matter draw analogies that sound quite logical. The event graph appears to follow a perfectly aligned path drawn by the same artist. Retail has been doing this with disjointed sets of data and have hit upon success many times; we don’t know what happened to the ones that did not work though. But then maybe life does have predictability that it wants us to find and we are getting better at it.

But I am digressing now, rambling about febrile correlations. IT gets sick quite often; whether it is business as usual or new initiatives, they do face challenges and require fixes from specialists and experts depending on the nature of ailment. We have prophylactic technology to keep things going while the next piece of hardware finds itself being resilient or more reliable than a decade back and networks become self-healing and storage can survive failure of a disk or two; Software still requires human intervention.

Human life expectancy has in the same vein gone up as we find better medicines for micro classified diseases. Our way of treating different patient types has been evolving rapidly with Internet of things allowing embedded Nano sensors connected to Big Data repositories analyzing symptoms as they happen and trigger corrective actions almost instantly with novel drug delivery systems. Okay, maybe the entire chain is not yet feasible, but getting there. Affordable access to such innovation would definitely be a paradigm shift.

As IT gets better, projects get more manageable, technology commoditization makes itself ubiquitous, IT wouldn’t matter ! What would matter is how we apply it to real life and help solve problems that have eluded solutions thus far. As more solutions go open source or relinquish patents for global availability, there would be a new world order where healthy humans will score over sick IT. Some of us will be part of this evolution if it happens within this generation. I hope our contributions would have made some difference.

Can Big Data deliver Big Insights ?

The other day I met a CIO friend who wanted to discuss a tricky situation in which he had landed; he worked in an industry which was in the thick of being projected as one of the industries that will benefit from investments in Big Data. His CEO wanted him to build a data warehouse to rival some of their global competitors, at least one of which was prominently talked about as the poster boy of Big Data analytics. He was thus under pressure to invest while the rest of his IT budget was under pressure.

Having a keen understanding of technology, his company and the industry, he was a non-believer in the Big Data story; according to him the hype around some of the Big Data insights were not commensurate to the investments made in the overall project. And there was nothing new since the first story broke out of one of the companies having found a use case that conventional technologies would not have delivered. He had many data warehouses and Business Intelligence successes in the past for which he was well known too.

By definition Big Data was all about big data sets that earlier available technologies could not bind together within tolerated elapsed time and budgets. Volume, Variety and Velocity defined Big Data; (Business) Value was added later. The availability of high compute resources and ability to store large volumes of data had made solving some problems easier, faster and cheaper; that is not necessarily success from the capitalized Big Data. It is just that larger data sets were analyzed as compared to the past.

The question at hand that needed an answer was whether he should let go and invest as directed by his CEO or he should help the business with a scalable data warehouse which would deliver immediate value. Is it possible to get started small with Big Data (an oxymoron if there was one) and then work with the business to find the needle (if they wanted to find the needle or a pin) in the haystack; after all Big Data is expected to throw up unknown possibilities by random correlations that human minds are not able to pick.

Big data works on “found” data, i.e. data that you have and complex algorithms which can provide some statistical probabilities. Analysts predict the value that different industries can gain from investments; no one is talking about the real value derived. Governments have been making investments with equal zeal as are large enterprises; the providers and consultants are happy to make hay not just while the sun shines but until by accident they discover a needle in the haystack and make a case study out of it putting pressure on the rest of the gold diggers.

What about the data that you don’t have ? Can you draw negative inferences from Big Data ? For that you have to know what you don’t have ! Can what you have tell you what you don’t ? The answer to that is still to be found; available data in a Big Data repository cannot indicate to what is missing. The concept of “found” data predicates that available data set is the whole universe from which correlations are to be created. And that is where many Big Data implementations are unable to deliver any meaningful insights.

The veracity (the 5^th V) of information in a Big Data store can throw up many false positives which have been the bane of many projects. Data will never be clean unlike conventional data warehouses and the velocity will keep you challenged to move with agility. The ability to come out of the clean and complete data mindset is the beginning of what Big Data may enable. From here to get to Value is a long journey with no near-term goals; if you hit something, consider yourself lucky and celebrate.

My suggestion to my friend was to get started the way he believed he will be able to deliver what the business wanted. Forget the discussion on technology and focus on what matters, insights driven by data. If he can get traction from some CXOs based on the results, no one will grudge whether they came from Big Data or Small Data. The business leader in him understood while the technologist wanted to fight; for his benefit, I hope the business guy prevails.