Does BI require Business to be Intelligent ?

“This is awesome, I love it ! Let’s make this live next week; and don’t tell me that it cannot be done in that timeframe ! After that we will work upon models for the other parts of the business and zap them with data that they have never seen before”. So said the CXO who had just been given a demonstration of the Proof of Concept that the vendor and the IT team had developed to highlight the capabilities of the tool. The Business Intelligence lead tried to get in a few words sideways and was brushed aside brusquely with a wave of the hand.

The company had history of failed projects and challenged BI implementations with business users running berserk. A new CIO had brought some semblance of order to the chaos with an IT governance framework. With some fresh talent as the journey progressed, the information needs started getting addressed which in turn raised the expectations bar. Earlier attempts at BI had faced multiple challenges which resulted in their being shelved. The new resources the CIO brought in had a pedigree of success and the result was visible.

In the initial stages of education the primary requirements revolved around reporting; can fragmented systems deliver consolidated reports from the data warehouse ? Any amount of persuasion would not deter them from their resolve for reports; so the first set of milestones did reports that did not require large investments in tools or great amount of intelligence. Some of the newer business users who had experienced a different reality in the world outside craved for dashboards and analytics. They were heard but not prioritized by the powers that be.

Seeing an opportunity the BI Subject Matter Expert worked with the business members to define the landscape and put in the extra effort post work hours to build the cubes and models. He sought new tools which were granted and with renewed vigor he created working prototypes that excited him and the business users who helped him. Tentatively they approached the self-professed technophile CXO who they felt would be excited with the new way of looking at information. They were however not prepared for the reception they received !

The CXO in his inimitable way was seen as the person who exuded overdoses of confidence and threw technology jargon liberally in every meeting. Know it all, been there done that was what he wanted everyone to believe; for the knowledgeable it was massacre with unrelated keywords interspersed with unverifiable anecdotes and visions of peaks achieved in the past. He was closed to ideas from others, derogatorily critiquing the world at large for not believing in him. The team feared his acidic behavior and kept away to whatever extent they could.

Recovering from the ridiculous, the BI SME stood firm that Proof of Concepts are exactly that and they cannot be deployed or scaled. He explained that for analytics to work effectively, he needed to build the model, test all data sources for efficacy, check for exception conditions and stress test for load. Almost all – barring the sycophants and the ignorant – agreed with him silently, putting the CXO in a spot. Taken aback not used to being challenged, he directed one of the minnows to ensure that his wish and command be done and huffed away.

Put in a spot the team member frantically looked around for a solution to his predicament. Gathering information from multiple sources to create a strong argument, she was able to enroll the business champions into her cause; they too wanted the new BI solution. It took some time and effort by the business champions to educate the CXO who reluctantly agreed to let go of his tantrum. Back on track the team worked together to build the solution which promised to open up new possibilities for the business.

The generally accepted new belief is that business knows what they want; the level of awareness and understanding has improved. Decisions are now information and data driven; BI solutions deliver this and more. Change management is a critical success factor towards adoption and the journey on the evolution curve. The data deluge and associated tools to manage them has created new challenges and opportunities; the ability of enterprises to leverage these will be determined by their maturity towards analytics.

The BI solution was a great success; everyone loved it. The CXO was last seen ranting about why the team did not use Big Data !

Can Big Data deliver Big Insights ?

The other day I met a CIO friend who wanted to discuss a tricky situation in which he had landed; he worked in an industry which was in the thick of being projected as one of the industries that will benefit from investments in Big Data. His CEO wanted him to build a data warehouse to rival some of their global competitors, at least one of which was prominently talked about as the poster boy of Big Data analytics. He was thus under pressure to invest while the rest of his IT budget was under pressure.

Having a keen understanding of technology, his company and the industry, he was a non-believer in the Big Data story; according to him the hype around some of the Big Data insights were not commensurate to the investments made in the overall project. And there was nothing new since the first story broke out of one of the companies having found a use case that conventional technologies would not have delivered. He had many data warehouses and Business Intelligence successes in the past for which he was well known too.

By definition Big Data was all about big data sets that earlier available technologies could not bind together within tolerated elapsed time and budgets. Volume, Variety and Velocity defined Big Data; (Business) Value was added later. The availability of high compute resources and ability to store large volumes of data had made solving some problems easier, faster and cheaper; that is not necessarily success from the capitalized Big Data. It is just that larger data sets were analyzed as compared to the past.

The question at hand that needed an answer was whether he should let go and invest as directed by his CEO or he should help the business with a scalable data warehouse which would deliver immediate value. Is it possible to get started small with Big Data (an oxymoron if there was one) and then work with the business to find the needle (if they wanted to find the needle or a pin) in the haystack; after all Big Data is expected to throw up unknown possibilities by random correlations that human minds are not able to pick.

Big data works on “found” data, i.e. data that you have and complex algorithms which can provide some statistical probabilities. Analysts predict the value that different industries can gain from investments; no one is talking about the real value derived. Governments have been making investments with equal zeal as are large enterprises; the providers and consultants are happy to make hay not just while the sun shines but until by accident they discover a needle in the haystack and make a case study out of it putting pressure on the rest of the gold diggers.

What about the data that you don’t have ? Can you draw negative inferences from Big Data ? For that you have to know what you don’t have ! Can what you have tell you what you don’t ? The answer to that is still to be found; available data in a Big Data repository cannot indicate to what is missing. The concept of “found” data predicates that available data set is the whole universe from which correlations are to be created. And that is where many Big Data implementations are unable to deliver any meaningful insights.

The veracity (the 5^th V) of information in a Big Data store can throw up many false positives which have been the bane of many projects. Data will never be clean unlike conventional data warehouses and the velocity will keep you challenged to move with agility. The ability to come out of the clean and complete data mindset is the beginning of what Big Data may enable. From here to get to Value is a long journey with no near-term goals; if you hit something, consider yourself lucky and celebrate.

My suggestion to my friend was to get started the way he believed he will be able to deliver what the business wanted. Forget the discussion on technology and focus on what matters, insights driven by data. If he can get traction from some CXOs based on the results, no one will grudge whether they came from Big Data or Small Data. The business leader in him understood while the technologist wanted to fight; for his benefit, I hope the business guy prevails.

Ignorance to Intelligence

A sparse gathering of smart people in a small room were discussing an important business hoping to change the outcome in their respective circles of influence. These were seasoned players in industries big and small across diverse geographies and line of business. It was crowdsourcing at its best and that gave rise to the expectation that the problem will indeed find itself vanquished. No I was not the fly on the wall or a passive listener; I was called in to moderate the discussion.

When I joined the group, they had already listed down the facts and figures about the rate of success that they had captured from published research as well as search engines. Anecdotal references were discarded focusing on empirical data and reference case studies. They had also listed down challenges and issues based on their real life experiences. The consistent and clearly emerging message was how to get Business back into BI, while the “I” actually stands for Intelligence and not Ignorance.

One of the discussion threads went something like this: the CEO needed quick reports and analysis of business with the ability to drill down; he also expected the ability to slice-dice the data. The business head wanted control over her team and did not want the CEO poking his big nose. IT simply wanted to buy the best tools since they did not have control over the environmental and political factors. The CEO in a social meeting was sold on the merits of a frightfully expensive platform and that was that.

The tool was bought, the wish-list generated across different stakeholders top down, the scope defined and captured by the principal vendor, and the execution outsourced. The requirements were skewed in favour of reports and very few dashboards or analytical cubes and that was deemed okay; everyone agreed that the phase two will cover the rest. So with a lot of fanfare the project got started and everyone believed that the future holds a lot of promise.

Through the discussion no one challenged the ask, no one sought to understand the impact of the multitude of reports on the stakeholders, no one challenged as to why the operational staff was not in the room to define what they wanted. The executives in the room blissfully proposed the metrics and data that others should be seeing, reality being far removed from the proposals. The few dashboards that were to be consumed by the CXOs changed shape based on assumptions of what they would like.

Running through the discussion I realized that this specific case had all the ingredients across the various representatives, so the solution could be broad based to apply across. Good news is that it was not a technology project considering initial active participation from the business, however they had disappeared post initial discussions leaving the baby with the vendor and the IT team. With the sketchy picture the solution was unwieldy and unusable across the layers of management and operational staff.

My submission to the team was to go back to basics and start asking some tough questions and not to proceed if the answers were not up to the mark. Their lack of enthusiasm depicted their unwillingness and inability; with some persuasion they agreed to plunge ahead. As we discussed the questions and the approach their eyes gained the missing spark; the conclusions were agreeable to all present as the best way forward. Here’s a synopsis; report is used as a representation, it could be a dashboard or cube too.

1. Why do you want this report ?
2. How do you get this information today ?
3. Who else could benefit from it ?
4. What will change for you after you get it ?
5. Which personal, group or company KPI (e.g. customer, employee, revenue, profitability) does it relate to ? How ?

As we closed the meeting, I realized that earlier some of these questions had got me into trouble from which I could extricate myself from with some effort. The project was however declared a great success and a case study by the company, vendor and the industry. I believe that it is a better place to be; ignorance to intelligence is a difficult journey. Educate the business; don’t get bullied into accepting inane requests for reports that can be fulfilled by transactional systems. Someone has to drive it, why not you ?

BI OIC for CIO

Business Intelligence has been on the CIO radar for over 3 years now as per research reports from the major and respected IT research companies. Billions have been spent by companies and as a category, BI projects have seen the least success across almost all IT enabled initiatives. BI consultants will go about advising you that they "know" how to make it work for your enterprise and if and when things start faltering, it's always you and your users who are responsible. I happened to ask a VP turned consultant from a large bank, "How is it that consultants have all the answers, but employees do not ?" and got no answer.

Every organization aspires to create intelligent insights from transactional data and act upon them to increase revenue, optimize profits, retain customers or create efficiencies. In most cases, users are unable to think or visualize what they want to achieve any of the above objectives and thus end up defining extremely complex scenarios and reports with a hope that they will provide some kind of "Eureka" brainwave and they will be heroes. IT organizations takes limited steps to dispell such myths and takes on the task of creating models that enable the complex reports. Maybe because they are not in the bath tub !

Typically such operational data based reports provide limited insight since the insight is driven by humans and not systems. A report viewed by different people creates different inferences and that a technology cannot provide. Technology works on models created by us and is thus limited by the information gathered and understood by the developer. This indeed is the key to actionable insight and the experience and frame of reference of the person which makes the difference.

This is not a great insight, but it is based on the experience of seeing many business leaders responding with differing insights to the same analytical report presented. It comes out of a deep domain expertise and lateral thinking. Such individuals if enrolled into the program can make the difference between a successful and not so successful BI project.

Left to IT organizations, which is the typical case with BI projects, the end result is multi-dimensional reporting which is finally used to review business but with limited insights. It is evident and obvious that right people make the difference, but the right people are normally too busy to spend long periods of time to understand the possibilities and list them down. If by chance that were to happen, the adoption by the rest of the organization suffers and gets blamed on bad "change management".

So what is the conclusion ? A few "actionable insights" based on 3 BI projects.

1. Find the "right" person within the company who can help create insights. It may be your CEO ! If you cannot find such a person, don't start the project

2. Start small and scale up as you taste success; before you attempt to build the Taj Mahal, practice some smaller buildings

3. Tools and technology matter, but in the end, the data quality makes the difference

4. Do not be averse to restarting from scratch in case the first model or the second model does not deliver. It's quicker to recreate than attempting to patch a bad one

5. Keep on asking questions at every stage, "Why do you want this ?", "How will it help you or the company ?", "Who else can benefit from this ?", you get the picture .....

6. Whatever you do in stage 1 may need to be discarded by the time you are in stage 3 and that's okay.