“They want 78 new reports from the EDW and Big Data which has taken more
than a year and a major part of my budget to build ! They already have hundreds
from the transactional systems which are printed on reams of paper which no one
reads. All the Excel sheets that they were churning out from data dumps from
various systems and a bit of external data get into management meetings where
everyone has a different number”. Looking at the CIO I sympathized with his
predicament, it was a familiar story.
We all have at
some time or the other been frustrated with endless requirements of reports and
data dumps from all and sundry; lot of effort is spent in analyzing the past
and validating hypothesis on what worked or what did not. Requests flow like
rainwater on a slope, never ending stream many similar to others from
neighbors at workplace not talking to each other. Reports get built for a
casual question in a meeting never to be used again; when another one pops up
from new quarters, the effort is repeated.
We hear of
associations, correlations and insights not possible in the past as we did not
know how to combine an apple and pineapple to get a watermelon. Structured data
was easy until we started going to multiple sources with limited commonality.
Even then with statistical models diving through seas of data, the proverbial
needle could be found in the haystack. People buying napkins buy beer, not vice
versa; owners of red cars have a higher propensity to be rash drivers, and so
on. You could correlate anything to sunspots !
Not too long ago
the need to explore unstructured data began and with social media explosion the
dimensions for analysis changed. Thus Big Data began its journey to challenge
conventional way of looking at data and information. Jumping on the bandwagon
the term was hyped by one and all to include variants that stretched
imagination. Came along new skills everyone thought were important for the
future: Data Scientists and Chief Digital Officer to name a couple; did such a
species exist or it was glamorized plain old profiles ?
Moving from hundreds
of GB of data to thousands of GB does not make it Big Data. The amount of data
being created and added to corporate storage is growing exponentially. Data
types are also expanding with technology offering ways to mine it. Dashboards
and cubes work well in selected situations, their action-ability is still
wishful thinking. Enterprise manager thinking has yet to evolve beyond reports
from transactional systems; thus the data scientist continues to remain a
glorified report writer.
The CIO narrated
his woes which started with the Company Board approving a really large budget
and unrealistic expectations from the project they called BBF (Bigger Better
Faster). With much fanfare the project was kicked off, many people inducted
into the team and a few pretentious youngsters hired to lead them to gaining
insights thus far unknown, from this prestigious first of a kind in the
industry Big Data project. The CIO kept his reservations to himself knowing his
meanderings would not be given a kind ear.
The project team
got bigger faster than anyone thought possible; the technology they bought was
deemed better than what they had. Everyone loved the progress they made in the
initial months. Then started the reality check with the target audience (managers)
putting across what they wanted to run their business better, to grow bigger
and reach out to customers faster than their competitors. Challenges with
technology and data consistency appeared small compared to the change required
in the mind set within.
Activity Reports
on social media, portal registration, access reports, keyword searches and some
more were the peak of expectation. There was no marriage between the old and
the new as if they lived in separate worlds. What could have been remained
buried somewhere while everyone wanted better and faster transactional or
tactical reports. The rich stream of data that could have been big for the business
was diverted and converted into wasted effort. In the corporate world, I
believe that the overwhelming data deluge is far from being tamed.
Do you know
different ?
