Journey from Data to Insight – PART I
Although deriving insight from data seems elementary, operating as a PM one’s faced up with the hard truth – “IT JUST ISN'T”! Here's how you gain the right amount of insight & ensure it is actionable.
PART 1:
Tera-data, Peta-data & counting…!
Coming from an era that’s over 2 decades into technology and seeing this space evolve so closely, having lived through all those joyous & how some were fraught with self-doubt, anxiety & fear as well, I’m so often & fondly reminded of this quote by Eric Schmidt.
As an afterthought, by the time you get to read this may be during AUG 2023, the numbers might have swelled to “exabytes of data processed in an hours’ time”.
Note: 1 exabyte = 1 billion GB
Indeed, it is true.
Take a good look at this graph for comparison of the amount of data being generated by geography in the last decade.
But the evolution in the past 5 years in this space has been exponentially exponential if one is able to space that & make sense out of that term, mind-boggling to say in the least. And seriously, we’ve gone from a pretty inexistent space from 2 decades back to evolving it stage-wise over data warehouse, data lake, data swamp, metadata.
Think about it, what would all those data gathering exercises mean if one is not able to derive any insight out of it? The futility of it all may stare one right in the face if they ever got to a realization over the amount of time spent & effort it took over the volume of data collected when it seems to point to no real insight at all.
To make data actionable means conquering the barrier of time so as to arrive just at the right amount of insight required to influence all the right decisions.
From Data to Insight…
Going from data to insight happens to be the ultimatum but the process is usually fraught with many steps, only that the number of steps and the intensity seems to have risen exponentially in the last decade.
Here’s a decade long comparison depicting the workflow of how teams got to arrive at that very insight starting off from a certain data set.
Let’s go over each of these steps as relevant to 2023 and try to understand them better over a product like say, “herbal energy drink” (extending the same example from our previous market positioning exercise to help build a complete picture and lead to some closure of thoughts):
NOTE:
Data collection activities have risen manifold, in fact exponentially over the last decade
Need for algorithms to aid quicker decision making is pretty evident today
Going from data to insight seems to have transitioned from manual to totally automated
Consideration of Metadata is imperative given today’s timeline
Advent of AI taking up all the load of hard-crunching / processing over ML algorithms / DL is seen as a need of the hour
But, given that very scenario, could the process of arriving at the insight over any sense of imagination be declared as “foolproof”?
Absolutely not.
There are tons of imperfections in the data itself that contribute to wrong decisions which is why one looks to subject it to a process of skimming (colloquially termed data cleaning) and normalization. When running a data sufficiency test and refinement (STEP 6 & 7) could help identify those issues and help resolve them largely, eliminating biases induced by various sources could be a gut-wrenching exercise owing to how the approach itself could largely be reactive in nature as one’d be at the mercy of lagging indicators entirely.
And of course, today we’re talking of AI making it to the far end of that spectrum as it is employed to process & make quick decisions given the exodus of data we happen to be dealing with.
Decision Making - Antipattern
Let me draw your focus to how decisions are made at some large corporate offices and why this happens to be a major antipattern.
Here’s a link to that very tweet about how some orgs. & teams believe in going from data to what they would like to term insight.
https://twitter.com/BgpInv/status/1688411480969150464?s=20
And, there is no doubt how that could be a huge problem and blow up in one’s face later on, hinging on research yearning to find ONLY that bit of data so as to substantiate a given hypothesis.
Does that even qualify as a validation in the first place (let alone discovery)?
Well! May be it could qualify as validation in a literal / trivial sense of the word. But it would most assuredly fail and terribly fall short of providing a complete picture of a given market scenario and could never come close to being termed valid reasoning.
But sadly, that method was being followed, is and will continue to, even as we speak today given how some teams believe in “optimizing for optics” (quoting Shreyas Doshi here).
Conclusion
The journey going from data to insight seems to have changed and changed for the better today as opposed to where it figured about a decade ago. Given the current day’s timeline there’s often a methodical organic process over which data is gathered over research, insights are derived, hypotheses tend to be proven or disproved so as to get to a state of semblance, an actionable insight, orchestrating into an action plan as a culmination.
To summarize:
understand that deriving insight happens to be a 12-step elaborate process as shown in the figure above and if you or your teams aren’t adhering to it in its entirety or skipping any of these steps in the process, not only are you missing out big time, it is certainly going to lead to a lot of schedule overruns if you are lucky enough & it is identified in time or it’d prove to be a total recipe for disaster in the worst case