“Ambiguity" is my middle name
There is a ginormous amount of onus on every product team to eliminate all possible ambiguity so that the other teams can have something concrete to work with. But, is that really easy? Hell! NO!!
1. INTRO
Product management is anything but exact / precise. It’s no joke that many find the job very vague as they make a transition into this position afresh often touted as an exciting one by the people who have accomplished it.
There is ginormous amount of onus on every member of every product team out there to get down to eliminating all possible ambiguity and clear out the clutter so that the other teams like Design, Development, UX can have something concrete to work with.
So, when dealing with unfavourable and ambiguous situations is just a regular day at the office, AMBIGUITY could as well be a Product Manager’s middle name.
Well! Does it sound daunting already?
When the daunting part is true, it is also true that there are many ways to combat it and you’re in luck as PM isn’t something that was invented out of a whim just yesterday, where you’d face a terrible lack of resources to cope with your regular jobs. So, just like any other job in any field with ample complications there is a so called “a method to the madness”.
The building block (or in this case the discernible building block) of every PM is the data and the insight that one gains out of studying that data which is employed for the crucial decision making and in lending direction to teams.
2. A Social Backdrop
It amazes me how data can be so crucial across positions, portfolios & domains. At a recent social gathering, there were a bunch of people from various domains who had just got introduced were talking about their professional lives. Some excerpts here.
As is evident from the conversations in the image above, many professionals hinge on data for various reasons, the most crucial and common one being - help them make those critical decisions & do them right over the very first trial itself.
Sometimes few teams operating in some select domains would have to get to variable lengths and iterations to obtain this “data”. I some cases it also involves paying their way into acquiring the data (some orgs that operate in market intelligence / customer data management platforms pay per record) because that’s the safest, easiest & most accurate route to facilitate and also speed up decision making at their end.
But not all professions have easy access to / can pay their way into acquiring the data. For some, it may be days and days of hard work of running experiments ruthlessly without even getting close to a state of semblance with regards to their data acquisition.
And obviously when you say experiment there a whole lot of factors, considerations that come into play there.
3. Experimentation
Anyone for OREO – Watermelon?
Get on the internet and you’d find this listed as a prominent product that failed and failed badly.
But facts also point out that the cookies were available only in select TARGET stores in the USA during JUNE 2013 and it wasn’t all that widely accessible either. Also, another noteworthy point is that there weren’t all that many adverts either. And later the product was removed permanently off the shelves.
What’s the chief use of spending all that money on building the product? Why didn’t the company think of a survey?
Well! May be, they could have. But, some experiments over physical products are best run over prototypes that are very basic versions of the actual product. The essential thing when it comes to appealing to the audience in FMCG / Food industry is “taste” and there was no other way to measure it than launch it over a small section of the audience and look for feedback over it.
So, when the stem of an experiment was to prove / disprove a theory, the chief value that it adds is, it lets you practically “learn” a host of things and perhaps gets you to:
understand the importance of altering a few parameters
playing with a few proportions
adjusting a few values to gauge what changes and how
leading to more and more practical fact finding about the product.
Is there a methodical way to conduct these experiments?
3.1 The “How” of Experimentation
Observation, Questions – Keen observation of how the users are going about their tasks is where it all starts from usually. And, those observations naturally have to lead to a host of questions – (Laddering)
Research – Once you have arrived at a ton of questions, the next step is to drill down to gain more qualitative & quantitative understanding over the problems, impact, frequency, importance, solutions, market, users et. al.
Hypotheses – Post all the assessment over the previous steps, you should have a clear definition of a hypothesis which is nothing more than a theoretical version of your estimation of the problem space. Validation would still be pending though.
Definition – Based on the generated hypothesis, you’d now define an experiment which may involve suitable parameters and variables that are important to track whilst also defining other things like duration, targets, sample space, factors that matter to classifying success / failure.
Run the Experiment – The said experiment will be allowed to run seamlessly tracking each and every move the user makes (easier if the delivery mode is WebApp / MobileApp).
Results / Data Collection – The data from the experiment will be collected as perceived meaningful – across product / feature / cohort depending on parameters defined earlier.
Analysis (to prove / disprove the hypothesis) – The visualisation of the data collected and the analysis of it in an aim to prove / disprove the hypotheses is a crucial part of the process. Marketing teams may largely be interested in factors that support / are in favour of their hypothesis when Product teams have to be on the lookout for all caveats that lead to disproving their hypothesis.
Discussion / Brainstorm – Post analysing all the findings, teams normally get into a session of discussion to brainstorm all possibilities and select / prioritise one of the routes to take which may lead towards the build phase of the product post attaining the buy-in from leadership.
By now the confusion should have given away completely and one certain path should have emerged as the clear winner over the others.
So, in following this method teams start to believe largely in the “fail and fail fast” theory than get mentally & emotionally stuck with an idea for a long period eventually draining themselves out.
PRO TIP: There is no “FAILURE” here in experimentation.
When it is quite possible that an experiment may have a negative outcome as in something that was visualised as a closest match and a fitting solution may not be able to derive enough traction from the market. And, it is important to understand that you still have made progress in eliminating a disaster by potentially not going ahead to the next steps in building that sans validation.
But a few questions ought to cross your mind at this stage:
what experiment do I have to conduct?
when and how do I decide to pull the plug on it?
how do I decide that I have gathered enough info in taking that confidence and progressing over to the next stage of the Design / Build?
3.2 The “What” of Experimentation
There are plenty of resources available on “how to conduct experiments” right from A/B Testing, Split Tests, Trapdoors / 404s, Multivariate Testing, Elevated Trapdoors, MVPs, Prototypes.
I’d stick to the “What to experiment” part here & it is the tricky bit.
3.2.1 Important Factors of Experimentation
For ex: Consider a modern home today where people are damn busy and have all the access to amenities, but the one thing they seem to be constantly short of and feel the chief necessity to keep track of is “The TIME”.
Supposing there is a watch making company that already has a few products launched and a few 100 thousand paying customers. As a part of their growth strategy, they now want to expand their product offering by foraying into what may be new customer segments in identifying and solving problems with their new range of product(s) bundling it with a bit of innovation.
Central IDEA
A unique talking clock that is voice controlled & could announce the time aloud when queried
User Story
As an individual living in my home, I’d like to have a device that could tell me the TIME aloud so that I can keep pace with my chores / tasks.
Probable Feature Set
Digital clock – regular digital watch face with the usual big digit display
Voice Commands – ability to understand a voice command “what’s the time” / “what is the time” / “time please” / any other variants meaning the same
Receptive to Voice / Talking – a small speaker with the ability to answer / reply back with the current time
Volume Control – able to control the volume via voice commands
Discovery & Research
the problem the product is trying to address
is there a need for such a clock in the market?
what does my supposed market think of the idea?
how excited are people about this idea / otherwise?
is there any such product available in the market?
is this a rate sensitive market?
Questions to ask & learn from / Experimentation
How many timepieces / clocks do you have at your house?
capture the spending curve / inclination to spend
get an understanding of the no. of rooms the person has in his house
Did you ever face any problem knowing the time when you are away from the room where the clock is mounted / placed?
are there areas in the house that don’t have a clock?
difficulty in gauging the time without access to the clock?
underpin the problem statement & the need for the solution
no. of instances where you have not been able to reach a place in time?
Would you think it would help your predicament if the time was spelled out loudly at intervals over a speaker so as to be audible enough inside every corner of your house?
any similar product(s) available in the market
awareness of the talking clock
inclination towards & necessity of the talking clock
the necessity to know and keep track of the time at continuous intervals
Have you ever felt it necessary to get an answer back when you exasperatedly, hurriedly asked “what is the time?”
inclination towards knowing the time while being away from the clock
need to know the time when in a hurry
accessibility constraints (if any), over knowing the time
justification of the central problem statement
External / Environmental Factors
disturbance caused by the recurring alarm of a talking clock
privacy of others in the house
could be a nuisance if queried frequently
Use Cases
Households
Selective sections of Factories
Manufacturing Units / Hangers
Schools
Colleges
Exam / Test centres
Public Transport (Bus, Train, Tram)
3.3 How much experimentation is enough?
When there is no straight answer for this, there are multiple ways to approach it though.
When most of them would like to keep the whole experimentation time-bound – as in, run it for a week / week tops and then collect data, it makes sense to continue the experiment until a sizeable share of the responses haven’t landed.
How would you determine the length / duration of your experiment given a product / variation of a product / a sample size?
Looks unrealistic, doesn’t it?
Even for the best of products and brands considering the launch of an additional feature, that (100 days) number could be too huge to cope with.
Going by that hypothetically, a company like Apple would take about 1,800 days to test out 2 variants of the iPhone @ 1mn visitors per day.
Does that really look possible / is that how it is done?
Naah!
So, when pure numbers / statistics like:
No. of visitors
Duration - No. of weeks
Confidence intervals
Confidence levels / Statistical significance
Share of the sample space
Baseline conversion rates
makes sense to keep a tab / track of, it may be necessary to use an extrapolation / a mathematical model to normalise that sample space to begin with.
Also, it is crucial that teams gain the buy-in to run experiment for that much longer until satisfactory results are obtained from the market, though that could be considered a very idealistic scenario in some cases.
4. Case Study – SKITTLES (Experimentation is Crucial)
Just hit the link and check out this 2-minute video on Skittles – a company that makes multi-coloured candy in the USA.
It talks about how the management of the company apologised individually over a video to the customers over its mistake of turning to GREEN APPLE over LIME and eventually had to bring back LIME flavour post a severe social media outcry / survey.
YaaY! Whoopee!! Lime is back!
Well! It’s crazy how much of brand value can be lost over something like this in today’s world. But it somehow took this company about 9 years to realise where they went wrong & reinstated the LIME flavour finally.
NOTE:
If something similar was to happen to an organisation building SaaS products, they could’ve well been staring at a 10-15% CHURN over a monthly cohort and whatever you do post an experience in the negative, churned users may not end up coming back to using your product like before, because of the competition and flexibility over the choices in pricing available to them in the market.
So in this case, before going ahead with the decision to change the “GREEN” coloured candy from “LIME” flavour to “GREEN APPLE”, I’d draft a questionnaire and push it to my existing customer base in an aim to understand their behaviours.
The question could very simply be embedded over a poll / survey / an e-mail and could be something on these lines:
PRO TIP: For any SaaS based product the same experiment might work to capture the choices of the market before the first launch. The only caveat may be in moulding it to work for the pre-launch situation when the audience may be a whole list of prospects and not customers who are already paying.
Conclusion
Experimentation could be crucial to a product-based organisation’s existence given today’s age of SaaS. If it is conducted & targeted well whilst maintaining relevance supported by continuous learning from the market, it could well be worth its weight in GOLD!
If you enjoyed this, then please:
Do like the article
Do share it to your peeps
Follow me on Twitter if you are interested in daily short updates on product management, strategy, culture - @BgpInv
Thanks for reading 🙏