Evidence-Based Innovation Portfolio Management
Incorporating customer behavior into portfolio decision-making
If you’ve ever been inside your company’s boardroom to observe how executives make innovation investment allocation or project prioritization decisions, you have likely seen them use one or more of the following approaches (in order of sophistication):
- Highest paid person’s opinion
- Checklists
- Scoring models
- Financial projections (e.g., NPV, IRR, revenue, profit)
- Bubble charts (e.g., risk/reward)
- Expected commercial value
- Monte Carlo, decision tree, or options modeling
Most of these approaches work just fine when you want to extend the life of existing product lines or develop products in your core business with familiar technologies going after known markets. But as you move away from the core to evaluate projects meant to drive growth in new categories, these traditional approaches become less and less reliable. Projecting an ROI in the early stages of a new-growth innovation, especially when you are going into brand new territory with no history to draw from, is guesswork at best. This is where customer evidence needs to take center stage as an input to portfolio decision-making.
In this article you will learn how leading companies incorporate customer evidence into project portfolio decision-making, replacing uncertain financial projections and opinions with facts.
Experimentation & Evidence
Early in the innovation process, when uncertainty is at its highest, you run experiments with target customers to test the most critical aspects of a transformative new idea. You start with the riskiest, highest impact business model assumptions to learn as quickly and cheaply as possible. These experiments start before you have an offering and continue as you validate if the evolving solution addresses a significant customer need, the customer problem you are solving is widespread, and the business model is viable.
The goal of these experiments is to produce evidence in the form of a measurable customer action or behavior that proves you are progressing toward a profitable, scalable business — or, in other words, leading indicators that you are on the right track. Proof in the form of data plus insights can inform you if customers desire your solution, are engaged when they use it, and remain engaged over time while demonstrating that the new offering can scale profitably. When using evidence of value from customer experiments, portfolio decision makers focus on the viability of the most critical assumptions that drive the numbers in the project’s business case. Customer evidence brings an increased confidence level to the ROI guestimate.
The cost and fidelity of the experiments you run and the evidence you seek will vary depending where you are in the development process. In the early stages you run a series of low-cost customer experiments to validate the significance of the customer problem, the feasibility of the initial solution, and the viability of the proposed business model. As you reduce uncertainty and progress toward a validated solution, experiments increase in fidelity until you get a representative product in the customer’s hands. At this point you look for evidence of purchase intent, recurring revenue, or virality.
When presented with customer evidence that demonstrates progress toward an ROI, portfolio decision makers can make fact-based investment allocation decisions, ratcheting up funding for those projects that are showing promise, quickly cancelling those that are not, and rebalancing priorities. They are no longer seduced by the false precision in hockey stick projections or assumption-laden spreadsheet projections alone. Scott Cook from Intuit summed this up well by saying, “For every one of our failures, we had spreadsheets that looked awesome.”
Uncertainty is Everywhere
Everything discussed so far will be familiar to anyone who is using an Agile or Lean Startup approach to new business creation. However, customer evidence can be used to inform portfolio decision makers across the entire new product development continuum, from product line extensions in the core business to adjacent new products and services to new-growth innovation. Uncertainty exists across all growth horizons, as shown in the following graphic (created by Brant Cooper, author of The Lean Entrepreneur).
Even low complexity product enhancements contain some level of market uncertainty or development risk that needs to be managed. Whether squeezing cost out of an existing product, getting regulatory approvals to ship into a new region, or adding a new feature to extend the life of an aging product line, uncertainty abounds. Anyone who has led a “simple” supplier change understands this. The main difference is that on the far left-hand side of this graphic you have sales history, market familiarity, less technical complexity, and a shorter development timeline, making your financial projections and development pathway relatively straightforward and reliable.
For many years, execution-oriented companies have conducted proof of concept experiments and have sought customer feedback early in core new product development. The difference now is that we are looking to turn internal lab tests and anecdotal voice-of-the-customer feedback into behavioral data. Not just capturing what customers say in a focus group or survey but measuring what they do.
By embedding iterative customer experiment cycles between phase gates, core business project teams build customer evidence to back up the assumptions in their business case, bringing increased confidence to portfolio and phase gate decision makers. This is especially important for physical products with spend profiles that accelerate after the planning gate and where hard to change commitments must be made (e.g., hard tooling, production equipment, or system architecture decisions).
Evidence Examples
The following table lists examples of behavioral evidence metrics that apply across the full uncertainty continuum (grouped by stage of innovation). The evidence in these examples provides measurable data with insight into why customers behaved the way they did.
Note how:
· The type of evidence changes depending on where the project is in the discovery and development cycle (i.e., problem validation, solution validation, market introduction & scale-up)
· The cost and fidelity of the experiments conducted to produce the evidence start out low and increase as the solution is validated and risk is reduced
· Iterative customer experimentation and evidence can be integrated into the core business phase gate process wherever there is uncertainty
· There are examples that work for both software (SW) and physical hardware (HW) products, B2C and B2B
Results
Let’s take a deeper dive into how some companies have successfully incorporated customer evidence metrics into their portfolio decision-making process. In the following three examples, each company applies evidence in a different way at different points along the uncertainty continuum.
Example One — Medical Device Company:
A medical device company with a mix of physical hardware and software products in an industry with heavy regulatory scrutiny spent six months designing and rolling out a more disciplined portfolio management capability that included customer evidence metrics as part of its project evaluation criteria. After two quarterly portfolio review cycles, the company has rebalanced its portfolio, killing projects without sufficient evidence of value and shifting resources toward validated opportunities in high growth markets. So far, their risk-adjusted portfolio value has increased by 30 percent.
Example Two — Financial Services Company:
A large financial services company with software products in an industry being challenged by small, nimble Fintech competition and a pace of change that has accelerated, still relies on its core platforms to stay competitive. Their customers have low switching costs, so it is essential that they introduce a steady stream of valuable platform enhancements to drive higher usage and bring new revenue streams. Product teams are learning to run low cost customer experiments and incorporate evidence metrics into daily scrum meetings to help them prioritize their software development backlog and make more informed decisions on release sequencing.
Bill Hazelton, the company’s head of enterprise innovation and agility, explains the new approach this way, “Over the last 3 years, we successfully implemented Agile development throughout the organization and have made great strides improving project throughput and time to market performance. Our focus has now shifted to project selection and making sure we fund the highest value projects in our portfolios. Rather than rely exclusively on traditional methods to prioritize customer experience improvements and feature set upgrades as we have in the past, we are now using customer evidence to inform these decisions.”
Example Three — Pharmaceuticals Company:
In the diabetes care division of a large pharmaceuticals company with a mix of hardware, software, and services for disease management, the company’s commercialization teams conducted customer experiments to test sales channel assumptions for a portfolio of recently launched solutions. Insights from these customer experiments were used to optimize marketing spend allocation and awareness generation tactics to scale more effectively.
These three companies are achieving impactful results, but more importantly, they experience more effective portfolio review meetings and replace internal opinions with decisions based on external facts. Long, drawn out debates over project priorities are short-circuited. Discussion now centers around critical assumptions and the evidence the team must uncover to warrant further investment. Instead of asking “When will we see an ROI?,” leaders ask “What did you learn from the customer about the viability of the solution?”
Implementation Tips
1) Start with the basics and keep it simple
A crawl, walk, run approach is advised when building evidence-based portfolio management capability. Depending on what level of maturity you start from, it is important to ensure leadership alignment on project evaluation criteria, but also on portfolio scope, governance structure, project category definitions, resource pools, review cadence, and investment allocation categories (strategic buckets). Once these operational elements are defined, rolled out, and working effectively, you can start to layer in advanced capabilities without overwhelming the organization. Keeping it simple as you ramp up the discipline is the best way to ensure organization-wide adoption. Complexity is the enemy.
2) Introduce evidence metrics as an enhancement to existing criteria
If your organization is new to the iterative customer experimentation and evidence approach, you can start by introducing evidence-based metrics as a supplement to existing portfolio decision criteria already being used. Companies with mature phase gate processes already ask project teams to call out high-risk business plan assumptions and plans to de-risk those assumptions at early gate reviews. What is new here is the increased focus on driver assumptions and the necessity to validate those assumptions with customers.
When evaluating new projects and platforms that are further away from the core, customer evidence increases in importance relative to traditional financial metrics. For early stage, new-growth innovation projects at the far end of the uncertainty continuum, evidence-based learning metrics and progress towards an ROI take over as the primary portfolio prioritization criteria (as shown in the uncertainty continuum graphic above).
3) Create guidelines to level set expectations
The goal in the early stages of implementation is to get portfolio leaders and project teams comfortable using customer evidence to inform funding trade-off decisions. Create guidelines to level set expectations for evidence thresholds at key milestones as projects move from problem confirmation to solution validation, market introduction, and scale up. A simple scoring model can be used to capture experiment progress and evidence strength across solution desirability, viability, and feasibility dimensions and is an effective input to the portfolio funding discussion.
4) Layer in more advanced approaches over time
Over time, consider implementing more advanced approaches that attempt to put a meaningful number on uncertainty. Thought leaders like David Binetti have come up with options modeling frameworks that put a risk-return dollar value on highly uncertain innovation investments. These approaches help get project teams and finance on the same page and speaking the same language when evaluating new-growth innovation opportunities.
So next time you are in the company boardroom observing an innovation portfolio review, pay close attention to how innovation investment decisions are made. Hopefully, the decision team is looking beyond traditional financial metrics or the opinions coming from the loudest voices in the room. Customer evidence trumps opinion every time.
