Idea evaluation – What is it and how should one do it?

Evaluating new ideas is one of the most challenging parts of the job for any manager or executive.

By definition, new ideas always have a lot of uncertainty involved and little data to back them up. Thus, prioritizing ideas and making decisions on which ones to implement right away, which ones to test or pilot, which ones to keep for later, and which ones to outright discard, will often put you in a bit of a pickle.

In addition, evaluating ideas is also an important factor in both successful idea management and innovation management. Given this background, it’s unsurprising that many of our customers have a lot of questions on the topic.

So, we thought it would be useful to share our experience on how to make idea evaluation work for you.

Evaluating innovation is a bit of a paradox 

To start off, let's first look at the big picture. Ideas related to existing business can, and should be, evaluated based on the rational metrics that a well-run business already has in place.

However, when it comes to "proper innovation" – not simply incremental improvement – it’s virtually impossible to evaluate ideas or initiatives by using metrics that make sense from the point of view of the organization, such as expected ROI or even market size. This is the very essence of the Innovator’s Dilemma.

Thus, evaluating innovation is very different from incremental improvement. The first thing you should do is to make a clear distinction between the two and to treat them separately. This is essentially what an ambidextrous organization is all about.

An ambidextrous organization is one that has successfully managed to organize themselves in a manner that makes it possible to manage these two fundamentally different practices, running and expanding an existing business, as well as building new ones, in a different way based on their unique characteristics.

By simply understanding and acknowledging this difference when evaluating ideas, you’ll have already gone a long way in avoiding many of the typical challenges associated with this dilemma. What’s clear is that there’s not a one-size-fits-all approach when it comes to evaluating ideas and trying to force one is simply going to lead to tons of challenges down the road.

There’s not a one-size-fits-all approach when it comes to evaluating ideas.

Before we delve deeper into these challenges, it’s important to also understand the characteristics of idea evaluation that are the same regardless of the situation and the process you choose.

Evaluation is a tool for making decisions, not a decision-making system 

In most organizations, it’s usually one person or a group of people that makes the go/no-go decision, and evaluation is simply a tool for providing them with the best possible information for making that decision.

When the idea is related to a big organization-wide initiative, this can mean the board, a single executive, or a steering group. In the case of smaller, more operational ideas, it is usually an individual manager or a functional expert in the respective team.

When it comes to evaluating ideas, common practice is to use a number of different criteria, referred to as metrics, for considering the different aspects and dimensions of the idea.

These metrics can be either qualitative or quantitative, but for the purposes of simplifying the analysis, qualitative metrics are usually represented with quantitative estimates, for example on a scale of 1-5. These metrics can then be combined to form an overall numeric rating for the idea, which we refer to as the score.

It’s important to remember that while this score provides you with a handy estimate for the overall potential of the idea, simply trying to maximize the score should not be an end in itself, nor should the score be the sole criteria for making the decision. There are a number of reasons for this: 

• The metrics you’ve chosen might be irrelevant or not tell the whole truth
• The evaluation ratings themselves are rarely entirely fact-based and usually just best guesses
• The best (disruptive) ideas will very often not make sense. When you think about them in the terms of your current business, they can even seem completely ludicrous.

The last part is especially important if you’re primarily looking for ideas to renew your organization or business. In these cases, you need to pay special attention the metrics you choose and rely more on the intuition of the people you trust to be most insightful and even amongst them, these ideas are still likely to be highly controversial.

The only way to know whether these ideas will work, is to simply test them out with real customers and/or users so make sure you’re prepared to do that form many of the raw ideas.

Having a systematic set of criteria for evaluating ideas, even with imperfect data, is going to lead to much better and more consistent decisions being made than simply relying on gut feeling.

Having said that, having a systematic set of criteria for evaluating ideas, even with imperfect data, is going to lead to much better and more consistent decisions being made than simply relying on gut feeling all the time. But more on that later.

Evaluation should be transparent

Regardless of the type of ideas being evaluated, you should always strive to make the evaluation of ideas transparent amongst those who are participating in the ideation.

When everyone knows how the ideas will be evaluated, they are much more likely to take those aspects into consideration, both in the ideation phase, as well as the implementation phase. This naturally means that you’ll end up with more relevant ideas that are going to be implemented better.

In addition, there are a number of important organizational benefits to people knowing which factors led to certain ideas being chosen and others being discarded. This will lead to: 

• Improved communication
• Organizational learning
• Sense of fairness
• Increased engagement

As the metrics used for evaluating ideas should be aligned with your organization’s strategic or tactical priorities, these will also serve an integral role for holding people accountable and reminding them of what’s truly important for the organization at the given time. These are both key for achieving strategic alignment across the organization, and for being able to effectively execute on your strategic or tactical priorities, depending on the types of ideas being collected. 

Evaluation methods – team effort or an individual decision

There are a number of different ways in which you can evaluate your ideas, which we refer to as evaluation methods. Generally speaking, it’s best to get as many people to evaluate the ideas as possible, as long as they have the necessary background information to be able to accurately do so. This provides you with more data points to use as the basis of your evaluation.

You can then choose to ignore some of the data if you wish to, but in general, it reduces the risk of your evaluation being based on just a single person’s potentially biased opinion.

However, if the idea is more complex to understand, you might not want everyone to voice their opinion on it as this can also lead to irrelevant and poor estimates. So, make sure you approach different kinds of ideas differently – and that your idea management system supports such an approach.

In some cases, such as when evaluating the feasibility or usefulness of ideas related to existing customer-facing processes, it's highly recommended to use the crowdsourced approach to engage as many people from the front-line as possible to see if the idea is actually useful for the customers. It is the front-line employees that, after all, have the best knowledge on this topic.

There are many cases, in which it makes sense to ask experts for their opinion, for example, if the idea is highly technical or complex. This is usually a good balance between avoiding individual biases but still keeping the evaluations relevant.

Occasionally there will be ideas that others might simply not understand without the kind of insight a single person in the organization has. In that case it’s obviously best to simply trust in the research and/or insight of these individuals and proceed to test these ideas without caring too much about what the evaluation would say.

As mentioned, the latter two approaches are often appropriate for some of the more disruptive ideas, which will likely be considered unrealistic or even crazy by most people. If you’re looking for these kind of ideas, a crowdsourced approach to evaluation simply isn’t going to be that useful.

Different kinds of ideas need to be evaluated differently

As we've discussed above, one size definitely doesn’t fit all when it comes to ideas. So, make sure your systematic process for managing ideas takes these differences into account.

We’ve outlined some typical ways of categorizing different kinds of ideas in the next chapter below and will explain how their characteristics should be taken into account in the evaluation process.

Trying to force all ideas through a single funnel increases both the risk and the impact of choosing the wrong ideas.

First and foremost, virtually every organization – no matter how small or large – will always need to have more than one idea management process to take into account the differences between ideas.

For example, the steps needed to implement an incremental improvement idea will be very different from those for building a completely new business. Thus, the evaluation criteria will also be different.

Trying to force all of the ideas through a single funnel is a common mistake all too many companies are making. It will almost inevitably lead to lengthened decision-making time and thus less ideas being implemented. This also increases both the risk and the impact of choosing the wrong ideas to work on.

The first and easiest way to think about the different kinds of ideas is to consider whether the idea is an incremental improvement to something that already exists, be it your product, service, process, tool etc. or whether it would be something completely new, such as a new business, process or piece of technology.

For the purposes of this post, lets refer to the latter as disruptive ideas. They should always have a completely separate process and set of metrics. If you’re using Viima, this is super simple to achieve by simply setting up different ideation boards for each purpose.

Also, every company has priorities of its own, which need to be taken into account in the evaluation process, usually in the form of metrics you choose. These can stem from your strategy or perhaps from the characteristics of your industry. Either way, what works for someone else, usually won’t work for you out-of-the-box.

Choosing the evaluation criteria

As you’ve probably come to understand by now, there are an endless number of variables at play when it comes to choosing the metrics that represent your evaluation criteria, such as your industry, the types of ideas to be collected, your strategic objectives etc.

In addition, you’ll always have to try to balance between making the evaluation too simplistic, or too complex. By using simplistic evaluation criteria, such as a two-dimensional impact/effort scale, you’re likely to miss certain key aspects of more complex ideas and are likely to bias the evaluation towards more “quick win” type of ideas.

If, on the other hand, you choose to use a highly detailed set of, for example 10 metrics, it’s going to take longer and cost more for you to collect the data. The increased amount of data can also lead to analysis paralysis, and is likely to make you overly confident in your estimate, even though it might not necessarily be any better than if you would’ve simply used a few of the most important metrics. Thus, we typically recommend you err on the side of simplicity and start with just a few simple metrics.

Here are a couple of example setups that we often recommend as starting points for companies looking to evaluate their ideas: 

Incremental improvement ideas

• Fit with current (strategic/tactical) priorities
• Business potential
• Effort (cost) to implement

Disruptive and new business ideas

• Fit with the vision or mission of the company
  • They key is to not be too constrained by the current strategy or operations
• Constraints and assets of the organization (e.g. growth targets, schedules, available skills and resources, and current customer base)
  • Choose whatever are the most significant constraints or assets for you, for example:
    • 10x growth requires very different ideas from 10% growth
    • if the company has only 2 months of runway left, it probably doesn’t make sense to focus on ideas that would require multiyear projects to complete
    • if you have a huge customer base, it makes sense to utilize that as an asset
• Potential to dramatically change the status quo
  • The best new businesses change the status quo dramatically, consider using “10x better” as an aspirational goal

Both of these setups are actually quite straightforward. As you can see, most of the metrics can often be very difficult to measure with exact numeric values, in which case you can simply use relative measures to quantify people’s qualitative assumptions, such as a simple scale from 1 to 5. However, especially for incremental ideas, it can also make sense to use monetary values for some of the cost/benefit related measurements if it isn’t too complicated to figure out these values.

Metrics can often be very difficult to measure with exact numeric values, in which case you can simply use relative measures to quantify people’s qualitative assumptions.

With disruptive and new business ideas, the metrics are often highly abstract, which is perfectly fine. It just comes with the territory. Evaluating these ideas is, however, much more difficult and laborous and often requires people who are highly analytical and understand strategy and business well. As mentioned, we usually recommend that these kinds of metrics to be evaluated by a small group of experts instead of being crowdsourced to a larger audience.

If you're looking for new and disruptive ideas, remember that you should primarily be looking to find ideas that polarize the experts' opinions and then proceed to test them in real life.

Evaluation is just a step in the process 

Now that you have a process, as well as a set of metrics in place, you have the basics in place for analysing even a large number of ideas. However, to truly identify the best ones, there’s no alternative to actually testing or piloting the ideas in practice, preferably with existing or potential customers.

There’s no alternative to actually testing or piloting the ideas in practice, preferably with existing or potential customers.

You can, and usually should, choose specific metrics that these tests are looking to answer, and then use the results in a second stage of your evaluation process to determine whether the ideas should proceed further in the process. This kind of a multi-tiered process often makes sense when you have more complex ideas that would require large up-front investment, and is sometimes referred to as the Phase-Gate model.

Prototyping and testing is a topic in which we won't go into too much detail here as there are numerous other resources on it, such as The Lean Startup by Eric Ries.

Regardless of the number of evaluation phases the idea needs to go through, it’s still very much a refined idea at that point. This is when the implementation begins, at which point the idea is usually transferred over to a project or product team to work on, which always have their own sets of practices.

Choosing a tool 

By now you’ve probably realized that systematically evaluating ideas takes quite a bit of work. Collecting data, keeping everything organized, and communicating about the whole process will be an administrative nightmare without the proper tool. The same also applies for using Excel or other tools that simply aren’t meant for the job, such as most project management tools.

If you’re at all serious about developing your business by systematically collecting and evaluating ideas, we’d recommend you get a tool designed specifically for that job.

A good tool keeps you organized and automates many of the otherwise laborous manual tasks. These days, there are quite a few alternatives for this purpose, but we obviously think Viima is the best tool for the job.

Regardless of the tool you end up choosing, make sure it supports at least the following set of criteria:

• Easy to use for both collecting and evaluating ideas
• Support for multiple simultaneous idea management processes
• Freely customizable set of metrics to use for the evaluation of each process
• Ability to collect evaluations from different sets of stakeholders
• Transparency of the evaluation results and ability to communicate and discuss them
• Automatic calculation of evaluation scores
• Visualization and ability to easily compare evaluation scores

Putting it in practice

Since we’ve now got the background covered, it’s time to put the pieces together. To demonstrate how this could work in real life, we’ve put together a fictional example of what a simple setup for a medium sized information intensive organization, Acme Company, could look like.

1. Differentiate between new business ideas and continuous improvement

Acme Company is doing fine as a business. They are profitable and are a market leader. However, their growth has stalled in the recent years. They’ve thus created a new growth strategy, which focuses on two primary drivers: improving customer satisfaction and coming up with new services and products to create additional value, primarily for existing customers. To achieve this, they’ve decided to ask employees for ideas on both fronts and have set up processes for driving these ideas into new initiatives.

2. Creating a simple but effective process for collecting ideas

For the sake of simplicity, we’ll now focus on just one of these aspects, namely the goal of improving customer satisfaction. To begin with, Acme has created a simple but effective process for making sure that those ideas are actually implemented. It begins with the employees that are closest to the customers pointing out issues that the customers are having challenges with, or that they simply don’t like.

As these ideas are categorized, they are automatically assigned to the person responsible for this type of ideas, who is then automatically notified of the idea. They’ll then proceed to ask a number of questions to dig deeper to really try to understand what the idea is about.

3. Evaluating the ideas

After this, the next step is for the person responsible for the idea to evaluate it, together with at least two other persons that are considered to be relevant for this particular idea.

In this case, the ideas would be evaluated on three different metrics, two of which represent the company’s current priorities with customer experience, combined with one representing the time and cost associated with the implementation of the ideas.

• Improves customer satisfaction (0-5)
• Reduces the number of support cases (0-5)
• Effort to implement (0-5)

4. Prioritizing the ideas and choosing the ones to pursue

Like most companies, there are obviously hundreds of ideas for improving customer satisfaction. By keeping the metrics simple and focusing on just the essentials, the next step, namely analysis and prioritization of the ideas, is much easier.

The heads of customer support, marketing and product divisions have formed a simple task force for coordinating the effort and have decided to meet bi-weekly to discuss the best ideas, and to choose which ones to implement. They’ve each prepared for the meeting by choosing three ideas to present.

Their evaluation tool has automatically ranked the ideas based on the collected data, so during the meeting, they can use the tool for easily comparing the ideas based on different metrics. Eventually, they end up choosing which ideas to pursue at the current time.

5. Next steps

At this point, they’ll proceed to mark the chosen ideas as such in their tool, which will then automatically notify all relevant stakeholders of the decision. They can also choose to comment on what the next steps for the implementation are going to be and tell whose responsibility it will be.

As mentioned, this is very simple example of what the process can look like in real life. For many use cases, this might seem too simple, whereas for some, the process might be overly bureaucratic. Regardless of the actual practicalities you end up using, the steps involved are usually going to be quite similar to this example, so consider this as just a starting point for your own design.

Start off simple and remember that it’s an iterative process

Now you'll hopefully have a pretty good idea of what idea evaluation is and how to get started. If you're planning your evaluation process, we recommend you start off simple. As mentioned, it’s better to have too few metrics than too many, and the same goes for the number of steps in the process.

Start by choosing just the metrics that have been proven to work for the types of ideas you’re looking to evaluate. Once you start to see a pattern of missing similar information in decision-making, you can go ahead and add another metric for that.

Your evaluation process will never be perfect, so we recommend you start off simple and then keep iterating on it.

In the end, it’s important to remember that your evaluation process will never be perfect. You’re always going to neglect some important aspects of an idea, as well as consider certain aspects of the same idea that turn out to be completely irrelevant.

In addition, the data that you receive will never be exactly right. You simply need to accept this reality and embrace change as an inevitable part of the process. Developing your idea evaluation process should always be an iterative process in itself. This approach has a number of benefits:

• Simple processes are quicker to implement and more likely to be well received by others
• People are more likely to focus on what really matters
• The number of ideas to evaluate will be higher if the threshold is lower
• You’ll maximize organizational learning by moving fast

To conclude, remember that evaluation is just one part of effective idea management and that even though it can be a tremendously helpful tool, it's not a substitute for actually testing or piloting the ideas!

Idea evaluation is a tool for decision-making, not a substitute for actually testing or piloting the ideas!

For more tips on how to create an effective idea management process, we’d recommend you read our guide on idea management processes.