Five Steps to Effective Idea and Product Testing
It is critically important that banks actually test in market their big ideas to ensure that they really will work. This can help de-risk big decisions and provide empirical information about how to proceed with the idea. (Think for example of USAir's announcement that it would charge for soda and subsequent embarrassed back-pedaling as an example of the harm of not testing.) Experimenting in market to de-risk ideas is one that is increasingly accepted as common practice.
However, while testing is important to do, it is very hard to do well. Retail banking is a noisy environment, making it hard to isolate whether a tested idea caused a change in performance. As a result, too many banks roll forward every idea they pilot, a sign that they aren't really learning from the pilot. There are five steps that I think lead to effective tests.
1. Test the right ideas. Testing resources are limited. There are only so many tests that a bank can devote attention to at a given time, and there are a limited number of branches and customers available for testing. Focus testing on ideas that meet at least these two criteria: (1) is this a big enough idea to matter to the business, and (2) will testing help us make the decision (i.e., have we not already decided we are proceeding regardless but truly want test results before we make the decision).
2. Agree on success criteria in advance. A frequent reason banks struggle to decide on whether to roll forward a tested idea is lack of clarity on the definition of success. Advocates of the idea then latch on to any indicators of success and resistors find data to support that the test is a failure. The two groups then waste time talking past each other. By defining success in advance, banks can simplify the decision process and take some of the emotion out of the ultimate decision.
3. Design the test well. Tests need to be designed properly to provide accurate results and inform go-forward decisions. First, the test needs to have the right number of customers, branches, or markets. If a test is too small, then it will be challenging to confidently measure the impact, and tests that are too large can be more expensive and impose more risks. It is important to find the sweet spot, which will vary depending on the nature of the action tested. Second, test customers or branches need to be selected to be representative of the network as a whole across key dimensions. If the test is biased in some way (e.g. we only test in city center locations), the results won’t hold when we extend the idea across the network.
4. Measure impact across the household, branch, and business as a whole. Understanding the true impact of a test requires additional measurement beyond the specific product and metric tested. For example, when testing a new fee strategy, you should measure not just fee income but also the impact on other products, retention, new account generation, customer satisfaction, and more. Any given action can have broad consequences, and test measurement should examine the impact on the business as a whole.
5. Identify drivers of success and target rollout. Testing should not only produce a go/no-go decision. Any initiative will work better with some customers than others or in particular types of branches or markets. Some ideas need to happen across the business to maintain a consistent brand and customer experience. However others, such as where to add staff or invest capital or increase marketing do not require uniform treatment. By only extending the idea to the locations or customers where it will work, banks can generate substantial profit gains over an all or nothing approach.
Patrick O'Reilly is president and chief operating officer of Applied Predictive Technologies, a provider of market testing services.
Tweet This