Views from the Hills by R. E. Stevens, GENESIS II (The Second Beginning) E-Mail views@aol.com

Testers vs. Users

I have often wondered how frequently a real product improvement is prevented from entering the market because of the research design and not the product or idea itself.

One of the main problems I have seen in consumer research is that the researchers seldom take into consideration a potential test environment bias in their research. It is my contention that when you ask a person to participate in a test, you affect how that person will use, look at, and evaluate the product. I believe that this tester environment is a valuable asset in the brand development stage but not in the market potential evaluation stage of assessment. In the market potential evaluation stage, the product/brand should be assessed under the more natural conditions.

I can best describe the above with a project conducted some years ago. At that time we had a product up-grade ready for test market. When comparing the up-grade with the current version, the up-grade held an 80/20 advantage. As we were preparing for the Test Market, our physical properties evaluation of the brand in the warehouse revealed a potentially serious outage* with the up-grade. Faced with a long delay, we decided to determine the awareness and reaction to the outage. Two testing approaches were utilized. The first was a subjective consumer assessment (Mrs. Jones technique**) and the second was a direct paired comparison of the current vs. the up-grade with the outage as a part of the assessment. In both cases, the up-grade results were less than desirable, especially in the Mrs. Jones study where it was a complete failure.

Still faced with a long delay on a valuable upgrade, we convinced our management to conduct an unorthodox (at this time with this company) third study to assess the outage awareness under real market conditions. This study involved the placing of the brand in six supermarkets, identifying the purchasers, and interviewing the purchasers by phone two weeks later. An important part of the protocol was to insure that the participants were not testers. They did not know that they were selected because of their purchase of the test brand. The five hundred and seventy-five (575) interviews indicated that the outage was not a serious problem. Not once was outage mentioned in response to a series of questions involving the awareness of change, product dislikes, intent to purchase, etc.

Based on the third test, we went to the market with the outage problem while it was being corrected for future production. After two months in the market, about the time the outage product was liquidated, we had not received a single written or 800-line complaint about the outage. A final note, the brand went on to become the category leader, based on the product change.

It's important to know if your problem at hand requires a test environment assessment or a real world environment assessment. It does make a difference.

*The amount of empty space in a container.
**A projective technique where the respondent projects feelings, beliefs, or motivation onto another person, object, or situation.


[Back][Index][Forward]