Views From The Hills of Kentucky :: Pair vs. Single Product Testing

Views from the Hills by R. E. Stevens, GENESIS II (The Second Beginning) E-Mail views@aol.com

Pair vs. Single Product Testing

Recently I was asked which method I prefer, pair or single product testing. The question took me a little by surprise. I did not view the choice as a preference, but rather which was appropriate for the problem at hand. While a book could be written on the differences in the two methods, I would like to cover what I believe to be the critical issues when selecting one method over the other.

What questions do the two methods answer?

Pair tests deal with choices between two stimuli.
Single product tests deal with assessments.

There are two common problems with the use of the pair test. Both problems are frequently overlooked by the users. First, the preferred choice in a pair test, does not reflect market success. It does not even indicate which of the two has a better chance in the marketplace. "Discrimination does not equal importance." The market configuration will determine success. As an example, if we are testing a green vs. blue shampoo, and color is a dominating factor, the blue shampoo may be significantly preferred to the green shampoo. But if the market has multiple blue shampoo brands and no green shampoos, the introduction of the blue candidate will split shares among the other blue brands, while the green shampoo will capture the green population. L. L. Thurstone wrote volumes on this subject in the early 1930's.

The second problem deals with carry-over effects. In a single pair test, the carry-over effect is seen as an order effect while in a round robin analysis, it is usually reflected as a product X order interaction. Basically, this effect says that the results are determined by the order of use. For example, if we are testing a laundry detergent against a laundry detergent with an additional active such as bleach or fabric softener, we would expect to see misleading pair results (significant order effect). The panelists using the regular detergent first and the detergent with bleach second will report a substantial win for the detergent + bleach, something like an 80/20 win. However, among the panelists using the products in the opposite order, a smaller difference will be reported because the detergent + bleach will clean up much of the redeposition and stain problems so that when the regular detergent is used, there will be very little visual differences in the results (55/45). The reported result would be a 68/32 when the 80/20 is probably closer to the truth.

Other areas of concern with pair testing involve the following:

Respondents focus on differences and ignore similarities.
Brand awareness of one or both the products.
Time required in some categories to assess both products.
The addition of a revolutionary attribute in either one or both products.

Take Aways:

Pair tests=choices; Single product tests=assessments
If you have an order effect in a pair test, you should have used a single product test.
Never use a pair test to assess market relevance. (Never may be a little strong.)

The above comments are relevant to Blind Testing and do not address designs such as: Brand Identified, Directed Interest, Concept & Use, Acceptability & Theme Support, etc.