Pair Tests Revisited

Last week a friend was telling me about a test he was proposing.  It seems that a new brand was introduced into one of his brand's product area.  In an effort to determine the potential effect of this new brand on his brand, he was proposing a paired comparison blind test and wanted to know what I thought of the proposal.

I had two thoughts:  First, the paired comparison would determine the consumer's choice between his brand and the new brand.  Second, in no way would the study give any information about how the two brands would fare in the marketplace.  Pair tests are all about choices.  If you want an evaluation of a brand, you conduct a single product test.  Pair tests create an artificial environment whereby the importance of similar attributes are down-played while the importance of differences are exaggerated.

Following are excerpts from a Views written on September 3, 1994, relating tot his topic:

There are two common problems with the use of the pair test. Both problems are frequently overlooked by the users. First, the preferred choice in a pair test, does not reflect market success. It does not even indicate which of the two has a better chance in the marketplace. "Discrimination does not equal importance." The market configuration will determine success. As an example, if we are testing a green vs. blue shampoo, and color is a dominating factor, the blue shampoo may be significantly preferred to the green shampoo. But if the market has multiple blue shampoo brands and no green shampoos, the introduction of the blue candidate will split shares among the other blue brands, while the green shampoo will capture the green population. L. L. Thurstone wrote volumes on this subject in the early 1930's.

The second problem deals with carry-over effects. In a single pair test, the carry-over effect is seen as an order effect while in a round robin analysis, it is usually reflected as a product X order interaction. Basically, this effect says that the results are determined by the order of use. For example, if we are testing a laundry detergent against a laundry detergent with an additional active such as bleach or fabric softener, we would expect to see misleading pair results (significant order effect). The panelists using the regular detergent first and the detergent with bleach second will report a substantial win for the detergent + bleach, something like an 80/20 win. However, among the panelists using the products in the opposite order, a smaller difference will be reported because the detergent + bleach will clean up much of the redeposition and stain problems so that when the regular detergent is used, there will be very little visual differences in the results (55/45). The reported result would be a 68/32 when the 80/20 is probably closer to the truth.

For further information on this topic go to www.popsg.org/views.