Discriminators and Non-Discriminators

Rather than just looking at numbers, I prefer to try to look beyond the numbers and into understanding.  Recently in a conversation about the results of a paired comparison blind test, a friend said that there was no difference between the two products.  I was concerned and asked what led him to believe that there was no difference between the two.  He said it was a 50/50 result and therefore, no difference.  Yes, no preference advantage but that does not mean that there was no difference between products and that the participants did not see the difference.  There may well have been a great deal of difference and that the difference was unimportant or was important between two equal sized groups.

I proposed to my friend that he may want to repeat the same test with the same respondents thereby creating a pair repeat format.  With this technique we could at least identify the proportions of discriminators and non-discriminators.  With a 50/50 result we could have a range of discriminators as low as 0% and as high as 100%.  If it is important enough to test the two products, it should be important enough to at least identify the ratio of discriminators.  It would seem to me to be an injustice to the program to just say there is no difference when in reality there may be a very big difference and that difference was seen and appreciated differently by different groups within the population.  It is all about doing real research.  Would you take the same course of action if "no one saw a difference" or if "everyone saw a difference but were in disagreement about which was the best as in the choice between chocolate and vanilla ice cream"?  (Note:  While my methods of choice for identifying the ratio of discriminators to non-discriminators are the Triangle Difference and the Duo Trio techniques, the Pair Repeat in this case was going to be more informative since preference was the basic issue.)

I do not have a current example of my own to show but the following example is from Dr. Richard Fox's Market Research book.  (Which I highly recommend as a resource.)

The following is the voting of 400 respondents in a pair repeat test.
 

The probability of non-discriminators is equal for all four cells.  We have two good estimates of non-discriminators, those who switch preferences, 80 and 70.  The best estimate of non-discriminators per cell is 75 [(80 + 70)/2].  Therefore there are expected to be 300 non-discriminators out of the total of the 400.  The estimated true percentage of discriminators is 25%.  While a majority of panelists were considered non-discriminators, Product A maintained a 65/35 advantage among the discriminators [(140 - 75)/100].

The pair repeat method can add greatly to the learning and understanding of preferences.  If you are not already using this technique, I recommend it as an addition to your tool box.