Sequential Monadic Testing -- Researching Research
Prior to the mid 1960's the state of the art in consumer research was the Paired Comparison Blind Test. In the mid 1960's, the Single Product Blind Test started to emerge as the prime tool for assessing acceptability of a product.
In the early 1970's, in an effort to get more bang for the buck, the Sequential Monadic Test design emerged. The SM design took on various protocols for interviewing where following the use of the first product either a complete battery of questions was asked or just the overall assessment question plus an open-ended question about likes and dislikes were asked. Following the use of the second product, some people would only conduct a paired comparison interview while others would conduct a single product interview, followed by a paired comparison interview.
In the late 1970's, I conducted a research program aimed at understanding
the effects of the different interviewing protocols. The focus was to understand
the order effects in the SM design. Two actions were taken as a result
of the research. Initially, we dropped the asking of direct questions (questions
about specific attributes) following the use of the first product and prior
to the use of the second product. We found that by asking questions about
specific attributes, we affected how the respondents assessed the second
product that was different from the criteria used in the assessment of
the first product. Consider the following data.
Product | Base | Ratings | ||
Used First | Used Second | Average | ||
A | 163 | 6.84 | 6.25 | 6.55 |
B | 159 | 6.53 | 6.48 | 6.50 |
C | 156 | 6.51 | 6.38 | 6.44 |
D | 162 | 6.89 | 6.44 | 6.66 |
Range | .38 | .23 | .22 |
Which set of data would you use: the results of the first use, the second use, or the average? (We used the results of the first product interview. This obviously defeats the purpose of the second placement.)
There is no doubt that you get more numbers for your research dollar with the Sequential Monadic design, but at what cost? I encourage anyone utilizing the Sequential Monadic design, to look at their data by order of use (questioning) to determine if they are introducing an element of error into the results via the protocol. Many will say that by balancing placement, they eliminate the order effect. True, if the order effect is an average order effect. Not true, if it is a product by order interaction (which in my experiences happens most frequently). A product by order interaction can have disastrous implications.
Later, we encouraged the reduction, if not elimination, of the Sequential Monadic design. The reasoning was that we were trying to reap the benefits of both the single product and paired comparison design. These two types of research serve different purposes, (see Views Pair vs. Single Product Testing September 3, 1994), and my experience is that you need to run them separately to achieve the desired research results.