Do the Jury and the Panelists Really Disregard the Evidence?

We often hear the reference to the judge instructing the jury to disregard the previous comment, but can they really blank it out in their minds? The same goes for consumer research. But we do not even have a judge to help us and we don't even attempt to instruct them to disregard the previous comment (or in our case, questions). We should ask ourselves how often do we inadvertently present evidence that could bias the jury (panelist). I frequently encounter research that is meant to be "evaluative" and projectable but because of the introduction of a bias, it becomes an "experimental" project, that is, it is a directed interest mode of research. We inadvertently directed the respondent's interest in a direction that would not have been taken had we not added some element to the study the respondent would not under normal circumstances have considered.

The most frequent introduction of bias comes from two areas: screening and in direct questions associated with multi-level testing such as sequential single product tests.

All too often we are interested in obtaining as much data as possible for our dollar of investment. In this mode of thinking we ask many questions where a substantial number are in no way related to the problem at hand. Consider the screening for a cake mix study. The eligibility may be that the respondent had made a cake from a mix within the past three months. But instead of asking just the pertinent questions, we ask other questions just because the potential is available. We may ask questions about the importance of specific attributes, about failures of the category, what they think about the importance of specific attributes (that we list) in the purchase of a cake mix, etc. In each case we run the possibility of raising awareness to a particular attribute that the respondent would never have considered in the evaluation of the test product. Imagine the thinking of the respondents, "If they think it is important, it must be. So I better watch for it (whatever it is). This is another case of "There is no such thing as a Free Lunch."

Another area where we frequently introduce bias is in the area of Protomonadic testing (sometimes referred to as sequential single product testing or sequential pair testing). All too often, in an effort to maximize the information collected, we will ask attribute or feature questions following the use of the first product and before the use of the second product that changes the way the respondent evaluates the second product. As a result we usually get a very big order effect in the data. The respondents use and evaluate the first product on the basis of their equation of excellence. However, when we ask them questions about features and attributes they never considered or felt important, all of a sudden these new factors enter into their assessment of the second product. They will assume that since you asked about the attributes they must therefore be important. Now they will start out using the second product with the new features firmly planted in their minds. We don't and can't say disregard what we have asked you and evaluate the second product exactly on the basis that you used with the first product.

We jeopardize the study to obtain additional data and "NO," balancing will not eliminate the problem.