I was often amazed at how easy my management thought it was to obtain an answer to a simple question. I guess they were right, it is easy, but that is without regard to the validity of the answer. One of the best market researchers I ever had the privilege to work with, Norb Baumann, would say," Ask any question you want and you will get an answer, however, it may not be to the answer to the question you thought you were asking." For instance, in a study I was asked to review for the purpose of explaining some inconsistencies in tile data, there was a question about "How frequently do you remove the hair from under your arms?" This particular wording led to different interpretations among the respondents. Some answered considering only the use of depilatories while others read it as depilatories and/or cutting (razor).

 

As in the case above, words can be interpreted differently, but how you conduct the task can also lead to variance in the data. Consider the following task. In an effort to establish parameters for our laundry detergent lab tests, we needed to determine how much detergent the homemaker used in the wash loads. We also decided to nest this research with some methods research. We determined the concentration through the preferred methods, concentration, and five other commonly used methods. The results are as follows.

 

a. Claimed Usage - usually use

If you ask the homemaker how much laundry detergent she normally uses per load, on average, the response will be:                                                                                                                 .90 cups /Ioad

 

b. Claimed Usage - while using

If you ask the homemaker to record how much product she is using while she is using the product, on average, they will record for white loads:                                                                  1.07 cups/load

for colored fabrics:                                                                                                            1.13 cups/load

 

c. Concentration - average load

If you analyze a sample of the wash water, on average, the usage will be:                1.01 cups/load

 

d. Usage by product disappearance - average load across all loads

Over a given period of time, homemakers are given product and asked to record information for each load of clothes washed. Given the amount of product used divided by the number of loads washed, the average product used per load across the total panel will be:               1.10 cups/load

 

e. Usage by product disappearance - average homemaker

Average consumption is calculated for each homemaker and the overall average is calculated across all households. The overall average will be:                                                       1.15 cups/load

 

f. Homemaker demonstration

If homemakers who usually measure in some way the amount of product used in their laundry, are asked to show how much they usually add to their washer, using their own measuring device, the average volume across homemakers who measure will be:                                       1.20 cups/load

Within the six commonly used methods, we see a variation of 33%. The words and the task can have a profound effect on your data. It is not easy to design and execute this type of research. The task requires experience and knowledge. Unfortunately we are too often concerned with speed and cost as opposed to validity.

 

In response to the variation in the data, one of the managers involved in the discussion pointed out that the Confidence Limit would reflect the variability. Not true, the C.L in almost all cases would be based on the within method variance. Seldom will you see multiple approaches utilized to assess the between method variance.