It's New, It's Relevant, It's Scary

During the past nine years my most frequently given talk has been "Researching Research."  In this talk I cover the four most important sources of error in consumer and market research we had found in a 15-year program aimed at understanding variables affecting our research.  One of these variables is sampling.  In the presentation I cover the wave off rate in Mall research as well as the refusal rates in various protocols such as central location testing, telephone interviewing, mail panel research, in-store research, etc.  Now a sampling problem from a different perspective has presented a very ugly picture.

I call your attention to the Journal of Marketing Research, May 1999 pages 286 to 294.  The title of the article is, "The Distributing of Survey Contact and Participation in the United States:  Constructing a Survey-Based Estimate."  In this article, the authors find that survey contact and participation rates are lower than previously assumed and that a small percentage of the population is completing the majority of surveys.

Their findings are that in a year, between 20% and 23% of adults are accounting for ALL survey responses, and between 4% and 5% of adults account for more than half the survey responses.

How are these people different (the 23% or more importantly the 5% from the non-participants)?  Do the differences have any effect on the research conclusions?  Would the results from the non-participants be different or lead to a different conclusion than the results from the participants?  Are we misleading our clients when we say we have a random sample without clarifying that it is a random sample of a minority segment of the population?

Is it irresponsible for us to continue to use research protocols without sharing the risk with the client?  For instance, should we not alert the client to the fact that Mall Intercept research is limited to about 5% or less representation of the U.S. population?  It is only representative of people who shop at malls and are willing to be interviewed.

Caution:  There are practitioners out there who will tell you that they cant take the completed interviews from a study that contains a sampling bias and balance the data to correct for bias.

Do you really believe that?  Think about it.  if you have a study of a new tool that is rejected by left handers who will not even participate in the study, do you think that someone can insert data into the study to compensate for the lack of left handers?  Ask yourself, how does the researcher know it is left handers and how does the research know what data to insert?

Coming soon:  While I don't know of a solution to the refusal bias, I am aware of at least tow ways that have been used to minimize the problem.  I will cover both protocols in the near future.