Views From The Hills of Kentucky :: Two Firms Concede Errors in Exit Polls

POPSG
home

MRlibrary

Sorensen
Associates

Views

Researcher
Resumes

In-store
Bibliography

Two Firms Concede Errors in Exit Polls

March 15, 2005 - by Robert E. Stevens, GENESIS II (The Second Beginning) E-Mail: views@aol.com

The statement, "Two Firms Concede Errors in Exit Polls" was a front page article in the January 20, 2005 Cincinnati Enquirer. My first reaction was "so what, we knew that." Actually anyone following the election day polling results knew this. My second reaction was more analytical, that is, what will participants and especially users of market research think about our profession? In this polling that lasted about a total of 12 hours, the researchers invested 1,400 interviewer days to come up with a prediction that picked the wrong man, and had an error of approximately 13%. Unfortunately for the researchers and the clients, validation of the results was almost instantaneous. They knew the true results within 24 hours.

In the analysis of what went wrong, the researchers cited a number of reasons such as:

The Kerry voters were more willing to be interviewed.
The interviewers were too young.
The interviewers were too inexperienced.
There was a programming error that led to too many women voters being interviewed.

The researchers, however, only recommended better training and more monitoring of the interviewers. They totally ignored their sampling model. There are solutions to these problems. The solutions are not easy and they definitely add to the cost of the research. From my perspective, additional information needs to be obtained beyond what the respondents had to say. There are two populations at play, first, those being interviewed and second those voting. The disproportionate number of women respondents should have been determined very early in the process so that adjustments could be made to the results before publication. The willingness to be interviewed in my estimation is the difficult problem but still do-able through sample control and data analysis.

I am amazed at how infrequently we verify the assumptions we make when designing the research. It seems to me that if it is important enough to build in a research parameter, it should be important that we verify the assumption is actually in-play. Build in checks and balances to verify your assumptions.

Early in my career, while auditing a research project for another corporate division, I asked the designer why some specific test parameters were not being collected and recorded. His response was that he did not want to know and he definitely did not want the client to know if some of the test effects were significantly affecting the results. I did not consider this individual to be a researcher because of his willingness to ignore accepted standards, in my mind, he became a clerk collecting numbers and not data.

I do have sympathy for pollsters. Their results, unlike those I had to deal with, are validated very quickly. Personally, considering my history, I do not think I would have liked being involved in a system whereby validation is almost immediate. I prefer spending time getting to know and understand my data before drawing conclusions.

Sponsor: Sorensen Associates Inc Portland, OR: 800.542.0123 Minneapolis, MN: 888.616.0123
the in-store research company™ -- Dedicated to the relentless pursuit of WHY?

POPSG
home

MRlibrary

Sorensen
Associates

Views

Researcher
Resumes

In-store
Bibliography