With the election in Georgia approaching fast, polls are beginning to appear every week. Unfortunately, many of these polls are taken at face value. The reality is that at this point there is not a single pre-election poll that has demonstrated credibility. This does not necessarily mean that polling firms and newspapers are simply fabricating their data -- it simply means that if they were simply fabricating their data, it would be very difficult for anyone to know.
So can we be confident that a poll is credible? There are a number of basic stipulations:
1. Reveal the sampling methodology. How, in other words, do the pollsters ensure that interviewing a few thousand people is representative of the entire electorate? Choosing respondents requires a) knowing where most people live, and b) having a very strong theory about which people are likely to turn out to vote on election day. This is very difficult stuff, and even tiny errors here can have tremendous consequences.
2. Tell us about the field work. Were the interviews done face to face or by telephone? When and how? Did the survey enumerators explain who they were working for, and is it possible that the respondents knew that they were looking certain answers?
3. Publish the questionnaire. What exactly was asked, and how, and in what sequence?
4. Document the non-response rate. How many people refused to answer? There are plenty of people who don't pick up the phone, or who don't have 30 minutes to talk to pollsters...and in this country, many of those people will vote.
5. Allow peer-review. Power point presentations for nonspecialists are fine, but make the data set available to peers for professional scrutiny (and of course you can restrict usage). If you really are confident in what you're doing, this is the way to go.
If polls do not meet the standards, they really do not deserve to be taken seriously.
Too many commentators forget that the burden of proof is on the polling firms, not on the public. We seem to be entering a dangerous cycle, where there is a lot of awful information floating around, and no one has the ability to sort the good from the bad. This is as much a problem with what the public is demanding as what the firms are supplying. The public should beware, and commentators should be very cautious about taking firms' power point slides at face value, until some basic methodological questions are answered transparently.
Finally somebody is raising the real issues with those polls! However, it is not enough to say this in a blog - this should be in the major newspapers!
ReplyDeleteIt's amazing how political commentators of various stripes are jumping right into commentaries without addressing these basic questions first. This only proves how little knowledge does the public have about polling and statistics. Instead of providing objective knowledge of the current state of affairs, these polls only serve to further confuse and polarize public opinion. Astonishing! Supposedly reputable people are commenting on the movements in numbers without having a slightest idea about where different polls stand in terms of sampling, nonresponse, let alone being interested if the margins of error in individual polls are known and how big they are.
G.G.
recently, we found an interesting piece on Greenberg Quinlan Rosner's (one of the leading polling firms) website. It describes what experienced pollsters look for: http://www.greenbergresearch.com/index.php?ID=977.
ReplyDeleteThis is, of course, assuming that all the other methodology is right. We thought it could be of interest to readers who want more details.
circling back to this many years later, and generally this is still on track. The one thing I would add is that original language questionnaires should be available, and that where possible a broad budget breakdown would be good. This may be uncomfortable for some institutions, who do not want to give away their cost structure, but would help to assess how much effort was put into particular components of work.
ReplyDelete