Making Sense of Polls and Polling
“Do you ever get the feeling that the only reason we have elections is to find out if the polls were right?” — Robert Orben
There can be little argument that this election season is both historic in its significance as well as in its level of negative discourse. One aspect of the campaign that has been under particular scrutiny has been the accuracy (or complete lack thereof) of polling numbers. With corporations, partisan campaigns and advocacy groups among the pollsters, it’s hard to know which results are trustworthy and which should be viewed with a grain of salt.
In a land without land lines
The attention on polls has continued to grow since Fivethirtyeight.com’s chief analyst Nate Silver drew praise for his extremely accurate election predictions in 2008, 2010, and 2012. But Silver’s methodology has come under fire this season, as polling organizations have to move away from their reliance on phone polling, once a reliable and accurate sample, but now becoming extinct as people drop their landlines. Further, there are restrictions for how pollsters can contact cellphone numbers — “robocalls” to cellphones are prohibited by law.
Probability sampling can fail
Online polls are notorious for being inaccurate and easily manipulated. YouGov and SurveyMonkey have tried to address this using probability samples and adjusting nonprobability samples to certain target populations. But this type of strategy led to a completely inaccurate prediction for the 2015 U.K. general election, when an independent inquiry by the British Polling Council and the Market Research Society concluded “unrepresentative samples were the primary cause of the polling error.”
We still don’t know
Mollyann Brode, president of the American Association of Public Opinion Research said in the Wall Street Journal, “We don’t know how reliable these new methods are. We don’t know how replicable they are. They seem to work in some cases. But we don’t know when they’re working and when they’re not working.”
Mixed methods are best
In his 2016 “State of the Polls” Silver said, “Although internet polls show promise as a potential alternative, they do not yet have a long enough or consistent enough track record to be placed on the same pedestal as high-quality, live-interview telephone polls, based on our view of the evidence.” Many pollsters now use some combination of telephone, mail, and internet samples – and statistically adjust results to better approximate views of the entire public.
How one guy in Illinois skews a prominent poll
In one particularly telling case of probability sampling gone wrong, a New York Times article dissected the way a single African-American Donald Trump supporter and panelist on the U.S.C. Dornsife/Los Angeles Times Daybreak poll has skewed their poll results in Trump’s favor, where most other polls nationwide favor Hillary Clinton. The article explains, “in some polls, he’s weighted as much as 30 times more than the average respondent, and as much as 300 times more than the least-weighted respondent.” In this case, weighting for a tiny group, i.e. 18-21-year-old-men, is both unusual and problematic because the weights assigned to those respondents will be extreme. Additionally, the U.S.C./LAT poll used self-report of past voting behavior which is very uncommon, since respondents often mis-report past voting behavior. Last but not least, the U.S.C/LAT poll uses a panel, which means the same people are included in multiple waves of polling surveys. All these sampling decisions have led to this poll being an outlier among polls nationwide.
At least we can laugh
At least polls are providing fuel for humor in an otherwise very contentious presidential race. On his “This Week Tonight” show, John Oliver focused his attention on Trump’s practice of citing non-scientific online polls that said he won the debates. Fortune noted, “Oliver pointed out the obvious problem with those polls: they don’t necessarily target likely election voters and allow Internet users to vote anonymously over and over again, if they so choose….even a Fox News executive sent around a memo to certain staff last week reminding them that such polls ‘do not meet our editorial standards’.”
The role of qualitative research
Ultimately, we need to remember the original intent of political polling, which is to reflect the extent to which candidates are connecting with the issues and concerns that are top of mind for their constituents. Polls are essential in gauging which way the wind is blowing, but the crucial context that qualitative research provides as to why people feel as they do should never be underestimated. Without qualitative insights, “polls can be inconsequential, and occasionally wrong.”
