A poli sci professor's assault on polling as unreliable

Stanford's professor of Public Policy Justin Grimmer relates a funsome tale of male cat fighting, at least on Nate Silver's side, who seemed to resent another feline using his litter box.

Even as Joe Biden’s presidential candidacy teetered and polls showed him clearly losing to Donald Trump, the election forecasting site 538 was still estimating that Biden was likeliest to win. It was a conclusion based on odd modeling assumptions that led the site’s original founder, Nate Silver, to declare the 538 model "very obviously broken" and for the site’s new chief to acknowledge an adjustment to its model when it relaunched with Kamala Harris’ candidacy.

Grimmer recalls the incident not just to entertain us. He's a political scientist who develops "machine learning methods" to make forecasts, though not about presidential races. His principal argument is that "we don’t have nearly enough data to know whether these models are any good at making presidential prognostications." 

Because the results are highly correlated — 'the estimated probability of an event happening corresponds to how often the event actually happens' — under best-case settings, determining whether one model is better than another at the state level can take at least 56 years — and in some cases would take more than 4,000 years’ worth of elections. The reason it takes so long to evaluate forecasts of presidential elections is obvious: There is only one presidential election every four years. In fact, we are now having only our 60th presidential election in U.S. history.

An example of modeling conflicts is the Economist's, below, updated today, and Silver's last update, which had Harris' Electoral odds at 45 to Trump's 55.

Prof. Grimmer also has a couple warnings about political forecasts. They both have the potential to drive down turnout. One is that voters tend to confuse a model's forecast of, his example, a candidate's 58% chance of winning with the candidate's vote share. They assume a 58% probability means 58% of the vote, which it most certainly does not.

His second warning is similar, although this one is all too obvious, more perilous and empirical. When voters see a high probability for their candidate's victory, they're more likely to stay home out of overconfidence. I discussed this phenomenon in relation to the Harris campaign here.

I wouldn't argue with a political scientist whose playtime is devoted to the development of "machine learning methods." But I'll note two items of contrast. Aggregates come, of course, from solitary polls, each with a margin of error, usually around 3 points. Models use polling data, too, hence a fudge factor is built into all three. And few readers accept any of them as omniscient Holy Scripture or realism art. They're just sketches. 

I'll also mention that before polling, before aggregators and models, there was this thing called instinct — gut, political instinct. Mine have served pretty well, with the black-swan exception of the 2016 presidential contest. If blame for diminishing the reverential value of instinct is to be cast on the modern racket of polling writ large, the net should fall on George Gallup and his mother-in-law, for whom George did his first surveying — the first political survey ever — in the early 1930s, when she was running for a statewide Iowa office. 

So I'll continue presenting criminally flawed polling, its awful aggregates and modeling forecasts from the world's most diabolical wretches — those swamis who employ "far more punditry than [precision]" then they'll admit, says Grimmer — along with commentary about the results, however feeble and thoroughly wrongheaded that may be.

Comments

[Anne J]

Don't polls ever come with explainers? Some note about how they reached their conclusions and what those conclusions may mean?

I have only been visiting your site for about 21 years, (thank you Michael Moore) so I am curious about what your political instincts were telling you in 2000, an election that was thrown to, and by, the supreme court.

[PM]

I thought Gore would walk away with the prize — which, by rights, he should have — but I never thought the Supreme Court would take it upon itself to "appoint" the next president, Anne. I'm curious about your thanks to Michael Moore. I don't recall an endorsement from him. news to me.

[Anne J]

I used to go to his web site and he had a list of other sites to read and the late Buzzflash was one of them and they would link to your site. So maybe I should thank Buzzflash too. But I honestly can't remember if he had a link directly to your site or not. But his website was the starting point that led me to yours.

[PM]

Thanks for the update, Anne. Poor Buzzflsh. I think I killed it. It hired me to write a daily column in the run-up to the 2008 election and post-election period. I was rabidly pro-Obama in the run-up, which really pissed off Buzzflash's Clinton supporters. Given readers' choice to read only that which agrees with them, the site became persona non grata with a huge chunk of its previous readership, which Buzzflash relied on for contributions. So it soon folded.