Myths and the reality of election forecasts

It is difficult to comment on the predictions of most Indian pollsters as they do not publish their methodology

The credibility of pollsters have undermined quite a bit. In 2020, U.S. (President-elect) Joe Biden won the election, and not many people are looking back to the predicted margin in nation-wide popular vote share which was huge (mostly within 8-12%), but never materialised.

Poll predictions have failed miserably on many historical occasions. For example, before the 1936 U.S. presidential elections, a reputed magazine, the Literary Digest, conducted an opinion poll survey with a massive 24 lakh samples, and predicted 57% vote share for Republican Alfred Landon, and 43% for President Franklin Roosevelt. In reality, Roosevelt got 62% votes against 38% favouring Landon. This episode was a lesson about the importance of ‘selection bias’ in sampling.

Sample size, sampling frame, method of sampling and estimation, and of course summary statistics across different margins of variability are not provided by most of the pollsters making it almost impossible to comment on the quality of their predictions, from a statistical point of view.

Atanu Biswas

Several exit polls have predicted a clear majority for the Bharatiya Janata Party in U.P and for the Aam Aadmi party in Punjab with Uttarakhand still sending mixed signals. However, exit polls which are mostly forecasted on the basis of random or systematic sampling have proven not to be always accurate. In this article dated November 12, 2020, Atanu Biswas outlines how Indian pollsters fall short due to their faulty methodologies.

Richard Charnin’s 2010 book, Proving Election Fraud: Phantom Voters, Uncounted Votes, and the National Exit Poll, investigated possible irregularities in Democrat John Kerry’s defeat in the 2004 United States presidential election. While political pundits struggled to explain the election outcome and claimed the polls “behaved badly”, Mr. Charnin considered the possible phantom voters and uncounted votes in the election procedure. However, the final national exit poll was the real “Smoking Gun”. The difference between the vote shares of John Kerry and George W. Bush was 6.5% more than the actual.

Global examples

Interestingly though, such was the credibility of exit polls even one and a half decades ago. Take the example of the Venezuelan recall referendum of 2004 to determine whether Hugo Chávez, then President, should be recalled from office. A huge discrepancy with the exit poll created a massive uproar worldwide.

The “Orange Revolution” in Ukraine during late November 2004 to January 2005 was also in the immediate aftermath of the presidential election. The allegations of electoral fraud were strengthened by several exit polls exhibiting a substantial lead for Viktor Yushchenko. Similarly, widespread protests over the disputed parliamentary elections of 2003 triggered the “Rose Revolution” in Georgia, and culminated in the ousting of President Eduard Shevardnadze.

One wonders if polls predictions are still so trustworthy. Certainly not. In 2020, U.S. (President-elect) Joe Biden won the election, and not many people are looking back to the predicted margin in nation-wide popular vote share which was huge (mostly within 8-12%), but never materialised. Also the credibility of pollsters must have been undermined quite a bit in between. People possibly learned to listen to them, but not to keep much faith in their predictions.

In reality, the difference in popular vote shares in this U.S. election is just about 3%. And, in most of the States, the difference between the predicted percentage of votes for Mr. Biden and U.S. President Donald Trump compared to Mr. Biden’s final lead is more than 3%. Should not this be regarded as a huge failure to gauge the pulse of the electorate?

The Bihar mandate

What about Bihar? Pollsters performed miserably in Bihar in 2015 in one of their worst performances in Indian elections. Although the opinion polls mostly predicted a victory by the National Democratic Alliance this time, very few of them could guess the quantum. The exit polls, on the other hand, mostly predicted a Rashtriya Janata Dal+ victory . A funny thing is that the minimum and maximum predictions for the RJD+ were 76 seats (a range of 71-81) and 180 seats (a range of 169-191), out of 243 seats. One wonders whether the RJD+ securing anything less than 71 or more than 191 was at all practicable. Thus, there will be some pollster or other to claim a perfect or near-perfect prediction.

Surveys and lessons learnt

Yes, poll predictions have failed miserably on many historical occasions including some which were in the developing process. For example, before the 1936 U.S. presidential elections, a reputed magazine, the Literary Digest, conducted an opinion poll survey with a massive 24 lakh samples, and predicted 57% vote share for Republican Alfred Landon, and 43% for President Franklin Roosevelt. In reality, Roosevelt got 62% votes against 38% favouring Landon.

This episode was a lesson about the importance of ‘selection bias’, as the individuals under study were affluent people having a telephone, club memberships and magazine subscription in that era; Roosevelt had less support among such people. Also, people understood the importance of ‘non-response bias’ as only 24% of the people who were approached actually responded to the survey.

Similarly, the severe failure of Gallup’s poll prediction of the 1948 U.S. election taught pollsters about the importance of ‘random’ sampling. In 2016, Mr. Trump supporters were under-counted. Later, there was an attempt to label that “shy Trump factor” or “hidden Trump vote”, similar to the “shy Tory factor” in the United Kingdom — a term which became popular after John Major’s victory in the 1990s.

Certainly, the winner of opinion/exit polls is not necessarily the winner in the election. Hillary Clinton, Ed Miliband or Atal Bihari Vajpayee could vouch for that. Most of the opinion polls of the 2009 and 2014 Lok Sabha elections either failed to project the winner or to foresee the margin of victory. A small State like Delhi has caused misery for pollsters, be it in 2015 or 2020. In the U.K., for example, no major survey could predict the victory of Conservatives in 2015. And the history of the Brexit referendum is a part of folklore, for sure.

Over decades, pollsters around the world have transformed poll prediction into a funny game. When Warren J. Mitofsky conducted an exit poll in a local election in Kentucky in 1967 for CBS News and also conducted the first national exit poll in 1972, he never envisioned such a future within half-a-century, for sure.

Posing the questions

Do present-day pollsters, in general, properly follow the underlying statistical principles in designing, sampling, and analysing their data? Do they go to remote corners of the country and cover sensitive booths for their surveys? Do they maintain the requirements for standard ‘3 percentage points margin of error’? Are their samples ‘random’? Do the samples represent the population by approximately maintaining the proportions across gender, age, income, religion, caste, and other important factors? How do they handle possible non-response? Sample size, sampling frame, method of sampling and estimation, and of course summary statistics across different margins of variability are not provided by most of the pollsters. Thus, it is almost impossible to comment on the quality of their predictions, from a statistical point of view.

The Literary Digest never attempted to predict the outcome of the U.S. elections after its 1936 debacle. That was never its main task. However, unless strong regulations enforce today’s pollsters to publish their methodology and summary statistics across different variables, poll predictions may not regain the trust of the people.

Atanu Biswas is Professor of Statistics at the Indian Statistical Institute, Kolkata