Pollsters must explain improvements before 2020 general election

Inquiry concludes UK polling companies should show how they will avoid repeat of 2015 general election inaccuracies

 Data editor

Britain’s pollsters will be required to explain what they have done to improve their techniques, after an inquiry into what went wrongin the run-up to last year’s general election concluded the surveys were systematically flawed.

An independent industry inquiry led by Prof Patrick Sturgis of the University of Southampton, commissioned in the aftermath of the polls’ failure to predict the scale of a Conservative victory, concluded that market research companies will have to show how they have improved by 2020.

The inquiry was commissioned by the British Polling Council (BPC), which represents the industry.

Its president, Prof John Curtice, said the body will look to implement the report’s recommendations on transparency but cannot compel members to use a certain type of methodology.

“At the end of the day, how people respond to the inquiry’s recommendations, how they decide to do their polls in future, is a decision for individual members,” he said.

“However, the BPC recognises that simply saying ‘well OK, it’s all fine, everybody go off and do their own thing’, may not be thought of as an adequate response to what happened in 2015.”

The BPC will issue a report before the 2020 general election detailing how companies have changed their sampling and weighting procedures since 2015, when most pollsters were predicting a hung parliament.

The inquiry noted that, in historical terms, the 2015 polls were some of the most inaccurate since election polling first began in the UK in 1945. On average, their final estimates put both the Tories and Labour on 34% – and these figures influenced the parties’ strategies, media coverage and, possibly, voter behaviour.

In the event, the Tories won a small majority with 38% of the vote compared with Labour’s 31%.

Sturgis said the media and others who rely on polling need to be “more realistic about the risks that polls can be wrong” and that polls in statistical terms are wrong “quite frequently”. Only significant shifts in party shares – a movement of 5% either way – were significant, he added.

The inquiry team also confirmed initial conclusions that the primary cause of the errors was unrepresentative sampling – those surveyed were not a representative selection of the nation’s voters.

The Sturgis inquiry concluded: “The methods the pollsters used to collect samples of voters systematically over-represented Labour supporters and under-represented Conservative supporters.”

Adjustments used historically to contend with the so-called “shy Tory” phenomenon – where Conservative voters were unwilling to identify themselves – also proved to be insufficient. “The statistical adjustment procedures applied to the raw data did not mitigate this basic problem to any notable degree.”

However, although initial steps towards fixing what went wrong will be taken immediately, the bulk of the changes recommended in Thursday’s report are unlikely to be in place before the 23 June EU referendum.

The inquiry also said that pollsters should review existing methods for determining turnout probabilities and how they allocate “don’t knows” – because existing procedures are often ad hoc and lack a coherent theoretical rationale.

Moreover, the report suggests including questions to determine whether respondents have already voted by post and recommends that new measures to obtain and weight samples are investigated and adopted.

It also recommends a series of changes to BPC rules in order to require polling companies to provide additional information and methodological detail around their polls, and more explicitly state the adjustments and variables they used to weight results of raw polls.

The report notes that the decrease in the Conservative polling lead in the final week of the campaign across all polls is consistent with herding – when pollsters tweak figures so that published results are all similar. It concludes that this convergence was unlikely to have been the result of deliberate collusion or malpractice.

(Source: http://www.theguardian.com/politics/2016/mar/31/pollsters-improvements-2020-general-election)


Leave a Reply