The UK election has a bit of everything for politicos and pundits – polls all over the place, poor punditry performance and the unveiling of a new rule on how to gauge polling errors.

As usual the majority of pundits got it wrong and the more confident they were in their assertions the more wrong they were. This is a phenomenon the blog has written a lot about, in particular drawing attention to alternatives which tend to work much better – Tetlock’s crowdsourcing and sifting prediction methodology and Nate Silver’s emphasis on Bayesian probability. It should be said that Tetlock and Silver do get things wrong – Tetlock on Brexit and Silver on Trump – but what is more significant is how often they get things right; show how you can make better predictions; and, how they demonstrate why others get things wrong so often. In Silver’s defence, with Trump he did always point to the possibility of Trump winning, and the blog wouldn’t have wanted to play Russian roulette on the basis of the probabilities Silver estimated.

But before Thursday’s election the pundits went through the full gamut of conventional wisdom – May landslide, Corbyn’s un-electability, the disaster of Labour’s manifesto and its left wing basis and the fluctuations in the polls. In the end YouGov, which was much ridiculed, got the poll just about right in its last poll and its penultimate one was spot on. They are now coming to Australia and how they fare in a compulsory voting situation will be instructive.

Perhaps there is one overwhelming truism to be confirmed by the campaign and the commentary – Galbraith’s comment that the conventional wisdom is always wrong is still valid decades after he said it. There are many reasons for this – group think, ignorance and poor analysis. Increasingly in a world of growing inequality the pundits are simply out of touch with the realities of ordinary people’s lives. Having been in Wales just recently, and visiting some former mining towns, the suggestion that the Tories might make big gains there  always seemed ridiculous to the blog. Similarly, back in the day when PR people were talking about the need for practitioners to get into boardrooms the blog always took the view that far more relevant to effective advice was travelling on public transport and standing in the outer.

But much of the pundits’ problems related to the changes in the polls over the UK campaign which appeared to most pundits to be counterintuitive. That there were all over the place was not the problem but rather that like the WWI generals the pundits were still trying to fight earlier wars – in this case the 2015 campaign when the polls under-estimated the Tory vote. There have been many sophisticated, and not so sophisticated, explanations for 2015. The ‘shy Tory voter’ was one explanation. This is a variation on the view that polls (wrongly as it turned out) under-estimated Marine Le Pen’s second round likely vote and the votes of Wilders and others. Assumptions about turn-out levels are a more legitimate explanation in non-compulsory voting but even these were undermined by erroneous assumptions about likely turn outs of young voters.

But Nate Silver has another explanation. This one combines a mixture of close reading of the evidence, behavioural psychology, the effects of group think and the conventional wisdom is always wrong thesis. In looking at the evidence that polling always underestimates Tory vote share he finds that it often does but that this isn’t an iron law. In discussing the polls and past voting realities  Silver (Note: five days before the election) said: “These experiences have given rise to what I’ve called the First Rule of Polling Errors, which is that polls almost always miss in the opposite direction of what pundits expect.

“The rule could potentially apply in the case of the UK election because there’s a gap between what the polls show and how the conventional wisdom assesses the race. Media coverage hasn’t been quite as much in lockstep as it was for Brexit or Trump, but the implication is usually that the ‘smart money’ is on May winning a more comfortable victory than polls show. (YouGov has been excoriated by some London media outlets for daring to publish its model showing a hung parliament, for instance.) And betting market prices imply a Conservative win by 9 or 10 percentage points rather than their 7-point lead in the polling average. Since pundits expect Conservatives to beat their polls, the First Rule of Polling Errors would therefore predict that Labour would beat their polls instead. That wouldn’t necessarily yield a hung parliament or a Labour-led government, but it would make things awfully close.

“So which idea — The Shy Tory Factor or the First Rule of Polling Errors — will prevail this time? Well, we’ll find out on June 8. But they have a complicated relationship with one another. Conservatives really do have a track record of beating their polls. But if everyone knows this and worries about it happening again — including pollsters — it could make the effect disappear or even reverse itself. My view, therefore…….. is that the error is about equally likely to come in either direction. There’s a significant chance of Conservatives beating their polls again, but Labour is roughly as likely to do so,” Silver said.

Incidentally, the 2016 Australian election campaign is another piece of evidence that the Silver rule has some validity. In that case the polls were right – it would be tight – but the punditry just wouldn’t believe them.