Getting forecasts right – a new project

Philip Tetlock (see the blog seriatum) has launched a new program – the Good Judgment Project – which is designed to help people make better forecasts about politics and other things. Tetlock is the famous psychologist whose 18 year research project, published as Expert Political Judgement in 2005, demonstrates that most forecasters get things wrong and that the more famous the pundit the more likely they are to be wrong. An indication of how significant it might be is the enthusiastic praise from Daniel Kahneman, of Thinking Fast and Slow fame who has said of Tetlock’s new project: “With some confidence, we can predict that another landmark of applied social science will soon be reached.” See www.edge.org The blog will be talking about all this, along with some other things, at a Melbourne Forum members’ discussion Why pundits get it wrong: Polls, probabilities and election predictions on October 8. Researcher John Armitage will be sharing the discussion which will be moderated by John Ridley of Clifton Consulting under Chatham House rules.  The Good Judgment research team is based in the University of Pennsylvania and the University of California Berkeley. The project is led by Tetlock, Barbara Mellers, an expert on judgment and decision-making, and Don Moore, an expert on overconfidence. Other team members are experts in psychology, economics, statistics, computer science and interface design. The projects website, https://www.goodjudgmentproject.com/ , says: “We are participating in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Activity). The ACE Program aims ‘to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts.’ The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public.” The fact that the project is being funded by US Intelligence may worry many but, as the blog reported in July this year, some of the best recent research on forecasting was carried out by a team at a Canadian defence agency. See http://noelturnbull.com/blog/more-predictive-perils/ The new Tetlock project is already getting quite a bit of publicity- see https://www.goodjudgmentproject.com/ , http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts and http://www.bbc.com/future/story/20140612-the-best-way-to-see-the-future Some of the project is based on the wisdom of crowds approach which Francis Galton talked about in the 19th century. Just as one should not discount the new Tetlock and the Canadian research because of the intelligence links one should not discount Galton’s views on forecasting and other subjects because of his unfortunate views on eugenics. What is particularly interesting is that the project is throwing up advice about how to make better forecasts around the acronym CHAMP which the FT (6 September 2014) summarised as: using comparisons as a starting point; looking at historical trends; averaging opinions (the Galton approach); using mathematical models; and, understanding your biases and avoiding clinging to old predictions in the face of new evidence. The last is the hardest for most people needless to say.