Why do political pundits, company directors, stock broking analysts, investors, policy people, governments and just about every one you can think of make dumb choices?

The easy answer is that their choices got skewed by emotion, or they didn’t have enough information or they just got unlucky and an otherwise rational decision went wrong. Thus, Fosters Chairman, David Crawford (in between lecturing investors, on behalf of the Australian Institute for Company Directors, on their audacity for questioning directors), came up with a number of explanations when pressed  about why Fosters came to cost investors billions of dollars in shareholder value through wine takeovers. Most of them were dependent on the Board having made rational choices but then being faced with changed circumstances.

Probably the most important book yet written on human decision-making, Thinking, fast and slow (Allen Lane, 2011) by Nobel Prize winner Daniel Kahneman puts not only the Fosters’ decision, but also the thousands of choices we all make every day, into a much more robust context.

Thus, mergers and acquisition are often driven by exactly the same modes of thinking that afflict individual investors and the man and the woman in the street. Just like the everyday decisions of ordinary people they are based on some deep-seated behavioural judgement and decision-making traits.

Kahneman, along with Amos Tversky, founded the behavioural economics field with papers such as: Judgement under Uncertainty: Heuristics, Biases; Prospect Theory: An Analysis of Decision under Risk and Choices Values and Frames. The first and third are published as appendices in the new book.

Kahneman and Tversky argued – and then proved – that systematic errors in people’s thinking are due to “the machinery of cognition rather than to the corruption of thought by emotion.” Their research is having a major influence on government policy in areas like social marketing, facilitating choices in things as diverse as retirement savings and organ donor programs and is slowly percolating into the awareness of some mainstream money market people.

In the new book Kahneman describes the mind as using two systems for choices. “System 1 operates automatically and quickly with little or no sense of voluntary control. System 2 allocates attention to the effortful mental activity that demands it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice and concentration.”

This doesn’t mean one system necessarily works better than the other – they both lead to mistakes. Much of the reason for those mistakes was described by David Hume in his principles of association: resemblance, contiguity of time and place and causality but Kahneman and Tversky provide empirical evidence of why Hume was right in this, as in so many other things. By the way – Kahneman critics (often among the neo-liberal school of rational choices economic theory) sometimes say the research is flawed because too many of the experiments involve students in paid research settings. The general criticism could be valid if, like the criticism of Freud’s samples as small and biased towards middle class Jewish Viennese men and women, the samples were not so large and diverse, the massive efforts of many researchers to replicate and extend the work, and the previous game theory research of von Neumann, Nash and others.

A very simple explanation of how this all works is provided early in the book with a question they put to a sample: An individual has been described by a neighbour as follows: “Steve is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.” Is Steve (pointed out to participants as randomly chosen) more likely to be a librarian or a farmer? Respondents ignored the statistics (farmers are 20 times more common than male librarians) and immediately focussed on the resemblance to a stereotype.

The book discusses framing (an old favourite of Come in Spinner), priming, how we jump to conclusions, anchors, association, cognitive laziness, intuitive predictions, the illusions of understand and validity, endowment effects and many other things which skew our choices.

There are also sections which explain what sort of influence media actually have, the impact of negative media coverage on media success, and why reforms (even ones supported by the majority of people) are so hard to implement. For instance, why the anti-mining tax got traction,  but David Crawford’s eminently-sensible sports funding recommendations didn’t, is explained by Kahneman’s models.

An optimist who wants a better world might ask – how can you avoid bad choices? Understanding how we come to make them is a good start. Hume is another although that can be a bit daunting. Not quite so daunting, but daunting nevertheless, is David Deutsch’s The Beginning of Infinity: Explanations that transform the world (Penguin, 2011) which talks about how we explain things – dividing our thought processes and researches into ‘good explanations’ and ‘bad explanations’. A couple of chapters -on Hilbert’s infinity hotel and the multiverse theory were totally incomprehensible to me even after patient explanations by an expert friend and  re-reading – and environmentalists will hate his technological optimism about climate change, but the rest of the book is a passionate explanation of everything the human mind is, and can be, capable of.

But for the cynical political and marketing types who wonder how they might prosper on the back of exploiting the flaws in the cognitive machine (if they are not already doing so) – read the books to learn how to do it better and hope that they take a while to filter through to your competitors and opponents.

Declaration of interest: The author’s former company undertook work for CUB.