Posted on by Rick Brenner

Unrealistic optimism: the planning fallacy and the n-person prisoner’s dilemma

Last updated on September 20th, 2018 at 03:51 pm

In a 1977 report, Daniel Kahneman and Amos Tversky identify one particular cognitive bias [Kahneman 2011], the planning fallacy, which afflicts planners [Kahneman 1977] [Kahneman 1979]. They discuss two types of information planners use. Singular information is specific to the case at hand; distributional information is drawn from similar past efforts. The planning fallacy is the tendency of planners to pay too little attention to distributional evidence and too much to singular evidence, even when the singular evidence is scanty or questionable. Failing to harvest lessons from the distributional evidence, which is inherently more diverse than singular evidence, the planners tend to underestimate cost and schedule. So for any given project, there’s an inherent tendency in human behavior to promise lower costs, faster delivery, and greater benefits than anyone can reasonably expect.

Aerial view of Hoover Dam, September 2017
Aerial view of Hoover Dam, September 2017. Under construction from 1931 to 1936, the dam was built for $48.8M ($639M in 2016 dollars) under a fixed-price contract. It was completed two years ahead of schedule. Apparently the planning fallacy doesn’t act inevitably. 112 men died in incidents associated with its construction. 42 more died of a condition diagnosed as pneumonia, but which is now thought to have been carbon monoxide poisoning due to poor ventilation in the dam’s diversion tunnels during construction. There’s little doubt that unrealistic optimism affects not only projections of budget and schedule, but also projections of risks, including deaths. Photo (cc) Mariordo (Mario Roberto Durán Ortiz), courtesy Wikimedia Commons.
But the problem is exacerbated by a dynamic described by Boehm et al. [Boehm 2016], who observe that because organizational resources are finite, project sponsors compete with each other for resources. They’re compelled by this competition to be unrealistically optimistic about their objectives, costs, and schedules. Although Boehm et al. call this mechanism the “Conspiracy of Optimism,” possibly facetiously, it isn’t actually a conspiracy. Rather, it’s a variant of the N-Person Prisoner’s Dilemma [Hamburger 1973].

Unrealistic optimism creates budget shortfalls and schedule pressures, both of which contribute to conditions favorable for creating non-strategic technical debt. And the kinds of technical debt produced by this mechanism, or any mechanism associated with schedule or budget pressure, tend to be subtle — they’re the types least likely to become evident in the short term. For example, technical debt that might make a particular kind of enhancement more difficult in the next project is more likely to appear than technical debt in the form of a copy of some code that should have been replaced by a utility routine. Copies of code are more easily discovered and more likely to be retired in the short term, if not in the current project. Awkward architecture might be more difficult to identify, and is therefore more likely to survive in the intermediate or long term.

In other words, the forms of technical debt most likely to be generated are those that are the most benign in the short term, and which are therefore more likely to escape notice. If noticed, they’re more likely to be forgotten unless carefully documented, an action that’s unlikely to be taken under conditions of schedule and budget pressure. In this way, the non-strategic technical debt created as a result of unrealistic optimism is more likely than most technical debt to eventually become legacy technical debt.

Policymakers can assist in addressing the consequences of unrealistic optimism by advocating for education about it. They can also advocate for changes in incentive structures and performance management systems to include organizational standards with respect to realism in promised benefits, costs, and schedules.

References

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Hamburger 1973] Henry Hamburger. “N-person Prisoner’s Dilemma,” Journal of Mathematical Sociology, 3, 27–48, 1973. doi:10.1080/0022250X.1973.9989822

Cited in:

[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.

Available: here; Retrieved: September 19, 2017

Cited in:

[Kahneman 1979] Daniel Kahneman and Amos Tversky, “Intuitive Prediction: Biases and Corrective Procedures,” Management Science 12, 313-327, 1979.

Cited in:

[Kahneman 2011] Daniel Kahneman. Thinking, Fast and Slow. New York: Macmillan, 2011.

Order from Amazon

Cited in:

Other posts in this thread

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Buttons
Hide Buttons