In a 1977 report, Daniel Kahneman and Amos Tversky identify one particular cognitive bias [Kahneman 2011], the planning fallacy, which afflicts planners [Kahneman 1977] [Kahneman 1979]. They discuss two types of evidence planners use. Singular evidence is specific to the case at hand. Distributional evidence is specific to similar past efforts. The planning fallacy is planners’ tendency to pay too little attention to distributional evidence and too much to singular evidence. They do this even when the singular evidence is scanty or questionable. Failing to harvest lessons from the distributional evidence, which is inherently more diverse than singular evidence, the planners tend to underestimate cost and schedule. So there’s a tendency to promise lower costs, faster delivery, and greater benefits than anyone can reasonably expect.
Enter the n-person prisoner’s dilemma
Boehm et al. [Boehm 2016] describe a dynamic that exacerbates the problem. They observe that because organizational resources are finite, project champions compete with each other for resources. This competition compels them to be unrealistically optimistic about their objectives, costs, and schedules. Although Boehm et al. call this mechanism the “Conspiracy of Optimism,” possibly facetiously, it isn’t actually a conspiracy. Rather, it’s a variant of the N-Person Prisoner’s Dilemma [Hamburger 1973].
A special property of pressure-induced debt
Hoover Dam, aerial view, September 2017. Under construction from 1931 to 1936, the cost of the dam was $48.8M ($639M in 2016 dollars) under a fixed-price contract. It was complete two years early. Apparently the planning fallacy doesn’t act inevitably. 112 men died in incidents associated with its construction. 42 more died of a condition diagnosed as pneumonia, but which experts now believe to have been carbon monoxide poisoning due to poor ventilation in the dam’s diversion tunnels during construction. There’s little doubt that unrealistic optimism affects more than budget and schedule projections. It also affects risk projections, including deaths. Photo (cc) Mariordo (Mario Roberto Durán Ortiz), courtesy Wikimedia Commons.Unrealistic optimism creates budget shortfalls and schedule pressures. In turn, they both contribute to conditions favorable for creating nonstrategic technical debt. And this mechanism, or any mechanism associated with schedule or budget pressure, tends to produce technical debt that’s subtle—it’s the type least likely to become evident in the short term. For example, technical debt that might make a particular enhancement more difficult in the next project is more likely to appear than technical debt that creates trouble in the current effort. Debt that creates trouble in the current effort is more likely to be retired in the short term, if not in the current effort. Awkward architecture might be more difficult to identify. It’s therefore more likely to survive in the intermediate or long term.
The bad news of schedule pressure
In other words, the technical debt most likely to be generated is that which is the most benign in the short term, and which is therefore more likely to escape notice. If noticed, it’s more likely to be forgotten unless carefully documented. And that action is unlikely under schedule and budget pressure. In this way, the nonstrategic technical debt created as a result of unrealistic optimism is more likely than most technical debt to eventually become legacy technical debt.
Last words
Policymakers can assist in addressing the consequences of unrealistic optimism by advocating for education about it. They can also advocate for changes in incentive structures and performance management systems. It’s good business to establish organizational standards with respect to realism in promised benefits, costs, and schedules.
References
[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
Alaska crude oil production 1990-2015. This chart [Yen 2015] displays Alaska crude oil produced and shipped through the Trans Alaska Pipeline System (TAPS) from 1990 to 2015. Production had dropped by 75% in that period, and the decline is projected to continue. In January 2018, in response to pressure from Alaskan government officials and the energy industry, the U.S. Congress passed legislation that opened the Arctic National Wildlife Refuge to oil exploration, despite the threat to ecological sustainability that exploration poses. If we regard TAPS as a feature of the U.S. energy production system, we can view its excess capacity as a source of feature bias. It creates pressure on decision makers to add features to the U.S. energy system. Alternatively, they could act to enhance the sustainability of Alaskan and global environmental systems [Wight 2017].
<
Enterprise decision makers affected by feature bias tend to harbor distorted views of the importance of new capability development compared to technical debt management. This tendency is likely due to the customer’s relative sensitivity to features, and relative lack of awareness of sustainability. Whatever the cause, customers tend to be more attracted to features than they are to indicators of sound technical debt management and other product sustainability practices. This tendency puts decision makers at risk of feature bias: unbalanced concern for capability vs. sustainability.
h4>Accounting changes can help
Changes in cost accounting could mitigate feature bias effects by projecting more accurately total MICs based on historical data and sound estimation. I explore possible accounting changes later in this post, and in future posts; meanwhile, let’s explore the causes and consequences of the distorted perspective I’m calling feature bias.
Causes and consequences of feature bias
For products or services offered outside the enterprise, the sales and marketing functions of the enterprise represent the voice of the customer [Gaskin 1991]. But customers are generally unaware of product or service attributes that determine maintainability, extensibility, or cybersecurity. These factors, the sustainability factors, affect the MICs for technical debt. But customers are acutely aware of capabilities—or missing or defective capabilities. Customer comments and requests are therefore unbalanced in favor of capability over sustainability. The sales and marketing functions tend to accurately transmit this unbalanced perspective to decision makers and technologists.
An analogous mechanism prevails with respect to infrastructure and its internal customers. Internal customers tend to be more concerned with capabilities than they are with sustainability of the processes and systems that deliver those capabilities. Thus, pressure from internal customers tends to emphasize capability at the expense of sustainability. The result of this imbalance is pressure to allocate excessive resources to capability enhancement, compared to activities that improve sustainability. And therefore controlling or reducing technical debt and its MICs gets less attention.
Nor is this the only consequence of feature bias. It provides unrelenting pressure for increasing numbers of features, despite the threats to architectural coherence and overall usability that such “featuritis” or “featurism” present. Featurism leads, ultimately, to feature bloat, and to difficulties for users, who can’t find what they need among the clutter of features that are often too numerous to document. For example, in Microsoft Word, many users are unaware that Shift+F5 moves the insertion point and cursor to the point in the active document that was last edited, even if the document has just been freshly loaded into Word. Useful, but obscure.
Feature bias bias
Feature bias, it must be noted, is subject to biases itself. The existing array of features appeals to a certain subset of all potential customers. Because it is that subset that’s most likely to request repair of existing features. And they’re also the most likely to suggest additional features. The pressure for features tends to be biased in favor of the needs of the most vociferous users. That is, there is pressure to evolve to better meet the needs of existing users. That pressure can force to lower priority any efforts toward meeting the needs of other stakeholders or potential stakeholders. These other stakeholders might be even more important to the enterprise than are the existing users. This bias in feature bias presents another risk that can affect decision makers.
Organizations can take steps to mitigate the risks of feature bias. An example of such a measure might be using focus groups to study how educating customers in sustainability issues affects their perspectives relative to feature bias. Educating decision makers about feature bias can also reduce this risk.
At the enterprise scale, awareness of feature bias would be helpful. But awareness alone is unlikely to counter its detrimental effects. These effects include underfunding technical debt management efforts. Eliminating the source of feature bias is extraordinarily difficult, because customers and potential customers aren’t subject to enterprise policy. Feature bias and feature bias bias are therefore givens. To mitigate the effects of feature bias, we must adopt policies that compel decision makers to consider the need to deal with technical debt.
A possible corrective action
One possible corrective action might be improving accounting practices for MICs, based on historical data. For example, there’s a high probability that any project might produce new technical debt. It might be prudent to fund the retirement of that debt in the form of reserves when we fund projects. And if we know that a project has encountered some newly recognized form of technical debt, it might be prudent to reserve resources to retire that debt as soon as possible. Ideas such as these can rationalize resource allocations with respect to technical debt.
These two examples illustrate what’s necessary if we want to mitigate the effects of feature bias. They also illustrate just how difficult such a task will be.
References
[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
When we try to understand the behavior of others, we often make a particularly human mistake. We tend to attribute too much to character and disposition and too little to situation and context. This mistake is so common that it has a name: The Fundamental Attribution Error (FAE) (See “The Fundamental Attribution Error” at my other blog). And although little experimental data is available regarding its effects on technical debt, we can plausibly argue that its effects are significant—and unwelcome.
Arapaho moccasins ca. 1880-1910. An American Indian proverb advises, “Don’t judge any man until you have walked two moons in his moccasins.” From the perspective of the FAE, the proverb is a way of mitigating FAE risks. Photo of Arapaho moccasins, ca. 1880-1910 on exhibit at the Bata Shoe Museum, in Toronto, Canada. Photo by Daderot, courtesy WikimediaThe FAE contributes to technical debt in at least two ways. First, it distorts assessments by non-engineers of the motivations of engineers as they warn of future difficulties from technical debt. Second, it distorts assessments by engineers of the motivations of non-engineers as they oppose allocation of resources to technical debt retirement. They oppose these allocations in order to conserve resources for their own efforts or to accelerate efforts in which they have more immediate interest. The two effects are symmetrical in the large, though not in detail.
Below is a description of the effects of the FAE on engineers and non-engineers. Some of the non-engineers are the internal customers of the engineers. I examine the effects of the FAE that arise from three different claims of the parties to the exchange.
Many engineers or their managers hold this position.
Engineers notice incidents in which some of the work they must perform on an asset would be much easier or even unnecessary were it not for the technical debt that the asset carries. They sense the burden of the extra effort because they know how much easier and faster the work would be if they could retire the debt.
The internal customers of engineers don’t see these circumstances as clearly as engineers and their managers do. Consequently, they tend to discount engineers’ claims of depressed productivity. Some experience engineers’ complaints, requests, and warnings as whining, self-serving nest feathering, or worse. They tend to attribute engineers’ complaints to faults in the character or “work ethic” of engineers.
Claim: Instead of retiring the technical debt now, just document it for later
This is a suggestion senior managers or the engineers’ internal customers often put forth.
The internal customers of engineers have pressing needs for immediate engineering results. They see new products or repairs to existing products as a means of achieving the objectives the enterprise sets. Focusing limited engineering resources on technical debt retirement conflicts with producing results that would help internal customers of engineers meet these more immediate objectives. As a compromise, non-engineers propose that engineers document instances of technical debt as they find them, so that they can be addressed more efficiently after engineers meet the immediate needs of internal customers.
Engineers discount the validity of this approach for three reasons. First, they don’t experience the pressures their internal customers do. To engineers, their customers’ reports of more pressing needs seem to be merely excuses to get what they want when they want it. Second, the proposed documentation work doesn’t advance the engineers’ customer’s current project toward its objectives. Instead, it actually delays the current project, in ways invisible to non-engineers. These delays induce increases in schedule pressure, and therefore technical debt. The technical debt occurs because the customer of the current project rarely cares enough about the technical debt documentation effort to allow for the extra time it takes. Finally, because many assets evolve continuously, such documentation has a short shelf life. And that limits its value in ways non-engineers might not appreciate.
In these ways, the FAE both creates the documentation suggestion, and limits the ability of engineers to appreciate its motivation. But it also limits the ability of non-engineers to appreciate how limited is the value of the documentation.
Claim: Addressing technical debt is important, but not urgent
Senior managers or the engineers’ internal customers are most common among adherents of this belief.
When the engineering organization presents a business case for investing in addressing technical debt, they aren’t alone. Other functions in the enterprise also make business cases of their own. Too often, these cases are evaluated against each other. Investment in one entails reduced investment in another. But the benefits of technical debt retirement tend to become most visible to non-engineers much later than do the benefits of some other proposals. Sometimes the benefits of technical debt retirement are wholly invisible to non-engineers. For these reasons, technical debt retirement projects tend perhaps more often than most to be deferred at best, or, worse, rejected.
The FAE is in part responsible for the perception of non-engineers that the benefits of technical debt remediation arrive in the distant future. Engineers notice the benefits relatively immediately, because they interact with the rehabilitated assets on a daily basis. Since non-engineers don’t have these experiences, they notice the benefits only upon delivery of the results of engineering work. This mismatch of the timescales of perceptions of engineers and non-engineers prevents non-engineers from perceiving what is in daily evidence to engineers.
Last words
Both engineers and non-engineers are subject to deadlines and resource limitations beyond their control. Their ability to appreciate the challenges their counterparts face is the key to effective collaboration. Too often, neither part feels that it has the time or resources to accommodate the needs of the other.
References
[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Separating responsibility for maintenance and acquisition or development of technical assets can lead to uncontrolled growth of technical debt. The problem arises when we measure without regard for technical debt the performance of the business acquisition function or the performance of the development organization. In that circumstance, technical debt is likely to expand unchecked. To limit such expansion, policymakers must devise performance measures that hold these organizations accountable for technical debt resulting from their actions.
Software systems
Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010 [NOAA 2013]. The storms were so severe that the floodwaters overtopped the gauge’s measurable range. Moreover, the National Weather Service (NWS) lacked a database of measurable ranges for flood gauges. Quoting the NWS report: “A lesson learned here was that maximum recordable river levels should be known by NWS staff, not only so staff aren’t surprised when this type of issue arises, but also to notify USGS personnel so that they can install a temporary gage and remove or elevate threatened equipment.” Technical debt, if ever I’ve seen it.For systems consisting solely of software, separating responsibility for maintenance and acquisition or system development is risky. It enables the acquiring organization to act with little regard for the consequences of its decisions vis-à-vis maintenance matters [Boehm 2016]. This is unfortunate—it increases the rate of accumulation of new technical debt. And it increases the lifetime of legacy technical debt. This happens more frequently when the acquiring organization doesn’t suffer the MICs associated with the technical debt.
For example, a focus on performance of the organization that’s responsible for acquisition biases them in favor of attending to the direct and immediate costs of the acquisition. They are likely to have little or no regard for ongoing maintenance issues. The maintenance organization must then deal with whatever the acquired system contains (or lacks).
An analogous mechanism operates for organizations that develop, market, and maintain products or services. When there are software elements in their respective infrastructures, separation of the development function from the maintenance function enables the development function to act independently of the maintenance consequences of its decisions.
Systems that include hardware
But the separation-of-responsibilities mechanism that leads to uncontrolled technical debt isn’t restricted to software. Any technological asset that has ongoing maintenance needs (and most of them do) can potentially present this problem.
For example, in the United States, and many other countries, two streams of resources support publicly-owned infrastructure [Blair 2017]. The funding stream covers construction, operations and maintenance, and repairs. Its usual sources are taxes, tolls, licenses, other user fees, sale of ad space, and so on. The financing stream covers up-front construction costs, to bridge the period from conception through construction, until the funding stream begins delivering resources. The financing stream usually comes from bond sales.
Although legislatures, or agencies they establish, control both streams of resources, the effects of the streams differ fundamentally. The financing stream is dominant during construction and the early stages of the asset’s lifecycle. The funding stream is dominant after that—when maintenance and operations are most important. Legislators and agencies are generally reluctant to supply funding because of the impact on taxpayers and users. Legislators and agencies find financing much more palatable. For this reason, among others, funds for U.S. infrastructure maintenance are generally insufficient, and technical debt gradually accumulates.
So it is with technological assets in organizations. For accounting purposes, capital expenses are treated differently from operational expenses. The result is that operational expenses can have a more significant impact on current financial results than capital expenses do. This leads organizations to underfund operations and maintenance, which contributes to technical debt accumulation.
Last words
Control of new technical debt accumulation and enhancement of technical debt retirement rates is possible only if we can somehow hold accountable the acquisition or development organizations for the MICs that result from their actions. Securitization of the debt incurred, as I’ll address in a forthcoming post, is one possible means of imposing this accountability. But reserves are also required, because some of the debt incurred might not be known at the time the asset is acquired or created.
Separating responsibility for maintenance and acquisition or system development is actually a form of stovepiping. See “Stovepiping can lead to technical debt” for more on stovepiping.
References
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Many an enterprise culture includes, perhaps tacitly, an unrealistic definition of done for projects. Some enterprise cultures assume definitions of done that fail to adequately acknowledge attributes related to sustainability. For such cultures, technical debt expands inexorably. In most organizations, the definition of done includes meeting the attributes that most internal customers understand and care about. These attributes might not include sustainability [Guo 2011]. Indeed, even among technologists, the definition of done might not enjoy precise consensus [Wake 2002].
Why retiring technical debt isn’t included in “done”
The 2009 Ford Focus SES coupe (North America) engine bay. Typical owners can no longer learn how to maintain their own vehicles. Engines have become so complex that even experienced mechanics must train to maintain the engines they work on. Since these vehicles are available for sale to consumers, clearly their manufacturers regard their designs as “done.” But is technical debt a factor in the growing complexity of modern engines? It’s probably present in their software, and it would be most surprising if we found no technical debt in the mechanical design. Photo (cc) Porsche997SBS courtesy Wikimedia.Internal customers understand less well the attributes of deliverables related to sustainability. It’s therefore perhaps unsurprising that sustainability might not receive the attention it needs. Applying scarce resources to enhance attributes the customer doesn’t understand, and cares about less, will always be difficult.
To gain control of technical debt, we must redefine done to include addressing sustainability of deliverables. Although there may be many ways to accomplish this, none will be easy. Resolution will necessarily involve educating internal customers to understand enough about sustainability to enable them to justify paying for it.
Redefining “done”
The typical definition of done for most projects ensures only that the deliverables meet the requirements. Because requirements usually omit reference to retiring newly incurred nonstrategic technical debt, we often declare projects complete with incremental technical debt still in place. A similar problem prevails with respect to legacy technical debt.
A more insidious form of this problem is intentional shifting of the definition of done. This can happen when the organization has adopted a reasonable definition of done that allows for addressing sustainability. But under severe time pressure, the definition is “temporarily” amended to allow the team to declare the effort complete, even though sustainability issues remain unaddressed.
For most projects, three conditions conspire to create steadily increasing levels of nonstrategic technical debt. First, for most tasks, the definition of done is that the deliverables meet the project objectives, or at least, they meet them well enough. Second, typical project objectives don’t restrict levels of newly incurred nonstrategic technical debt, nor do they demand retirement of incidentally discovered legacy technical debt. Third, budget authority usually terminates upon acceptance of delivery. These three conditions, taken together, restrain engineering teams from immediately retiring any debt they incur. Nor can they retire—or document or report—any legacy technical debt they encounter while fulfilling other requirements.
For example, for one kind of incremental technical debt—what Fowler calls [Fowler 2009] Inadvertent/Prudent (“Now we know how we should have done it”)—the realization that we’ve incurred new debt often occurs after the task is “done.” If budget authority has terminated, there are no resources available—financial or human—to retire that form of technical debt.
Last words
Unless team members document the technical debt they create or encounter, there is risk of lost knowledge. After team members move on to their next assignments the enterprise is likely to lose track of the location and nature of that debt. A more realistic definition of done would enable the team to continue working post-delivery to retire or document any newly incurred nonstrategic technical debt. They could also note any incidentally encountered legacy technical debt. Moreover, teams most likely leave in place any strategic technical debt—technical debt incurred intentionally for strategic reasons. Although the enterprise must eventually address such debt as well, the widespread definition of done doesn’t address it.
Policymakers are well positioned to advocate for the culture transformation needed to redefine done.
References
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Stovepiping can lead to technical debt. Actual stovepipes are the tubes that vent exhaust from stoves. These tubes serve as a metaphor for the flow of information in “stovepiped” organizations. In stovepiped organizations, information can flow predominantly (or only) up or down along the parallel chains of command. But information can flow only rarely (or never) from a point in one chain of command across to some other chain of command [Waters 2010]. The stovepipe metaphor is imperfect, in the sense that in actual stovepipes, smoke and fumes rarely flow downwards. By contrast, in organizations, some information does flow down the chains of command. But the metaphor does clarify the problem of limited flow of information. Transferring whatever the organization learns in one metaphorical stovepipe into other metaphorical stovepipes is difficult.
Two forms of stovepiping
A wood-burning stove in a farm museum in Lower Bavaria (German: Niederbayern). Lower Bavaria is one of the seven administrative regions of Bavaria, Germany. The stovepipe, which is the black tube running upwards from the stove, channels smoke and fumes out of the kitchen into the chimney.Stovepiping can occur in both organizational structures and in engineered systems. These two forms of stovepiping are intimately related, and both can lead to uncontrolled formation of new technical debt, or increased persistence of existing technical debt.
In organizational structures, stovepiping occurs when elements of different organizational units with similar capabilities act relatively independently. An example is the dispersal of some elements of the IT function out into IT’s customers. When independent organizations have similar technical needs, they’re at risk of generating new technical debt. The debt they generate results from independently implementing technological capabilities that duplicate each other.
Stovepiping occurs in engineering, for example, when the organization manages and maintains independently two distinct technological assets [McGovern 2003]. In that situation, distinct engineering efforts working on those assets might happen to solve the same problem, possibly in two different ways. Then each party might be either ignorant or possibly disparaging, of the other’s efforts.
How stovepiping relates to technical debt
In whichever way duplication of technological capability comes about, it can increase levels of technical debt. Alternatively, it can increase persistence of existing technical debt. These effects happen because the organization might need to execute future maintenance or enhancement efforts multiple times—once for each instance of the technical artifact. That exposes the organization to additional cost, additional load on its staff, and additional risk of creating defects and incurring liability. Compare this situation to one in which all units that need a particular asset share it. Duplication is expensive.
The problem is actually even more worrisome. First, suppose there exists a defect in one version of a technological artifact. The people who are aware of the defect might not realize that another version of the artifact exists. If that second version also has an analogous defect, its defect might go unrecognized for some time, with all the usual attendant negative consequences. Second, suppose there is a necessary extension of the artifact’s capabilities. The maintainers of one version might recognize the need for the extension and implement it. Meanwhile, the maintainers of other versions might not recognize the need for the extension. They might not take action until something bad happens or a possibly urgent need arises. It’s easy to conjure other unfavorable—and costly—scenarios.
Stovepiping in technological systems
In engineering more generally, stovepiping can occur internally in systems, even though only one business unit is involved, and even though the stovepiped artifacts serve purposes invisible to the world outside the system. This can occur whenever there is weak communication between the teams designing or maintaining the portions of the system that host the similar artifacts. For readers familiar with the Apollo XIII incident, the incompatibility of the two carbon dioxide scrubbers in the command module and the lunar excursion module serves as an example of the risks of technical stovepiping.
When distinct business units or functions operate their own engineering or IT organizations, there’s an elevated probability of duplicating technological assets. The same effect can occur when they depend on a shared engineering function but require similar work. This happens when the organizational structure or the timing of the work encourages separate engineering efforts. Engineering or IT functions operated separately under the control of distinct business units or functions can clearly produce duplicated capabilities. However, duplication can also occur when the engineering function is shared across distinct business units or functions. This happens when the actual people and teams performing the work differ for different efforts. And it can happen too when communication is weak between those teams, whether or not the efforts are conducted contemporaneously.
Last words
Because identifying these forms of technical debt after they appear is notoriously difficult, preventing their formation is preferable. Prevention is possible if the enterprise establishes mechanisms that facilitate consultation and sharing among elements of different, separately operated technology development or maintenance functions. In other words, the organization must “break” the stovepipes—no mean feat, politically speaking.
Another challenge, of course, is providing resources for such sharing mechanisms, because preventing technical debt is rarely recognized as a value generator. If it were so recognized, the resources would likely appear. Changes in cost accounting might make such recognition more likely.
References
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Defining technical debt explicitly enough for project objectives is difficult. Confronted with this difficulty, some internal customers of technologists adopt a zero-tolerance approach to technical debt. But rarely do they define technical debt explicitly. Post-delivery, when customers discover technical debt, they hold technologists responsible. They do so even in cases when no one could have predicted that a specific artifact would eventually become technical debt. This establishes an adversarial culture in which technologists contend with their internal customers. To defend themselves, technologists sometimes adopt a work-to-rule approach to their work.
How adversarial cultures develop
Trouble at the airport. When airline pilots engage in work-to-rule actions, the immediate result can be large numbers of delayed or cancelled flights. The longer-term result might be beneficial to pilots, the airline, and the public, but only if labor peace can be restored, and the damage to the flying public can be overcome. So it is with work-to-rule deliveries as a way of dealing with zero-tolerance technical debt policies. The organization must overcome the adversarial culture that results from indiscriminate attempts to control technical debt. Technologists do gain some measure of protection by working to rule, but the longer-term benefit of the organization’s learning to manage technical debt arrives only if the adversarial culture can be overcome. Image (cc) Hotelstvedi courtesy Wikimedia.And that’s when the trouble begins.
Within this adversarial dynamic, technologists try to protect themselves against future recriminations by “working to rule.” They perform only work that the internal customer specifies. If they find something else that needs work, they perform that work only if they successfully obtain the customer’s approval. Some customers continue to adhere to zero-tolerance policies with respect to technical debt. But engineers cannot meet such non-specific requirements. Because technologists are “working to rule,” they use the ambiguity of the zero-tolerance requirement to assert that they performed all explicitly specified work. This level of performance is analogous to the work-to-rule actions of some employees involved in labor disputes with their employers. In these actions, employees are literally in compliance with the requirements of the employer, but only literally [LIBCom 2006].
Preventing formation of an adversarial culture
Requiring that deliverables be free of technical debt contributes to formation of an adversarial culture. In such cultures the adversaries are the technologists and their internal customers. Shedding that adversarial culture can be difficult once it sets in. Compelling employees, vendors, or contractors to deliver work that’s free of all technical debt is therefore unlikely to succeed. Whether employees or contractors perform the work in-house, or the work is outsourced, the results can be problematic. In the context of an adversarial culture, deliverables that meet the minimum possible interpretation of the stated objectives are almost certain to carry unacceptable levels of technical debt. What can we do to prevent this?
To avoid creating an adversarial culture, we can specify in project objectives the total removal of some kinds of technical debt. To ensure steady progress, develop a statement of objectives that includes complete retirement of at least one well-defined class of technical debt. Emphasize debt classes that have the highest anticipated MICs in the near term. Address other well-defined classes of technical debt on a best-effort basis.
We must accept as the “cost of doing business,” any other forms of technical debt that remain at the end of a given project. We must also accept that some artifacts might later become technical debt, even though they aren’t at present. We’ll get to them, but unfortunately, not this time.
References
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
A dog receiving a reward. Many performance management systems implement a model that assumes that the correct configuration of incentives and disincentives will produce the desired result. This theory is questionable.
Few performance management systems provide guidance with respect to behaviors relating to technical debt. One reason, perhaps, is that technical debt isn’t widely understood. Or perhaps only engineers and their managers regard technical debt as a concern. Still, to gain control of technical debt organizations must ensure that performance standards are clear. They must clearly state expectations with respect to behaviors that could affect technical debt. If changes are necessary, policymakers can be effective advocates—provided that they understand what the appropriate role for performance management is in controlling technical debt. This post should be helpful.
A fundamental premise of many performance management systems is that incentives can encourage desirable behavior. Likewise, disincentives can discourage undesirable behavior. Unfortunately, serious questions have arisen about the effectiveness of these behavioral control mechanisms in general [Kohn 1999]. Employees find ways to harvest incentives without exhibiting the desired behavior. Similarly, they find ways to circumvent disincentives while continuing to engage in undesired behavior.
Behavioral control for technical debt is problematic
Moreover, specifically for technical debt management, behavioral control is especially problematic. Troubles arise because some of the desirable behaviors are inherently immeasurable. For example, the design of an incentive structure to encourage legacy technical debt retirement is debatable. The technical difficulties involved relate to the problems of defining legacy technical debt.
Managing performance vis-à-vis technical debt, therefore, presents a problem of the kind Austin calls partially supervised [Austin 1996]. Supervising engineers whose work can affect technical debt can only be partial. The issue is that measuring technical debt is only partially practical given the state of the art. Austin shows how partial supervision frequently leads to dysfunctional performance management. But the problem is especially vexing for managing technical debt. For example, engineers’ work can sometimes incur technical debt that remains unrecognized for months or years after the work is completed. To fully supervise such work would require inventing retroactive incentives and disincentives. This class of motivators doesn’t exist, but even if they did, they’re of questionable legality in most jurisdictions.
The doctrine of commander’s intent
Although incentives and disincentives cannot serve to manage performance relative to technical debt, a very effective model is available. Enterprise leaders could communicate their intentions relative to technical debt, and empower the people of the organization to take steps to reduce debt. In the United States military, and others as well, a doctrine that implements this approach is called commander’s intent [Mattis 2008][US Army 2010].
Gen. Mattis offers five principles that guide what the military calls “effect-based operations.” For technical debt management, the effect we seek is rational control of the technical debt portfolio. Here are his five principles, transformed to the field of technical debt.
Technology development, maintenance, and cyberdefense in the future will require a balance of conventional and unconventional approaches.
Technology evolves rapidly, and we must be willing to adapt our methods.
Technologies are dynamic, with an infinite number of variables; therefore, it isn’t scientifically possible to accurately predict the level of technical debt that will result from any given effort. To suggest otherwise runs contrary to historical experience and the nature of modern technological assets.
We are in error when we think that what works (or does not work) in efforts involving one technology in one enterprise will be universally applicable to all technologies in all enterprises.
Finally, to paraphrase General Sherman, “Every attempt to make technical debt management easy and safe will result in humiliation and disaster.”
Limitations of the doctrine of commander’s intent
Most organizations rely on supervisors to communicate the analog of commander’s intent to their subordinates. Currently, it’s fair to say that few supervisors outside the technology-oriented elements of the enterprise communicate much about technical debt to their subordinates.
That situation might explain why most performance management systems encourage behaviors that unwittingly expand the body of technical debt, especially for non-technologist performers. There are situations in which the widely applauded actions of the outstanding performer actually incur technical debt strategically and responsibly. Technical debt so incurred is what McConnell calls Type II [McConnell 2008] and what Fowler calls Deliberate and Prudent [Fowler 2009]. But most performance management systems, especially for non-technologists, say nothing about technical debt. They risk encouraging behaviors that indirectly exacerbate the problems associated with technical debt.
An illustrative story
Distinguishing responsible and irresponsible behaviors is possible only if understanding of the nature of technical debt is widespread in the organization, even beyond the technologists. Here’s an example:
It was ambitious. It was what advocates called a “stretch goal.” But the VP of Marketing approved the plan to release the new app by the end of the fiscal quarter. After a month of meetings, and much jawboning, the CTO agreed. The VP of New Product had serious objections, but the executive team set them aside. Engineers and testers were able to meet the date, but they had to incur significant technical debt. When they asked for resources to retire that debt after the release, the VP of Marketing opposed the request. She needed additional resources for the promotional campaign due to our late entry into the market.
Stories like this illustrate scenarios in which technical debt considerations are subordinate to “business priorities.” These latter include market timing, market development, and revenue generation. Standards for setting priorities closely parallel the standards defined in the performance management system. Indeed, performance management should support enterprise goals. In the scenario above, the organization might meet the immediate goal of a successful release. But it does so by incurring technical debt, thereby imperiling the next release. This scenario illustrates why changing the performance management system might achieve a better balance between immediate goals responsible technical debt management.
Last words
Since anyone in the enterprise can take actions or make decisions that lead to incurring new technical debt, or cause existing technical debt to remain in place, organizations need performance standards that guide employees with respect to technical debt. To provide guidance for distinguishing responsible behavior from irresponsible behavior, performance management systems must acknowledge the potential of any employee to affect technical debt. Performance management systems must be reviewed with respect to alignment with technical debt policy. They might then support a mechanism analogous to Gen. Mattis’s vision of commander’s intent.
References
[Austin 1996] Robert D. Austin. Measuring and Managing Performance in Organizations. New York: Dorset House, 1996. ISBN:0-932633-36-6
Contains an extensive discussion of the consequences of partial supervision of performance. Since technical debt can only be partially supervised, the concept is relevant to understanding the effects of performance management systems on technical debt. Order from Amazon
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Kohn 1999] Alfie Kohn. Punished by rewards: The trouble with gold stars, incentive plans, A's, praise, and other bribes. Boston: Houghton Mifflin Harcourt, 1999. ISBN:0-395-71090-1
[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.
[US Army 2010] U.S. Army (2010) Field Manual 5.0 – The Operations Process U.S. Department of the Army.
Describes the concept, value, and importance of the doctrine of commander’s intent. See the index for “commander’s intent,” and especially paragraphs 2-90 and 2-91. Available: here; Retrieved: Dec. 22, 2019.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Enterprises that grow by acquisition inevitably acquire the technological assets of the organizations they acquire. And most enterprises acquire technological assets by other means as well. The contract negotiation teams need technical knowledge to accurately project the effects of these acquisitions on enterprise technical debt. But as total enterprise technical debt grows, the capacity of enterprise technologists to support new contract negotiations declines. That leads to a self-sustaining cycle of technical knowledge deficits. Hiring more technologists is the way to break the cycle. And policymakers and strategic planners are likely the most effective possible advocates for adopting that solution.
Negotiating contracts with significant technological content relating to acquiring assets can require special expertise. In the asset acquisition process, assessing the technical debt status of the assets is often necessary. Such contracts include contracts with vendors that provide outsourcing services or subcontracting work, or contracts with organizations to be acquired. Moreover, the technical debt in question includes more than just the debt the asset carries as it stands in its pre-acquisition context. It also includes the debt that the asset will carry after insertion into the asset portfolio of the acquiring enterprise.
These two debts—pre-acquisition and post-acquisition—can differ. The main source of difference is that the interfaces, standards, and approaches of the acquiring organization likely differ from those prevailing within the vendor organization or the acquired organization. Making a valid assessment of the post-acquisition levels of technical debt requires knowledge of the interfaces, standards, and approaches of both organizations.
A vicious cycle
The enterprise negotiation team therefore requires the services of technologists. These technologists must be familiar with the modifications the acquired assets will need after the acquisition. But problems can arise. Specifically, resource contention occurs when technical debt levels in the enterprise reach a level so serious that debt service requires the full attention of all available technologists. If that happens negotiating contracts usually gets less attention. Contract negotiation teams then experience a deficit of knowledge concerning the consequences of acquiring assets laden with technical debt. That leads to increasing levels of nonstrategic technical debt. And that condition has the potential to exacerbate the technical knowledge deficit for the negotiating teams.
This situation is an example of a dynamic widely known as a vicious cycle. After technical debt has reached a critical level, there are really only two tactics that can break the cycle—get more engineers, or suspend some work.
References
[Austin 1996] Robert D. Austin. Measuring and Managing Performance in Organizations. New York: Dorset House, 1996. ISBN:0-932633-36-6
Contains an extensive discussion of the consequences of partial supervision of performance. Since technical debt can only be partially supervised, the concept is relevant to understanding the effects of performance management systems on technical debt. Order from Amazon
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Kohn 1999] Alfie Kohn. Punished by rewards: The trouble with gold stars, incentive plans, A's, praise, and other bribes. Boston: Houghton Mifflin Harcourt, 1999. ISBN:0-395-71090-1
[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.
[US Army 2010] U.S. Army (2010) Field Manual 5.0 – The Operations Process U.S. Department of the Army.
Describes the concept, value, and importance of the doctrine of commander’s intent. See the index for “commander’s intent,” and especially paragraphs 2-90 and 2-91. Available: here; Retrieved: Dec. 22, 2019.
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
Cropped detail from Charles Robert Darwin, a painting by John Collier (1850-1934). The painting was given to the National Portrait Gallery, London, in 1896. Darwin writes, in The Descent of Man (1871): “… ignorance more frequently begets confidence than does knowledge …” which is the essence of the Dunning-Kruger effect. Image courtesy WikiQuote.
The Dunning-Kruger effect [Kruger 1999] can lead to formation or persistence of technical debt in two ways. First, it can cause technologists or their managers to overestimate their ability to maintain the resource focus needed for retiring technical debt in a timely fashion. Second, it can cause senior managers to be reluctant to accede to resource requests of technologists and their managers in support of technical debt management programs.
Kruger and Dunning conducted experiments that yielded results consistent with the following four principles (paraphrasing):
Incompetent individuals, compared to their more competent peers, tend to dramatically overestimate their own ability and performance
Incompetent individuals, compared to their more competent peers, tend to be less able to gain insight into their own true levels of performance
Incompetent individuals can gain insight about their shortcomings, but, paradoxically, this comes about by gaining competence
Incompetent individuals, compared to their more competent peers, are less able to recognize competence when they see it
The first three principles lead to distorted assessments of one’s own capabilities. The fourth principle leads to distorted assessments of the capabilities of others.
How the Dunning-Kruger Effect affects teams
As an example of distorted self-assessment, consider a team or its managers who must undertake retirement of some types of technical debt. And suppose that they must do this in the course of enhancing or repairing an asset. Such a task plan seems at first to offer efficiencies. The engineers can readily make both kinds of changes at one go. Metaphorically, if we must go to the store for milk, we can also pick up bread while we’re there, rather than making two trips.
However, modifying an existing complex technological asset is unlike shopping for bread and milk. The two kinds of modifications—debt retirement and asset enhancement or repair—might seem at first to be separable. Often they are. But if they aren’t separable, and we undertake the two tasks together, testing and debugging can become extremely complicated. The complications arise because of interactions between defects in the two kinds of modifications. Under some circumstances, an experienced team and its managers might be more likely to anticipate these difficulties. An inexperienced team and its managers might be more likely to underestimate the difficulties, as a consequence of the Dunning-Kruger effect. Budget and schedule overruns are possible consequences of underestimating the complexity of the problem.
How the Dunning-Kruger Effect affects decision makers
As an example of the fourth principle above, the Dunning-Kruger effect can cause some decision makers to discount the warnings and resource requests of engineers and their managers. Decision makers who are unsophisticated in matters related to technical debt must nevertheless assess the validity of the resource requests. In making these assessments, these decision makers may be at a disadvantage for a number of reasons. Examples:
Decision makers might hold mistaken beliefs about technical debt. For example, many believe that the main causes of technical debt are poor decisions by engineering managers. And others believe that technical debt is the result of slovenly work habits of engineers. Those who hold such beliefs might be reluctant to allocate yet more resources to engineers to address technical debt.
If the advocates for resources for technical debt management aren’t fully informed about the strategic direction of the enterprise, their requests might be inconsistent with enterprise strategy. As a result of a cognitive bias [Kahneman 2011] known as the halo effect [Thorndike 1920], decision makers might notice that portions of those proposals don’t take enterprise strategy into account properly. This can cause them to tend to discount valid portions of the technologists’ proposals.
Decision makers might be affected by unrealistic optimism [Weinstein 1996], also known as optimism bias. It’s a cognitive bias that can cause them to discount the sometimes-vivid warnings of technologists about the unfavorable consequences of failing to provide technical debt management resources.
Last words
Investigations of organizational behavior relative to technical debt and the Dunning-Kruger effect could be fruitful. For example, what is the degree of correlation between burdens of technical debt and the incidence of rejected or severely curtailed proposals for resources? Also rewarding would be a survey of the nearly 200 known cognitive biases, to determine which of them might be most likely to affect decision-making relative to technical debt.
References
[Austin 1996] Robert D. Austin. Measuring and Managing Performance in Organizations. New York: Dorset House, 1996. ISBN:0-932633-36-6
Contains an extensive discussion of the consequences of partial supervision of performance. Since technical debt can only be partially supervised, the concept is relevant to understanding the effects of performance management systems on technical debt. Order from Amazon
[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.
[Gaskin 1991] Steven P. Gaskin, Abbie Griffin, John R. Hauser, Gerald M. Katz, and Robert L. Klein. “Voice of the Customer,” Marketing Science 12:1, 1-27, 1991.
[Guo 2011] Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, and Clauirton Siebra. “Tracking Technical Debt: An Exploratory Case Study,” 27th IEEE International Conference on Software Maintenance (ICSM), 2011, 528-531.
[Kahneman 1977] Daniel Kahneman and Amos Tversky. “Intuitive Prediction: Biases and Corrective Procedures,” Technical Report PTR-1042-7746, Defense Advanced Research Projects Agency, June 1977.
[Kohn 1999] Alfie Kohn. Punished by rewards: The trouble with gold stars, incentive plans, A's, praise, and other bribes. Boston: Houghton Mifflin Harcourt, 1999. ISBN:0-395-71090-1
[Kruger 1999] Justin Kruger and David Dunning. “Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments,” Journal of Personality and Social Psychology, 77:6, 1121-1134, 1999.
[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.
[Thorndike 1920] Edward L. Thorndike. “A constant error in psychological ratings,” Journal of Applied Psychology, 4:1, 25-29, 1920. doi:10.1037/h0071663
The first report of the halo effect. Thorndike found unexpected correlations between the ratings of various attributes of soldiers given by their commanding officers. Although the halo effect was thus defined only for rating personal attributes, it has since been observed in assessing the attributes of other entities, such as brands. Available: here; Retrieved: December 29, 2017
[US Army 2010] U.S. Army (2010) Field Manual 5.0 – The Operations Process U.S. Department of the Army.
Describes the concept, value, and importance of the doctrine of commander’s intent. See the index for “commander’s intent,” and especially paragraphs 2-90 and 2-91. Available: here; Retrieved: Dec. 22, 2019.
[Weinstein 1996] Neil D. Weinstein and William M. Klein. “Unrealistic Optimism: Present and Future,” Journal of Social and Clinical Psychology 15:1, 1-8, 1996. doi:10.1521/jscp.1996.15.1.1
[Wight 2017] Philip Wight. “How the Alaska Pipeline Is Fueling the Push to Drill in the Arctic Refuge,” YaleE360, Yale School of Forestry & Environmental Studies, November 16, 2017.
[Yen 2015] Terry Yen, Laura Singer. “Oil exploration in the U.S. Arctic continues despite current price environment,” Today in Energy blog, U.S. Energy Information Administration, June 12, 2015.
