Last updated on July 15th, 2021 at 07:17 pm
Some irreplaceable assets carry legacy technical debt. Although retiring an asset retires its technical debt, that option isn’t available for irreplaceable assets. We need another option. For irreplaceable assets, we need to find a way to retire the debt. As decision makers gather information and recommendations from around the organization, most will come to an unsettling conclusion. They’ll find that information and recommendations aren’t sufficient for making sound decisions about technical debt retirement. The issues are complex. Education is also needed. It’s entirely possible that in some organizations, the existing executive team might be out of its depth. To understand how this situation can arise, let’s explore the nature of legacy technical debt retirement decisions.
A common technical debt retirement scenario
What compels the leaders of a large enterprise to consider retiring the technical debt encumbering one of its irreplaceable assets is fairly simple: cost. Decision makers usually begin by investigating the cost of replacing the asset. This is the option I’ve cleverly called “Replace the Asset.” They then typically conclude that replacement isn’t affordable. At this point, many decision makers choose the option I’ve called “Do nothing.” Time passes. A succession of incidents occurs, in which teams attempt required repairs or enhancements of the asset. And I use the term required here to mean “essential to the viability of the business.”
Engineers then do their best to meet the need, but the cost is high, and the work takes too long. The engineers explain that the problems are due, in part, to the heavy burden of technical debt. Eventually someone asks the engineers to estimate the cost of “cleaning things up.” Decision makers receive the estimates and conclude that it’s “unaffordable right now.” They ask the engineers to “make do.” In other words, they stick with the Do Nothing option.
After a number of cycles repeating this pattern, decision makers finally agree to provide time and resources for technical debt retirement, but only because it’s the least bad alternative. The other alternatives—Replace the Asset, and Do Nothing—clearly won’t work and haven’t worked, respectively.
So there we are. Events have forced the organization to address the technical debt problems in this irreplaceable asset. And that’s where the trouble begins.
Decisions about retiring legacy technical debt
In scenarios like the one above, people have already made the fundamental decision: the enterprise will be retiring legacy technical debt from an irreplaceable asset. But that’s just the first ripple of waves of decisions to come. Many people in a variety of roles throughout the enterprise will be making many decisions. Let’s now have a look at a short catalog of what’s in store for such an enterprise.
Recall that most large technical debt retirement projects probably exhibit a high degree of wickedness in the sense of Rittel and Webber [Rittel 1973]. One consequence of this property is the need to avoid do-overs. That is, once we make a decision about how to proceed to the next bit of the work, we want that decision to be correct, or at least, good enough. The consequences of that decision should not leave the enterprise in a state that’s more difficult to resolve than the state in which we found it. Since another property of wicked problems is the prevalence of surprises, most decisions must be made in a collaborative context, which affords the greatest possibility of opening the decision process to diverse perspectives. We must therefore regard collaborative decision-making at every level as a highly valued competency.
What follows is the promised catalog of decision types.
- Strategic decisions
- Organizational decisions
- Engineering decisions
- Decisions about pace
- Resource decisions
Strategic decisions
This decision category leads the list. It provides the highest leverage potential for changing enterprise behavior vis-à-vis technical debt. Organizations confronting the problem of technical debt retirement from irreplaceable assets would do well to begin by acknowledging that although they might be able to deal with the debt burdening these assets right now, they must make a strategic change if they want to avoid even worse trouble. Accumulating debt to a level sufficient to compel chartering a major debt retirement project took years of deferring the inevitable. A significant change of strategy is necessary.
When changing complex social systems, applying the concept of leverage provides a critical advantage. Following the work of Meadows [Meadows 1997] [Meadows 1999] [Meadows 2008], we can devise interventions at several points that can have great impact on the rate of accumulation of technical debt. The leverage points of greatest interest are Feedback Loops, Information Flows, Rules, and Goals. For example, the enterprise can set a strategic goal of a specific volume of incremental technical debt incurred per project, normalized by project budget. See “Leverage points for technical debt management.”
One might reasonably ask why enterprise strategy must change; wouldn’t a change in technology strategy suffice? Changing how engineers go about their work would help—indeed in most cases it’s necessary. But because the conditions and processes that lead to technical debt formation and persistence transcend engineering activities, additional changes are required to achieve the objective of controlling technical debt.
Some technical debt is incurred as the result of a conscious decision. But some is nonstrategic. We might even be unaware of how it occurred. Both kinds of technical debt can arise as a result of nontechnical factors. Read a review of nontechnical precursors of nonstrategic technical debt.
Organizational decisions
Before chartering a technical debt retirement project (DRP) for an irreplaceable asset, it’s wise to consider how to embed the DRP in the enterprise.
The default organizational form for DRPs concerned with an asset A is usually analogous to that used for major projects focused on asset A. If the Information Technology (IT) unit would normally address issues in asset A, the debt retirement effort usually would be organized under IT. If A is a software product normally attended to in a product group, that same group would likely have responsibility for the DRP for asset A.
Although these default organizational structures are both technically and politically sensible, there’s an alternative approach worth investigating. It entails establishing a technical debt retirement function that becomes a center of excellence for executing technical debt retirement projects. That unit is also responsible for developing sound technical debt management practice. Such an approach is especially useful if the organization contemplates multiple debt retirement projects.
The fundamental concept that makes the center-of-excellence approach necessary is the wickedness of the technical debt retirement problem. To address the problem at scale requires capabilities beyond what IT, product units, or any conventional organizational elements can provide. The explosion of technical debt in most organizations is an emergent phenomenon. Every organizational unit contributed to the formation of the problem. And every organizational unit must contribute to its resolution.
Engineering decisions
Engineers tend to identify and classify technical debt items on technical grounds. Further, they tend to set technical debt retirement priorities on a similar basis. That is, they tend to set priorities highest for those debt items that they (a) recognize as debt items and (b) see as imposing high levels of MICs charged to engineering accounts. Engineers are less likely to assign high priorities to technical debt that generates MICs that are charged to revenue, or to other accounts, because those MICs are less evident—and in many cases less visible—to engineers.
Decisions regarding recognition of technical debt items and setting priorities for retiring them must take technological imperatives into account, but they must also account for MICs of all forms. Priorities must be consistent with enterprise imperatives.
Decisions about pace
Paraphrasing Albert Einstein, technical debt retirement projects should be executed as rapidly as possible, and no faster. The tendency among nonengineers and nontechnical decision makers is to push for rapid completion of debt retirement projects, for three reasons. First, everyone, like the engineers, wants the results that debt retirement will bring. Second, everyone, like the engineers, wants an end to the inevitable disruptions debt retirement projects cause. And finally, the longer the project is underway, the more it might cost.
For these reasons, once the decision to retire the debt is firmly in hand, the enterprise might have a tendency to apply financial resources at a rate that exceeds the ability of the project team to execute the project responsibly. When that happens, rework results. And for wicked problems like debt retirement, rework is the path to catastrophe.
Decisions about pace and team scale need to be regarded as tentative. Regular reviews can ensure that the resource level is neither too low nor too high. Even when the engineers are given control over these decisions, these decisions must be reviewed, because pressures for rapid completion can be so severe that they can compromise the judgment of engineers about how well they can manage the resources applied to the project.
Resource decisions
Debt retirement projects concerned with legacy irreplaceable assets are different from most other projects. Estimates of the labor hours required are more likely to be incorrect on the low side, because so much of the work involves pieces of assets with which few staff have experience. But with respect to resources, underestimating labor requirements isn’t the real problem. Nonlabor resources are the real problem.
Irreplaceable assets probably provide critical support of ongoing operations. In some cases, the need for the assets is continuous. Many organizations have kept such assets operational by using periods of low demand for maintenance, usually scheduled and announced in advance. These practices are likely adequate for routine maintenance and enhancement. But debt retirement need a level of access to the asset that continuous delivery practices can provide [Humble 2010].
However, assets whose designs predate the widespread use of modern practices such as continuous delivery might not be compatible with the infrastructure that these practices require. In organizations that haven’t yet adopted such practices, few if any staff are experienced with them. We must therefore regard as developmental any projects whose objectives are retiring technical debt from irreplaceable assets. They’re retiring the technical debt, but they’re also developing the practices and infrastructure needed for debt retirement projects. This dual purpose drives the surprisingly high nonlabor costs associated with early debt retirement projects.
The investments required might include such “items” as a staging environment, which “is a testing environment identical to the production environment” [Humble 2010]; extensive test automation, including results analysis; blue-green deployment infrastructure; automation-assisted rollback; and zero-downtime release infrastructure. Decisions to make investments require an appreciation of their value to the enterprise. They enable the enterprise to deal effectively with the wicked problem of technical debt retirement.
Last words
Because every situation and every organization is unique, few general guidelines are available for making these decisions. The criteria most organizations have been using for dealing with (or avoiding) the issue of technical debt have produced the problems they now face. So, to succeed from this point, whatever criteria they use in the future must be different. My own view is that short-term thinking is at the heart of the problem, but it’s a wicked problem. The long-term solution will not be simple.
References
[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.
[Meadows 1997] Donella H. Meadows. “Places to Intervene in a System,” Whole Earth, Winter 1997.
Available: here; Retrieved: June 28, 2018
[Meadows 1999] Donella H. Meadows. “Leverage Points: Places to Intervene in a System,” Hartland VT: The Sustainability Institute, 1999.
Available: here; Retrieved: June 2, 2018.
- Leverage points for technical debt management
- Synergy between the reification error and confirmation bias
- Legacy technical debt retirement decisions
[Meadows 2008] Donella H. Meadows and Diana Wright. Thinking in Systems: A Primer. White River Junction, VT: Chelsea Green Publishing, 2008.
[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.
Available: here; Retrieved: October 16, 2018
- Glossary and Terminology
- Retiring technical debt can be a wicked problem
- Degrees of wickedness
- Nine indicators of wickedness
- Retiring technical debt from irreplaceable assets
- Legacy technical debt retirement decisions
Other posts in this thread
- Retiring technical debt from irreplaceable assets
- Where is the technical debt?
- Auxiliary technical debt: Rules of engagement
- Retiring localizable technical debt
- Controlling incremental technical debt
- Automation-assisted technical debt retirement
- Outsourcing Technical Debt Retirement Projects
- Refactoring for policymakers
- The trap of elegantly stated goals
