Decisions to retire the legacy technical debt carried by irreplaceable assets are not to be taken lightly. As decision makers gather information and recommendations from all around the organization, most will discover that information and recommendations aren’t sufficient for making sound decisions about technical debt retirement. The issues are complex. Education is also needed. It’s entirely possible that in some organizations, confronted with a set of decisions regarding legacy technical debt retirement in irreplaceable assets, the existing executive team might be out of its depth. To understand how this situation can arise, let’s explore the nature of legacy technical debt retirement decisions.
A common technical debt retirement scenario
What compels the leaders of a large enterprise to consider retiring the technical debt encumbering one of its irreplaceable assets is fairly simple: cost. Decision makers usually begin by investigating the cost of replacing the asset—the option I’ve oh-so-cleverly called “Replace the Asset.” They then typically conclude that replacement isn’t affordable. At this point, many decision makers choose the option I’ve called “Do nothing.” Time passes. A succession of incidents occurs, in which repairs to the asset or enhancements of the asset are required. And I use the term required here to mean “essential to the viability of the business.”
Engineers then do their best to meet the need, but the cost is high, and the work takes too long. The engineers explain that the problems are due, in part, to the heavy burden technical debt in this particular asset. Eventually the engineers are asked to estimate the cost of “cleaning things up.” Decision makers receive the estimates and conclude that it’s “unaffordable right now.” They ask the engineers to “make do.” In other words, they stick with the Do Nothing option.
After a number of cycles repeating this pattern, decision makers finally agree to provide time and resources for technical debt retirement, but only because it’s the least bad alternative. The other alternatives—Replace the Asset, and Do Nothing—clearly won’t work and haven’t worked, respectively.
So there we are. The organization has been forced by events to address the technical debt problems in this irreplaceable asset. And that’s where the trouble begins.
Decisions about retiring legacy technical debt
In scenarios like the one above, the fundamental decision has already been made: the enterprise will be retiring legacy technical debt from an irreplaceable asset. But that’s just the first ripple of waves of decisions to come, made by many people in a variety of roles throughout the enterprise. Let’s now have a look at a short catalog of what’s in store for such an enterprise.
Recall that most large technical debt retirement projects probably exhibit a high degree of wickedness in the sense of Rittel and Webber [Rittel 1973]. One consequence of this property is the need to avoid do-overs. That is, once we make a decision about how to proceed to the next bit of the work, we want that decision to be correct, or at least, good enough. It should not leave the enterprise in a state that’s more difficult to resolve than the state in which we found it. Since another property of wicked problems is the prevalence of surprises, most decisions must be made in a collaborative context, which affords the greatest possibility of opening the decision process to diverse perspectives. We must therefore regard collaborative decision-making at every level as a highly valued competency.
What follows is the promised catalog of decision types.
- Strategic decisions
- Organizational decisions
- Engineering decisions
- Decisions about pace
- Resource decisions
This decision category leads the list because it provides the highest leverage potential for changing enterprise behavior vis-à-vis technical debt. Organizations that are confronting the problem of technical debt retirement from irreplaceable assets would do well to begin by acknowledging that although they might be able to devise tactics for dealing with the debt burdening these assets right now, they must make a strategic change if they want to avoid a recurrence. Accumulating debt to a level sufficient to compel chartering a major debt retirement project took time. It took years of deferring the inevitable. A significant change of enterprise strategy is necessary.
When changing complex social systems, applying the concept of leverage provides a critical advantage. In this instance, following the work of Meadows [Meadows 1997] [Meadows 1999] [Meadows 2008], we can devise interventions at several points that can have great impact on both the level of technical debt and its rate of accumulation. The leverage points of greatest interest are Feedback Loops, Information Flows, Rules, and Goals. For example, the enterprise can set a strategic goal of a specific volume of incremental technical debt incurred per project, normalized by project budget, as I discussed in the post, “Leverage points for technical debt management.”
One might reasonably ask why enterprise strategy must change; wouldn’t a change in technology strategy suffice? Changing how engineers go about their work would help—indeed in most cases it’s necessary. But because the conditions and processes that lead to technical debt formation and persistence transcend engineering activities, additional changes are required to achieve the objective of controlling technical debt.
Some technical debt is strategic—it’s incurred as the result of a conscious business decision. But some is non-strategic. We might even be unaware of how it occurred. However, both kinds of technical debt can arise as a result of non-technical factors. Read a review of non-technical precursors of non-strategic technical debt.
Before chartering a technical debt retirement project (DRP) for an irreplaceable asset, or a group of irreplaceable assets, it’s wise to consider how to embed that project in the enterprise.
The default organizational form for debt retirement projects concerned with an asset A is usually the same form that would be used for major projects focused on asset A. If the Information Technology (IT) unit would normally address issues in A, the debt retirement effort usually would be organized under IT. If A is a software product normally attended to in a product group, that same group would likely have responsibility for the DRP for asset A.
Although these default organizational structures are somewhat sensible, both technically and politically, there’s an alternative approach worth investigating. It entails establishing a technical debt retirement function that becomes a center of excellence for executing technical debt retirement projects, and for developing and injecting sound technical debt management practice into the enterprise. Such an approach is especially useful if multiple debt retirement projects are needed.
The fundamental concept that makes the center-of-excellence approach necessary is the wickedness of the technical debt retirement problem. To address the problem at scale requires capabilities beyond what IT can provide; beyond what product units can provide; indeed, beyond what any of the conventional organizational elements can provide. The reason for this is that the explosion of technical debt in most organizations is an emergent phenomenon. Every organizational unit contributed to the formation of the problem. And every organizational unit must contribute to its resolution.
A technical debt center of excellence is an approach that might be capable not only of synthesizing the expertise of all elements of the enterprise, but also might be capable of bringing new approaches into the enterprise from external sources.
Engineers have a tendency to identify and classify technical debt items on technical grounds. Further, they tend to set technical debt retirement priorities on a similar basis. That is, they’re inclined to set priorities highest for those debt items that they (a) recognize as debt items and (b) see as imposing high levels of MICs charged to engineering accounts. Engineers are less likely to assign high priorities to technical debt that generates MICs that are charged to revenue, or to other accounts, because those MICs are less evident—and in many cases invisible—to engineers.
Decisions regarding recognition of technical debt items and setting priorities for retiring them must take technological imperatives into account, but they must also account for MICs of all forms. Priorities must be consistent with enterprise imperatives.
Paraphrasing Albert Einstein, technical debt retirement projects should be executed as rapidly as possible, and no faster. The tendency among non-engineers and non-technical decision-makers is to push for rapid completion of debt retirement projects, for three reasons. First, everyone, like the engineers, wants the results that debt retirement will bring. Second, everyone, like the engineers, wants an end to the inevitable disruptions debt retirement projects cause. And finally, the longer the project is underway, the more it might cost.
For these reasons, once the decision to retire the debt is firmly in hand, the enterprise might have a tendency to apply financial resources at a rate that exceeds the ability of the project team to execute the project responsibly. When that happens, rework results. And for wicked problems like debt retirement, rework is the path to catastrophe.
Decisions about pace and team scale need to be regarded as tentative. Regular reviews can ensure that the resource level is neither too low nor too high. Even when the engineers are given control over these decisions, they must be reviewed, because pressures for rapid completion can be so severe that they can compromise the judgment of engineers about how well they can manage the resources applied to the project.
Debt retirement projects concerned with legacy irreplaceable assets are different from most other projects the enterprise undertakes. Estimates of the labor hours required are more likely to be incorrect on the low side than are analogous estimates for other projects, because so much of the work involves pieces of assets with which few engineering staff have any experience. But with respect to resources, underestimating labor requirements isn’t the real problem. Non-labor resources are the real problem.
Because the assets are irreplaceable, it’s likely that they’re needed for ongoing operations. In some cases, the assets are needed continuously. Many organizations have kept such assets operational by exploiting hours of downtime during periods of low demand, usually scheduled and announced in advance. While these practices are likely sufficient for the relatively minor and infrequent changes usually associated with routine maintenance and enhancement, debt retirement imposes much more severe burdens on the organization than these short access windows can support. Effective debt retirement projects need far more access to the asset—a level of access that continuous delivery practices can provide [Humble 2010].
However, assets whose designs predate the widespread use of modern practices such as continuous delivery might not be compatible with the infrastructure that these practices require. And in organizations that haven’t yet adopted such practices, staff familiar with them might be in short supply. For these reasons, we must regard as developmental any early projects whose objectives are retiring technical debt from irreplaceable assets. They’re retiring the technical debt, of course, but they’re also developing the practices and infrastructure needed to support technical debt retirement projects. This dual purpose is what drives the surprisingly high non-labor costs and investments associated with early technical debt retirement projects.
The investments required might include such “items” as a staging environment, which “is a testing environment identical to the production environment” [Humble 2010]; extensive test automation, including results analysis; blue-green deployment infrastructure; automation-assisted rollback; and zero-downtime release infrastructure. Decisions to make investments require an appreciation of their value to the enterprise. They enable the enterprise to deal effectively with the wicked problem of technical debt retirement.
Because every situation and every organization is unique, few general guidelines are available for making these decisions. The criteria most organizations have been using for dealing with (or avoiding) the issue of technical debt have produced the problems they now face. So, to succeed from this point, whatever criteria they use in the future must be different. My own view is that short-term thinking is at the heart of the problem, but it’s a wicked problem. The long-term solution will not be simple.
Available: here; Retrieved: June 28, 2018
Available: here; Retrieved: June 2, 2018.
- Leverage points for technical debt
- Synergy between the reification error and confirmation bias
- Legacy TDR decisions
Available: here; Retrieved: October 16, 2018
- Glossary and Terminology
- Retiring technical debt can be a wicked problem
- Degrees of wickedness
- Indicators of wickedness
- Retiring technical debt from irreplaceable assets
- Legacy TDR decisions