Month: December 2018

Posted on

Controlling incremental technical debt

Last updated on July 16th, 2021 at 03:40 pm

A sinking rowboat provides a metaphor for controlling incremental technical debt
A sinking rowboat provides a useful metaphor for illustrating the process of controlling incremental technical debt. The enterprise is the rowboat. The leaks are the properties of the enterprise and its environment that lead to creating incremental technical debt. The water entering the boat through leaks is incremental technical debt. The accumulated water in the bottom of the boat is legacy technical debt.

All technical debt in enterprise assets is either incremental technical debt or legacy technical debt or both. Incremental technical debt is technical debt newly incurred. It can be newly incurred exogenous technical debt, or it can be endogenous technical debt incurred either in projects currently underway, or projects just recently completed. Legacy technical debt is technical debt associated with assets, and which wasn’t incurred recently or which exists in any form prior to undertaking work on those assets. All legacy technical debt was at some point incremental technical debt. The vast amounts of legacy technical debt most organizations now carry are nothing more than the accumulation of incremental technical debt. The path to managing legacy technical debt therefore begins with controlling incremental technical debt.

The leaky rowboat metaphor

Organizations are more likely to gain control of their legacy technical debt portfolio if they begin by controlling the formation of incremental technical debt, and its transformation into legacy technical debt. A metaphor might make this clear:

If you find yourself in a sinking rowboat, bailing out some of the water is a good idea. It might even be necessary in the short term. But at some point, fixing the leaks where the water comes in is advisable. Unless you address the existing leaks, and prevent new ones from forming, your fate is sealed. You’ll spend increasing portions of your time, energy, and resources bailing out your leaky rowboat. You’ll spend declining portions of your time, energy, and resources rowing the boat towards your objective. And when you do devote some time and energy to rowing towards your objective, you’ll find the rowing surprisingly difficult. The boat’s leaks make it ride lower in the water. And because you must propel not only the boat and its payload, but also the dead weight of the water in the bottom of the boat.

In this metaphor, legacy technical debt is the water in the bottom of the boat, and incremental technical debt is the water coming in through the leaks. The leaks are the proximate “causes” of technical debt. The root causes of the leaks are the root causes of technical debt.

Setting technical debt management priorities

If the enterprise is in the midst of a legacy technical debt emergency, retiring some of it is necessary in the short term. But unless the enterprise addresses incremental technical debt and its root causes, a new burden of legacy technical debt will accumulate. That accumulation is then likely to eliminate the benefits of having retired the current burden of legacy technical debt.

So after the legacy technical debt emergency is passed—or if resources permit, during the emergency—establishing measures, procedures, and practices for controlling incremental technical debt would be prudent.

This change might be less challenging than it sounds. With respect to endogenous incremental technical debt, the teams that incurred it are either still at work, or just recently dispersed. Their understanding of the incremental technical debt is still fresh in their minds. If their projects are still underway, and if budget and schedule permit, retiring the incremental technical debt in the context of those projects is a superior strategy. For projects that have already delivered their work, a less preferable but still practical approach involves re-assembling some of the team. They then work to retire the incremental technical deb, while memories are still fresh.

For incremental exogenous technical debt, the team that was active when the debt formed might have little knowledge of how the debt came to be. In those cases, re-assembling the team provides little advantage. Specialized knowledge of the technical debt might prove more helpful in devising a strategy for retiring it.

For the most part, the problem of controlling incremental technical debt isn’t a technical one. It usually reduces to a problem of finding time and resources to undertake the task.

Why resources aren’t available to retire incremental technical debt

The immediate reason why most teams don’t have enough resources to retire their incremental technical debt is that the organization, as a whole, doesn’t plan for retiring incremental technical debt incrementally. This immediate reason, though, isn’t fundamental. The lack of resources is a symptom of deeper dysfunctions in the organization. The real question is this: Why do so many organizations fail to allocate time and resources to retire incremental technical debt incrementally? Here are three reasons.

Misunderstanding (or no understanding) of the concept of technical debt

The organization is unlikely to be able to manage any kind of technical debt unless its people understand the concept. They must understand that technical debt isn’t necessarily the result of engineering malpractice. For example, much technical debt arises as a natural result of working with technology. Another example: it can be a result of organizational forms that compel people to behave in ways that lead to technical debt. Unless the people of the organization accept these truths, allocating sufficient resources to managing incremental technical debt is unlikely.

Not appreciating the MICs

Decisions regarding technical debt management ultimately reduce to a choice between allocating precious resources to technical debt retirement, and allocating them elsewhere. To make this choice responsibly, it’s necessary to fully appreciate the cost of carrying technical debt. Most believe that these costs appear in the form of lost engineering productivity. While that is indeed a factor, other factors can be far more important.

For example, if entry into an important market is delayed by even as little as 30 days due to debt-depressed engineering productivity, the financial consequences can be enormous and insurmountable. Or delays in diagnosing and repairing a fault in a product can produce financial liabilities that can actually sink the company. When one considers all possible financial consequences of carrying technical debt, it becomes clear that managing technical debt effectively is actually a strategy for survival. The decision to allocate appropriate resources to incremental technical debt retirement does require modeling these costs—calculations that few organizations actually undertake.

Miscalculating projections of returns on investments

Failing to estimate MICs with sufficient precision is problematic because it reduces the quality of decisions regarding short-term resource allocations. But it also affects long-term projections, which depend on estimating returns on investments. For example, to choose between investing in retiring incremental technical debt from an asset and investing in new capabilities for the same asset, one must compare the projected value of each choice. The investment decision could be biased in favor investing in new capabilities for many reasons. For example, the decision maker’s understanding of technical debt might be deficient. Another example: the calculations of MICs might be incomplete or underestimated. Incremental technical debt retirement might therefore be systematically deferred or avoided altogether.

The widespread belief that MICs consist largely of lost productivity of engineers almost ensures a dramatic underestimate of MICs.

Policy recommendations for controlling incremental technical debt

The simplistic approach to controlling incremental technical debt is to provide more money to projects and to engineering functions. While that approach will be somewhat helpful, its results will likely be disappointing when compared to approaches that combine resource augmentation with changes in enterprise policy, processes, and culture.

Let’s begin with an example of a needed cultural change. Nearly anyone who makes or influences decisions might occasionally bear some responsibility for incurring incremental technical debt. To achieve effective control of technical debt requires that everyone understand how to change their behavior. Any guiding principle must be simple to state and easy to understand, because we must communicate it to nearly everyone. Here’s a sample statement of a useful such a principle:

Those whose decisions lead to new technical debt are accountable for securing the resources needed to retire that debt. They might also be accountable for supplying compensating resources to those within the enterprise, or among its customers, who are negatively affected. For example, those negatively affected might suffer depressed operational effectiveness during the period in which that technical debt is outstanding.

I call this the Principle of Accountability. It’s a corollary to what Weinberg calls “Ford’s Fundamental Feedback Formula” [Weinberg 1985], which captures the idea that people make better choices when they must live with the consequences of those choices.

General guiding principles are necessary, but not sufficient. Here are five examples of changes that help in controlling incremental technical debt [Brenner 2017b].

1.       Adopt a shared concept vocabulary

There must be general agreement among all parties about the meanings of concepts that relate to incremental technical debt formation.

Examples: the definitions of “done” vis-à-vis projects, strategic technical debt, reckless technical debt, unethical technical debt, exogenous-technical-debt, endogenous technical debt, MICs, MPrin, and more. This material must be available in an education program and a new-employee orientation program.

2.       Accept that technical debt is a fact of technological life

There is a widespread belief that most technical debt results from engineering malpractice. Although some technical debt does arise this way, most does not. For examples of other causes, see “Nontechnical precursors of nonstrategic technical debt.” Some technical debt arises because of advances external to the enterprise, beyond its control. Development-induced or field-revealed discoveries are especially difficult to avoid. In many instances, technical debt is an inevitable result of using technology.

3.       Track the cost of carrying technical debt

The cost of retiring a particular class of technical debt (its MPrin) is significant only in the context of planning or setting priorities for resource allocation. In all other contexts, knowing that cost has little management value. What does matter, at all times, is the cost of carrying that technical debt—the MICs, or metaphorical interest charges. (See “The Principal Principle: Focus on MICs.”) MICs can fluctuate wildly [Garnett 2013].

Build and maintain expertise for estimating and tracking the costs of incurring and carrying each class of technical debt. Know how much each kind of technical debt contributes to these costs, now and for the next few years.

4.       Assign accountability for kinds of technical debt

Some kinds of incremental technical debt result from actions (or inactions) within the enterprise. Some do not. To control the incremental technical debt that arises from internal causes, hold people accountable for the debt their actions generate. Use Fowler’s Technical Debt Quadrant [Fowler 2009] as the basis for assessing and distributing internal financial accountability. It’s useful for both debt retirement costs (MPrin) and metaphorical interest charges (secured technical debt is technical debt for which resources have been allocated (possibly in a forward time period) to guarantee the debt’s retirement and possibly its associated MICs.

This policy implies that deliberately incurred technical debt, whether incurred strategically or recklessly, must be secured. If anyone involved feels that technical debt has been incurred, a dispassionate third party reviews project deliverables for the presence of technical debt. However, allocating resources might require securing commitments of resources for future fiscal periods. For many organizations, such forward commitments might require modifying the management accounting system.

Last words

Controlling incremental technical debt requires changes well beyond the behavior and attitudes of engineering staff, or the technologies they employ. Achieving control of incremental technical debt formation requires engagement with enterprise culture to alter the behavior and attitudes of most of the people of enterprise.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Posted on

Retiring localizable technical debt

Last updated on July 17th, 2021 at 06:53 am

Electricity pylons, Hamilton Beach, Ontario, Canada
Electricity pylons, Hamilton Beach, Ontario, Canada: a small part of the AC power grid, which seems destined one day to manifest a great deal of nonlocalizable technical debt. Photo by Ibagli courtesy Wikimedia Commons. 
Pylons in the same line are visible in Google Street View.

When technical debt appears as discrete chunks—when it’s localizable—we can often retire the debt incrementally. We can retire it system-by-system, module-by-module, or even instance-by-instance. These approaches offer great flexibility, both technically and financially. That makes retiring localizable technical debt a particularly manageable challenge.

Localizable technical debt

In “Technical debt in a rail system,” I explored the case of Amtrak’s Acela Express. In that example, I explained that Acela’s passenger cars can tilt to compensate for centrifugal forces that appear when the train rounds curves. The technical debt is in the form of tracks that are too close together to permit the trains to tilt as much as they’re designed to. That limits the trains’ speed rounding curves. The instances of this debt are the curves in which the tracks are too close together. These instances are thus inherently localizable.

In “Debt contagion: how technical debt can create more technical debt,” I described an example in which an organization is unable to upgrade its desktop computers from Windows 8 to Windows 10. In this case, each computer running Windows 8 is an instance of this form of technical debt.

Both of these examples illustrate localizable technical debt. Each instance is self-contained. We can “point” to it as an instance of the debt in question. But localizable technical debt need not be associated with hardware. In software systems, for example, localizable technical debt can exist in a module interface. If interface meets a requirement that’s no longer relevant, it might contain technical debt. That module and any other modules that interact with that interface therefore manifest that technical debt.

Nonlocalizable technical debt

Nonlocalizable technical debt is debt for which the instances are amorphous or system-wide. Or they span the bulk of the system, if not all of it. Retiring nonlocalizable technical debt typically requires extensive reengineering of the assets that manifest it.

For the most part, nonlocalizable technical debt arises at the level of system architecture or above. One can easily imagine this occurring in software systems, where physical constraints have little meaning. But let’s consider a hardware system to illustrate the importance of this concept.

An illustration of nonlocalizable technical debt

Until relatively recently in the United States, most electric power consumers used power for incandescent lamps, heating, or for electric motors. These applications are compatible with an alternating current power distribution system (AC grid). The AC grid is more efficient than an equivalent direct-current architecture (DC grid) when power generation plants are few and relatively distant from power load sites. The efficiency advantage is due to AC’s lower transmission losses compared to DC.

However, advances in electronics and in distributed power generation are eroding the advantages of the AC grid [Dragičević 2016]. Most electronic devices—phones, computers, rechargeable power tools, LED lighting, and electric vehicles—use DC internally. To access the AC grid, they change AC power into DC power, which entails efficiency losses.

Moreover, most renewable power generation systems generate DC inherently. Wind turbines generate AC at a frequency determined by wind power conversion efficiency, but they then convert it to DC before a second conversion to AC at the frequency of the AC grid. And because solar and wind power generators are geographically dispersed, they’re often situated near their load sites. Therefore, the losses due to transmission from generation site to load site are less important than they would be if the generation sites were few, concentrated, and at great distances from the loads they serve.

Our current AC grid architecture is likely to become a net disadvantage in the not-too-distant future. If that happens, we could come to regard the current AC grid as manifesting technical debt. The devices that are designed for the AC grid would also manifest that debt. However, localizing that debt in each device and each component of the AC grid would make little sense. The technical debt in question would reside in the grid architecture, as a whole. It would be inherently nonlocalizable.

Addressing localizable technical debt

As noted above, we can often retire localizable technical debt incrementally—instance-by-instance. In many cases, this enables engineers to address the debt at times and in sequences that are compatible with organizational priorities. By spreading the effort over time, the organization can ensure that costs are within the organization’s capacity in any given fiscal period. This isn’t always possible for localizable technical debt. And engineers are often justifiably averse to the temporary non-uniformity that results from incremental debt retirement. But exploiting localizability when planning debt retirement is often a useful strategy for retiring technical debt economically.

Temporary structures

Retiring localizable technical debt incrementally does present some problems. During the retirement process, for any given instance, temporary structures might be necessary to support continued operation with minimal service disruption. For example, with the Acela tracks, an alternate line might serve while the new track installation is in progress. Or the new track might follow a course at some distance from the existing track while trains continue using the existing track. Both approaches require investment beyond the investment required for the new track itself. Some managers have little appetite for such temporary investments. But temporary investments are in a real sense part of the MICs on that debt. They’re unusual in the sense that they’re part of the debt retirement effort, but they’re still MICs. In a way, they’re analogous to the charges that might appear when terminating an auto lease.

Entanglements of different kinds of technical debt

Another consideration when addressing localizable technical debt is its entanglement with other forms of technical debt. With respect to the effort to retire one kind of localizable technical debt, these other forms of technical debt are what I’ve called auxiliary technical debt (ATD). Consider carefully the time order of efforts to retire the localizable technical debt and one or more forms of its ATD. Because retiring localizable technical debt can seem deceptively straightforward, the temptation to deal with it before addressing some of the ATD can be difficult to resist. But dealing with some of the ATD first might actually be the wiser course. For example, when doing so eliminates numerous instances of the localizable technical debt, dealing with the ATD first can produce real savings.

One note of caution

Within the category of localizable technical debt are kinds of debt that are so widespread that retiring them affects a large part of the asset. Each instance of such debt might be identifiable and localized. But the instances are so widespread that they collectively have the properties of nonlocalizable debt. Incremental retirement might still be possible, but a more global retirement effort might be safer and less disruptive.

One approach technologists usually favor is suspending all other work while the debt in question undergoes retirement. While that approach might indeed be safest, all stakeholders must accept and understand the technical issues. And the technologists must understand the concerns of all stakeholders. A joint decision about the retirement strategy among all stakeholders, including technologists, is probably safest.

Last words

In the context of debt retirement projects, localizable technical debt provides needed flexibility. Often, the non-uniformity that results from retiring localizable technical debt instance-by-instance can be reduced before the debt retirement project is completed. In the meantime, the team can be relatively free to retire the localizable debt in whichever order is most fitting.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Dragičević 2016] Tomislav Dragičević, Xiaonan Lu, Juan C. Vasquez, and Josep M. Guerrero. “DC Microgrids–Part II: A Review of Power Architectures, Applications and Standardization Issues,” IEEE Transactions on Power Electronics, vol 31:5, 3528-3549, 2016.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Posted on

Legacy technical debt retirement decisions

Last updated on July 15th, 2021 at 07:17 pm

Two alternatives to retiring legacy technical debt in irreplaceable assets
Two alternatives to retiring legacy technical debt in irreplaceable assets. Neither one works very well.

Some irreplaceable assets carry legacy technical debt. Although retiring an asset retires its technical debt, that option isn’t available for irreplaceable assets. We need another option. For irreplaceable assets, we need to find a way to retire the debt. As decision makers gather information and recommendations from around the organization, most will come to an unsettling conclusion. They’ll find that information and recommendations aren’t sufficient for making sound decisions about technical debt retirement. The issues are complex. Education is also needed. It’s entirely possible that in some organizations, the existing executive team might be out of its depth. To understand how this situation can arise, let’s explore the nature of legacy technical debt retirement decisions.

A common technical debt retirement scenario

What compels the leaders of a large enterprise to consider retiring the technical debt encumbering one of its irreplaceable assets is fairly simple: cost. Decision makers usually begin by investigating the cost of replacing the asset. This is the option I’ve cleverly called “Replace the Asset.” They then typically conclude that replacement isn’t affordable. At this point, many decision makers choose the option I’ve called “Do nothing.” Time passes. A succession of incidents occurs, in which teams attempt required repairs or enhancements of the asset. And I use the term required here to mean “essential to the viability of the business.”

Engineers then do their best to meet the need, but the cost is high, and the work takes too long. The engineers explain that the problems are due, in part, to the heavy burden of technical debt. Eventually someone asks the engineers to estimate the cost of “cleaning things up.” Decision makers receive the estimates and conclude that it’s “unaffordable right now.” They ask the engineers to “make do.” In other words, they stick with the Do Nothing option.

After a number of cycles repeating this pattern, decision makers finally agree to provide time and resources for technical debt retirement, but only because it’s the least bad alternative. The other alternatives—Replace the Asset, and Do Nothing—clearly won’t work and haven’t worked, respectively.

So there we are. Events have forced the organization to address the technical debt problems in this irreplaceable asset. And that’s where the trouble begins.

Decisions about retiring legacy technical debt

In scenarios like the one above, people have already made the fundamental decision: the enterprise will be retiring legacy technical debt from an irreplaceable asset. But that’s just the first ripple of waves of decisions to come. Many people in a variety of roles throughout the enterprise will be making many decisions. Let’s now have a look at a short catalog of what’s in store for such an enterprise.

Recall that most large technical debt retirement projects probably exhibit a high degree of wickedness in the sense of Rittel and Webber [Rittel 1973]. One consequence of this property is the need to avoid do-overs. That is, once we make a decision about how to proceed to the next bit of the work, we want that decision to be correct, or at least, good enough. The consequences of that decision should not leave the enterprise in a state that’s more difficult to resolve than the state in which we found it. Since another property of wicked problems is the prevalence of surprises, most decisions must be made in a collaborative context, which affords the greatest possibility of opening the decision process to diverse perspectives. We must therefore regard collaborative decision-making at every level as a highly valued competency.

What follows is the promised catalog of decision types.

Strategic decisions

This decision category leads the list. It provides the highest leverage potential for changing enterprise behavior vis-à-vis technical debt. Organizations confronting the problem of technical debt retirement from irreplaceable assets would do well to begin by acknowledging that although they might be able to deal with the debt burdening these assets right now, they must make a strategic change if they want to avoid even worse trouble. Accumulating debt to a level sufficient to compel chartering a major debt retirement project took years of deferring the inevitable. A significant change of strategy is necessary.

When changing complex social systems, applying the concept of leverage provides a critical advantage. Following the work of Meadows [Meadows 1997] [Meadows 1999] [Meadows 2008], we can devise interventions at several points that can have great impact on the rate of accumulation of technical debt. The leverage points of greatest interest are Feedback Loops, Information Flows, Rules, and Goals. For example, the enterprise can set a strategic goal of a specific volume of incremental technical debt incurred per project, normalized by project budget. See “Leverage points for technical debt management.”

One might reasonably ask why enterprise strategy must change; wouldn’t a change in technology strategy suffice? Changing how engineers go about their work would help—indeed in most cases it’s necessary. But because the conditions and processes that lead to technical debt formation and persistence transcend engineering activities, additional changes are required to achieve the objective of controlling technical debt.

Some technical debt is incurred as the result of a conscious decision. But some is nonstrategic. We might even be unaware of how it occurred. Both kinds of technical debt can arise as a result of nontechnical factors. Read a review of nontechnical precursors of nonstrategic technical debt.

Organizational decisions

Before chartering a technical debt retirement project (DRP) for an irreplaceable asset, it’s wise to consider how to embed the DRP in the enterprise.

The default organizational form for DRPs concerned with an asset A is usually analogous to that used for major projects focused on asset A. If the Information Technology (IT) unit would normally address issues in asset A, the debt retirement effort usually would be organized under IT. If A is a software product normally attended to in a product group, that same group would likely have responsibility for the DRP for asset A.

Although these default organizational structures are both technically and politically sensible, there’s an alternative approach worth investigating. It entails establishing a technical debt retirement function that becomes a center of excellence for executing technical debt retirement projects. That unit is also responsible for developing sound technical debt management practice. Such an approach is especially useful if the organization contemplates multiple debt retirement projects.

The fundamental concept that makes the center-of-excellence approach necessary is the wickedness of the technical debt retirement problem. To address the problem at scale requires capabilities beyond what IT, product units, or any conventional organizational elements can provide. The explosion of technical debt in most organizations is an emergent phenomenon. Every organizational unit contributed to the formation of the problem. And every organizational unit must contribute to its resolution.

Engineering decisions

Engineers tend to identify and classify technical debt items on technical grounds. Further, they tend to set technical debt retirement priorities on a similar basis. That is, they tend to set priorities highest for those debt items that they (a) recognize as debt items and (b) see as imposing high levels of MICs charged to engineering accounts. Engineers are less likely to assign high priorities to technical debt that generates MICs that are charged to revenue, or to other accounts, because those MICs are less evident—and in many cases less visible—to engineers.

Decisions regarding recognition of technical debt items and setting priorities for retiring them must take technological imperatives into account, but they must also account for MICs of all forms. Priorities must be consistent with enterprise imperatives.

Decisions about pace

Paraphrasing Albert Einstein, technical debt retirement projects should be executed as rapidly as possible, and no faster. The tendency among nonengineers and nontechnical decision makers is to push for rapid completion of debt retirement projects, for three reasons. First, everyone, like the engineers, wants the results that debt retirement will bring. Second, everyone, like the engineers, wants an end to the inevitable disruptions debt retirement projects cause. And finally, the longer the project is underway, the more it might cost.

For these reasons, once the decision to retire the debt is firmly in hand, the enterprise might have a tendency to apply financial resources at a rate that exceeds the ability of the project team to execute the project responsibly. When that happens, rework results. And for wicked problems like debt retirement, rework is the path to catastrophe.

Decisions about pace and team scale need to be regarded as tentative. Regular reviews can ensure that the resource level is neither too low nor too high. Even when the engineers are given control over these decisions, these decisions must be reviewed, because pressures for rapid completion can be so severe that they can compromise the judgment of engineers about how well they can manage the resources applied to the project.

Resource decisions

Debt retirement projects concerned with legacy irreplaceable assets are different from most other projects. Estimates of the labor hours required are more likely to be incorrect on the low side, because so much of the work involves pieces of assets with which few staff have experience. But with respect to resources, underestimating labor requirements isn’t the real problem. Nonlabor resources are the real problem.

Irreplaceable assets probably provide critical support of ongoing operations. In some cases, the need for the assets is continuous. Many organizations have kept such assets operational by using periods of low demand for maintenance, usually scheduled and announced in advance. These practices are likely adequate for routine maintenance and enhancement. But debt retirement need a level of access to the asset that continuous delivery practices can provide [Humble 2010].

However, assets whose designs predate the widespread use of modern practices such as continuous delivery might not be compatible with the infrastructure that these practices require. In organizations that haven’t yet adopted such practices, few if any staff are experienced with them. We must therefore regard as developmental any projects whose objectives are retiring technical debt from irreplaceable assets. They’re retiring the technical debt, but they’re also developing the practices and infrastructure needed for debt retirement projects. This dual purpose drives the surprisingly high nonlabor costs associated with early debt retirement projects.

The investments required might include such “items” as a staging environment, which “is a testing environment identical to the production environment” [Humble 2010]; extensive test automation, including results analysis; blue-green deployment infrastructure; automation-assisted rollback; and zero-downtime release infrastructure. Decisions to make investments require an appreciation of their value to the enterprise. They enable the enterprise to deal effectively with the wicked problem of technical debt retirement.

Last words

Because every situation and every organization is unique, few general guidelines are available for making these decisions. The criteria most organizations have been using for dealing with (or avoiding) the issue of technical debt have produced the problems they now face. So, to succeed from this point, whatever criteria they use in the future must be different. My own view is that short-term thinking is at the heart of the problem, but it’s a wicked problem. The long-term solution will not be simple.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Dragičević 2016] Tomislav Dragičević, Xiaonan Lu, Juan C. Vasquez, and Josep M. Guerrero. “DC Microgrids–Part II: A Review of Power Architectures, Applications and Standardization Issues,” IEEE Transactions on Power Electronics, vol 31:5, 3528-3549, 2016.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.

Cited in:

[Meadows 1997] Donella H. Meadows. “Places to Intervene in a System,” Whole Earth, Winter 1997.

Available: here; Retrieved: June 28, 2018

Cited in:

[Meadows 1999] Donella H. Meadows. “Leverage Points: Places to Intervene in a System,” Hartland VT: The Sustainability Institute, 1999.

Available: here; Retrieved: June 2, 2018.

Cited in:

[Meadows 2008] Donella H. Meadows and Diana Wright. Thinking in Systems: A Primer. White River Junction, VT: Chelsea Green Publishing, 2008.

Order from Amazon

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Posted on

Auxiliary technical debt: Rules of engagement

Last updated on July 16th, 2021 at 05:08 pm

Guardrails in a track bed as a rail line crosses a bridge
Guardrails in a track bed as a rail line crosses a bridge. The guardrails are the inner pair of rails. The rails just outside the inner pair are the running rails. Guardrails (also known as check rails) keep the wheels of derailed cars from straying too far from their proper locations. This is a risk mitigation function in high-risk geometries such as curves. It’s also advantageous even if the probability of risk events is low, as in this straight section of track. It’s worthwhile when the consequences of risk events are extremely costly. A derailment on a railway bridge or in steep terrain can result in rail vehicles falling to the earth below. That can cause them to pull other vehicles with them. Derailments under highway overpasses can also be problematic. Such derailments can result in damage to rail or highway bridge structures. That damage can result in service interruptions for periods much longer than the time needed to clear the derailment. For this same reason, guardrails are also used in tunnels and tunnel approaches. Because uncontrolled scope expansion can have such devastating effects, we need policy guardrails to control scope expansion when retiring technical debt from assets that contain auxiliary technical debt.

As noted earlier, a technical debt retirement project (DRP) emphasizes retiring a particular kind or kinds of technical debt from a set of assets. But those assets might also carry other kinds of technical debt. With respect to a given DRP, we can regard this other technical debt as Auxiliary Technical Debt (ATD). Because the presence of ATD can defocus debt retirement projects, it presents a risk we must anticipate and mitigate.

A word of caution

One word of caution. The technical debt discussed in this post is assumed to be localizable. Localizable technical debt is technical debt that manifests itself as discrete chunks. Each instance is self-contained, and we can “point” to it as an instance of the debt in question. More: “Retiring localizable technical debt”

Mitigating the risks associated with the auxiliary technical debt

This post explores concepts and approaches for mitigating the risks associated with the auxiliary technical debt (ATD) of a given debt retirement project (DRP). As might already be evident, these initialisms (ATD, DRP, and one more to come) can be difficult to keep straight. Here’s a quick guide:

  • T always means Technical
  • D always means Debt or Design
  • R always means Retirement
  • P always means Project

So, DRP is Debt Retirement Project.

Also, if you have a pointing device, you can hover the cursor over the first mention of each initialism in each section. Then your browser displays the expansion of the initialism. Touch screen users and keyboarders: sorry, I haven’t yet figured out how to help you in an analogous way, so let me know if you have an idea.

I’ve been using the term TDIQ—Technical Debt In Question—to denote the kinds of technical debt whose retirement is the objective of a given DRP. The ATD of that DRP, then, is the collection of instances of any other kinds of technical debt. That is, all types differing from the TDIQ of the DRP, and which are present in the assets being modified by the DRP. Notice that the property of being auxiliary technical debt is relative. It’s relative to the objectives of a given DRP. A particular instance of technical debt might be ATD for one DRP, and TDIQ for another DRP, depending on the respective objectives of each DRP. Notice also that the ATD of a given DRP can include several different kinds of technical debt.

The temptation to retire auxiliary technical debt

Let’s now examine a scenario in which ATD can generate risk for a DRP. In this scenario, we’ll consider only one kind of ATD; call it ATD0.

Suppose several members of the DRP team undertake work to retire the DRP’s TDIQ in a portion of one of the debt-bearing assets. In performing this work, they encounter some instances of ATD0. Studying these instances of ATD0 carefully, they devise a plan. They realize that “fixing” the ATD0 along with the TDIQ in that portion of the asset would be easier and less risky than amore focused approach. The more focused approach would be to leave the ATD0 in place and attend only to the TDIQ. Let’s call the approach they adopted the ATD approach. And let’s say that the TDIQ approach is one in which the team addresses only the TDIQ. It leaves in place the ATD0 and all other ATD it finds.

Compared to the TDIQ approach, the advantages of the ATD approach are fairly clear. After the work is complete, in either approach, the team must test and re-certify the asset. In the TDIQ approach, when a subsequent DRP tries to retire ATD0, that second DRP team will need to test and re-certify the asset again. In the ATD approach, we can avoid modifying, re-testing, and re-certifying the asset a second time. We can avoid it because we’ve already retired all instances of ATD0 from the asset. Thus, in the ATD approach we can avoid a second round of modification, testing, and re-certification.

Risks associated with retiring auxiliary technical debt

But the ATD approach also has some serious disadvantages.

Enterprise assets might be left in a mixed state

Unless the team plans to retire all instances of ATD0, then upon completion of the DRP, enterprise assets will be in a mixed state. Some will be free of both the TDIQ and ATD0; some will be free of the TDIQ but continue to harbor ATD0. This non-uniformity can create complications for subsequent maintenance, documentation, testing, training, enhancement, automation assisted development, and so on.

Complications in testing and re-certification

If test results for the modified assets indicate the possibility of new defects, the cause might be associated with the TDIQ work, or the ATD work, or both. Resolving any issues in the test results is thus more complicated under the ATD approach than it is under the TDIQ approach. Similar considerations affect re-certification. Thus, there is a risk that the ATD approach will complicate interpretation of test and re-certification results.

Questions about the reliability of technical debt inventory data

As noted in an earlier post, for any given DRP, the DRP team needs to know which assets bear that project’s TDIQ. In the TDIQ approach, any data previously or concurrently gathered about the location of instances of ATD0 remains valid. It maintains its validity because the TDIQ approach doesn’t retire any instances of ATD0. However, in the ATD approach, such inventory data must be corrected to account for the retirement of whatever instances of ATD0 are retired in the ATD approach. Thus, there are problems if ATD0 inventory data has already been collected, or if it’s being collected in parallel with the DRP. The DRP team must then take steps to adjust the inventory data regarding locations of ATD0 as it retires instances of it.

There is one minor exception to this claim about TDIQ’s validity-preserving qualities. In some instances, retiring an instance of TDIQ can incidentally retire an instance of ATD0. It happens.

There is of course a risk that this will not occur as needed. If that risk materializes, it can create problems for any subsequent DRP for which the ATD0 is contained in its TDIQ. This can be especially challenging if there are multiple DRPs in process simultaneously. Suppose each DRP is working on different TDIQs, potentially in different debt-bearing assets. If they all encounter and retire instances of ATD0, keeping the inventory current can be a complicated affair.

Unconstrained scope creep

Suppose there is a DRP whose objective is retiring its TDIQ. And suppose it has decided to also retire instances of a particular kind of ATD, say ATD0. Although that activity would represent an expansion of scope beyond retiring the TDIQ, it might be acceptable and it might even be prudent. But as the team undertakes to retire ATD0, it might confront a similar problem. That problem relates to the relationship between the ATD0 and yet another kind of ATD, which we might call ATD1. The DRP team might then decide to expand scope again. And so on. In general, there is no self-evident stopping point for such a chain of scope expansion. In these circumstances, scope creep can become an unmitigated risk, threatening the coherence and focus of the DRP, with consequences for its budget and schedule.

Last words

In some cases, some of the ATD might be so intertwined with the TDIQ that retiring some instances of the TDIQ necessarily retires some of the ATD. And in other cases, leaving the ATD in place severely complicates retiring the TDIQ. In still other cases, leaving the ATD in place leaves the assets in a complex state that makes ongoing maintenance or enhancement work more difficult. In these cases, what I called the ATD approach above is plainly the wiser course, compared to the TDIQ approach.

Policymakers have a role to play here. They can develop guidance for DRP teams to apply as they come upon these difficult situations. That guidance can help them decide whether to take the ATD approach or the TDIQ approach. The military calls this guidance “rules of engagement,” while politicians call it “guardrails.”

Deciding between the ATD and TDIQ approaches on a whim, or on what feels right at the moment, inevitably leads to a chaos of inconsistency and scope creep. The safest course is to adopt wise rules of engagement. Then adjust them as the organization learns more about retiring technical debt from its assets.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Dragičević 2016] Tomislav Dragičević, Xiaonan Lu, Juan C. Vasquez, and Josep M. Guerrero. “DC Microgrids–Part II: A Review of Power Architectures, Applications and Standardization Issues,” IEEE Transactions on Power Electronics, vol 31:5, 3528-3549, 2016.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.

Cited in:

[Meadows 1997] Donella H. Meadows. “Places to Intervene in a System,” Whole Earth, Winter 1997.

Available: here; Retrieved: June 28, 2018

Cited in:

[Meadows 1999] Donella H. Meadows. “Leverage Points: Places to Intervene in a System,” Hartland VT: The Sustainability Institute, 1999.

Available: here; Retrieved: June 2, 2018.

Cited in:

[Meadows 2008] Donella H. Meadows and Diana Wright. Thinking in Systems: A Primer. White River Junction, VT: Chelsea Green Publishing, 2008.

Order from Amazon

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Posted on

Where is the technical debt?

Last updated on July 15th, 2021 at 10:47 am

Part of the cutting head of an 84-inch (2.13 m) tunnel boring machine. An obsolete sewer line is out of sight to nearly everyone. Invisibility can raise the question, Where is the technical debt?
Part of the cutting head of an 84-inch (2.13 m) tunnel boring machine. The machine was used for installing a sewer in Chicago, Illinois, USA, in 2014. An obsolete sewer line is out of sight to nearly everyone. Invisibility can raise the question, Where is the technical debt? Photo © 2014 by J. Crocker.

When we first set out to plan a large technical debt retirement project (DRP), a question arises very early in the planning process. It is this: Which assets are carrying the kind of technical debt we want to retire? And a second question is: Which operations will be affected—and when—by the debt retirement work? Although these questions are clear, and easily expressed, the answers might not be. And the answers are important. So where is the technical debt?

The challenges of identifying debt-bearing assets

Determining which of the enterprise’s many technological assets might be carrying the Technical Debt In Question (TDIQ) can be a complex exercise in itself. It’s challenging because inspecting the asset might be necessary. Inspection might require temporarily suspending operations, or determining windows of time during which inspection can be performed safely and without interfering with operations. Further, inspection might require knowledge of the asset that the DRP team doesn’t possess. Moreover, access to the asset might be restricted in some way. In these cases, staff from the unit responsible for the asset must be available to assist with the inspection.

Although asset inspection might be necessary or preferable, it might not be sufficient for determining which assets are carrying the TDIQ. This is easy to understand for physical assets. For example, physical inspection cannot determine the release version of the firmware of the hydraulic controller electronics of a tunnel boring machine. But asset inspection might also be insufficient for purely software assets. For determining the presence of the TDIQ in software assets, reading source code might not be sufficient or efficient.

To locate the technical debt, it might be easier, faster, and more accurate to operate the asset under special conditions. For example, an inspector might want to provide specific inputs to various assets and then examine their responses. As a second example, we might use automation assistance to examine the internal structure of an asset, searching for instances of the TDIQ. And as with other assets, the assistance of the staff of the business unit responsible for the asset might be necessary for the inspection.

Which enterprise operations depend on debt-bearing assets?

Knowing which assets bear the TDIQ is useful to the DRP team as it plans the work to retire the TDIQ. But part of that plan could include service disruptions. If so, it’s also necessary to determine how those disruptions might affect operations. That information enables the team to control the effects of the disruptions and negotiate with affected parties. Thus for each asset that bears the TDIQ, we must determine what operations would be affected by service suspension.

Observations of actual operations in conditions in which the asset is out of service in whole or in part can be valuable. Such observations might be the only economical way to discover which enterprise functions depend on the assets that carry the TDIQ. Other techniques include examining historical data such as trouble reports and outstanding defect lists, and correlating them across multiple asset histories and operations histories.

Last words

In some cases, these investigations produce results that have a limited validity lifetime, or “shelf life.” The short shelf life is mainly due to ongoing evolution of the debt-bearing assets and the assets that interact with them. That’s why the work of retiring the TDIQ must begin as soon as possible after the inventory is complete. This suggests that the size of the DRP team is a critical success factor. Larger size teams can complete the inventory inspections rapidly. Speed is important because of the validity lifetime of the team’s research results.

Managing teams of great size is a notoriously difficult problem. One approach that can help involves delegating some of the DRP research effort. The people most qualified for this work are in the business units that own the assets in question. Properly motivated, they can provide the labor hours and expertise needed for the research. In this way, the DRP can deploy a team-of-teams structure, known as a Multi-Team System (MTS) [Mathieu 2001] [Marks 2005]. The DRP team can then bring to bear a large force in a way that renders the overall MTS manageable.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Dragičević 2016] Tomislav Dragičević, Xiaonan Lu, Juan C. Vasquez, and Josep M. Guerrero. “DC Microgrids–Part II: A Review of Power Architectures, Applications and Standardization Issues,” IEEE Transactions on Power Electronics, vol 31:5, 3528-3549, 2016.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.

Cited in:

[Marks 2005] Michelle A. Marks, Leslie A. DeChurch, John E. Mathieu, Frederick J. Panzer, and Alexander Alonso. “Teamwork in multiteam systems,” Journal of Applied Psychology 90:5, 964-971, 2005.

Cited in:

[Mathieu 2001] John E. Mathieu, Michelle A. Marks and Stephen J. Zaccaro. “Multi-team systems”, in Neil Anderson, Deniz S. Ones, Handan Kepir Sinangil, and Chockalingam Viswesvaran, eds., Handbook of Industrial, Work, and Organizational Psychology Volume 2: Organizational Psychology, London: Sage Publications, 2001, 289–313.

Cited in:

[Meadows 1997] Donella H. Meadows. “Places to Intervene in a System,” Whole Earth, Winter 1997.

Available: here; Retrieved: June 28, 2018

Cited in:

[Meadows 1999] Donella H. Meadows. “Leverage Points: Places to Intervene in a System,” Hartland VT: The Sustainability Institute, 1999.

Available: here; Retrieved: June 2, 2018.

Cited in:

[Meadows 2008] Donella H. Meadows and Diana Wright. Thinking in Systems: A Primer. White River Junction, VT: Chelsea Green Publishing, 2008.

Order from Amazon

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Show Buttons
Hide Buttons