Controlling incremental technical debt

Last updated on January 20th, 2019 at 01:54 pm

All technical debt in enterprise assets is either incremental technical debt or legacy technical debt.  Incremental technical debt is technical debt newly incurred. It can be newly incurred exogenous technical debt, or it can be endogenous technical debt incurred either in projects currently underway, or projects just recently completed. Legacy technical debt is technical debt associated with assets, and which wasn’t incurred recently or which exists in any form prior to undertaking work on those assets. All legacy technical debt was at some point incremental technical debt. The vast amounts of legacy technical debt most organizations now carry are nothing more than the accumulation of incremental technical debt. The path to managing legacy technical debt therefore begins with controlling incremental technical debt.

A sinking rowboat
A sinking rowboat provides a useful metaphor for illustrating the effects of incremental technical debt. The enterprise is the rowboat; the leaks are the properties of the enterprise and its environment that lead to creating incremental technical debt; water entering the boat through leaks is incremental technical debt; the accumulated water in the bottom of the boat is legacy technical debt.

Organizations are more likely to gain control of their legacy technical debt portfolio if they begin by controlling the formation of incremental technical debt, and its transformation into legacy technical debt. A metaphor might make this clear:

If you find yourself in a sinking rowboat, bailing out at least some of the water is a good idea, and it might be necessary in the short term. But at some point, fixing the leaks where the water comes in is advisable. Unless you address the leaks that already exist, and prevent new ones from forming as the rowboat ages, your fate is sealed. You’ll spend increasing portions of your time, energy, and resources bailing out your leaky rowboat, and declining portions of your time, energy, and resources rowing the boat towards your objective. And when you do devote some time and energy to rowing towards your objective, you’ll find the rowing surprisingly difficult, because the boat is lower in the water, and because you must propel not only the mass of the boat and its payload, but also the dead weight of the water in the bottom of the boat.

In this metaphor, legacy technical debt is the water in the bottom of the boat, and incremental technical debt is the water coming in through the leaks. The leaks are the proximate “causes” of technical debt. The root causes of the leaks are the root causes of technical debt.

If the enterprise is in the midst of a legacy technical debt emergency, retiring some of it is necessary in the short term. But unless the enterprise addresses incremental technical debt and its root causes, a new burden of legacy technical debt will accumulate. That accumulation is then likely to eliminate the benefits of having retired the current burden of legacy technical debt.

So after the legacy technical debt emergency is passed—or if resources permit, during the emergency—establishing measures, procedures, and practices for controlling incremental technical debt would be prudent.

This change might be less challenging than it sounds. With respect to endogenous incremental technical debt, the teams that incurred it are either still at work, or just recently dispersed. Their understanding of the incremental technical debt is still fresh in their minds. If their projects are still underway, and if budget and schedule permit, retiring the incremental technical debt in the context of those projects is a superior strategy. For projects that have already delivered their work products, a somewhat less preferable—but still practical—approach involves re-assembling some of the team to retire the incremental technical debt as soon as possible, while memories are still fresh. Other approaches might be needed for incremental exogenous technical debt.

For the most part, the problem of controlling incremental technical debt isn’t a technical one. It usually reduces to a problem of finding time and resources to undertake the task.

Why resources aren’t available to retire incremental technical debt

The immediate reason why most teams don’t have enough resources to retire their incremental technical debt is that the organization, as a whole, doesn’t plan for retiring incremental technical debt incrementally. This immediate reason, though, isn’t fundamental. The lack of resources is a symptom of deeper dysfunctions in the organization. The real question is this: Why do so many organizations fail to allocate time and resources to retire incremental technical debt incrementally? Here are three reasons.

Misunderstanding (or no understanding) of the concept of technical debt

The organization is unlikely to be able to manage any kind of technical debt unless its people understand the concept. They must understand that technical debt isn’t necessarily the result of engineering malpractice. Much technical debt arises either as a natural result of working with technology, or as a result of organizational forms that compel people to behave in ways that lead to generating technical debt. Unless the people of the organization accept these truths, allocating sufficient resources to managing incremental technical debt is unlikely.

Not appreciating the MICs

Decisions regarding technical debt management ultimately reduce to a choice between allocating precious resources to technical debt retirement, and allocating them elsewhere. To make this choice responsibly, it’s necessary to fully appreciate the cost of carrying technical debt. Most believe that these costs appear in the form of lost engineering productivity. While that is indeed a factor, other factors can be far more important.

For example, if entry into an important market is delayed by even as little as 30 days due to debt-depressed engineering productivity, the financial consequences can be enormous and insurmountable. Or delays in diagnosing and repairing a fault in a product can produce financial liabilities that can actually sink the company. When one considers all possible financial consequences of carrying technical debt, it becomes clear that managing technical debt effectively is actually a strategy for survival. The decision to allocate appropriate resources to incremental technical debt retirement does require modeling these costs—calculations that few organizations actually undertake.

Miscalculating projections of returns on investments

Failing to estimate MICs with sufficient precision is problematic, as noted, because it reduces the quality of decisions regarding short-term resource allocations. But it also affects long-term projections, which depend on estimating returns on investments. For example, to choose between investing in retiring incremental technical debt from an asset and investing in new capabilities for the same asset, one must compare the projected value of each choice. If the decision-maker’s understanding of the technical debt concept is deficient, or if the calculations of MICs now or in the future are incomplete or underestimated, the investment decision is likely to be biased in favor investing in new capabilities. Incremental technical debt retirement is thereby systematically deferred or avoided altogether.

Policy recommendations for controlling incremental technical debt

The simplistic approach to controlling incremental technical debt is to provide more money to projects and to engineering functions. While that approach will be somewhat helpful, its results will likely be disappointing when compared to approaches that combine resource augmentation with changes in enterprise policy, processes, and culture.

Let’s begin with an example of a needed cultural change. Nearly anyone who makes or influences decisions might occasionally bear some responsibility for incurring incremental technical debt. To achieve effective control of technical debt requires that all such people understand how to change their behavior, whether they’re acting individually or in collaboration with others. Any guiding principle offered to them must be simple to state and easy to understand, because we must communicate it to nearly everyone in the enterprise. Here’s a sample statement of a useful such principle:

Those whose decisions cause the enterprise to incur technical debt are accountable for securing the resources needed to retire that debt, and for supplying compensating resources to those within the enterprise, or among its customers, who suffer depressed operational effectiveness during the period in which that technical debt is outstanding.

I call this the Principle of Accountability. It’s a corollary of what Weinberg calls “Ford’s Fundamental Feedback Formula” [Weinberg 1985], which captures the idea that people make better choices when they must live with the consequences of those choices.

General guiding principles are necessary, but not sufficient. Here are five examples of changes that help in controlling incremental technical debt [Brenner 2017b].

1.       Adopt a shared concept vocabulary

There must be general agreement among all parties about the meanings of concepts that relate to incremental technical debt formation.

Examples: the definitions of “done” vis-à-vis projects, strategic technical debt, reckless technical debt, unethical technical debt, exogenous-technical-debt, endogenous technical debt, MICs, MPrin, and more. An enterprise-wide education program, including on-line reference material and new-employee orientation components, are probably also necessary.

2.       Accept that technical debt is a fact of technological life

There is a widespread belief that most technical debt results from engineering malpractice. Although some technical debt does arise this way, most does not. For examples of other causes, see “Non-technical precursors of non-strategic technical debt.” Some technical debt arises because of advances external to the enterprise, beyond its control. Development-induced or field-revealed discoveries are especially difficult to avoid. In many instances, technical debt is an inevitable result of using technology.

3.       Track the cost of carrying technical debt

The cost of retiring a particular class of technical debt (its MPrin) is significant only in the context of planning or setting priorities for resource allocation. In all other contexts, knowing that cost has little management value. What does matter, at all times, is the cost of carrying that technical debt—the MICs, or metaphorical interest charges. (See “The Principal Principle: Focus on MICs.”) MICs can fluctuate wildly [Garnett 2013].

Build and maintain expertise for estimating and tracking the costs of incurring and carrying each class of technical debt. Know how much each kind of technical debt contributes to these costs, now and for the next few years.

4.       Assign accountability for kinds of technical debt

Some kinds of incremental technical debt result from actions (or inactions) within the enterprise; some do not. To control the kinds of incremental technical debt that arise from internal causes, hold people accountable for the debt their actions generate. Use Fowler’s Technical Debt Quadrant [Fowler 2009] as the basis for assessing and distributing internal financial accountability for debt retirement costs (MPrin) and metaphorical interest charges (MICs). For example, in Fowler’s terms, a Reckless/Inadvertent incident could carry a rating that would lead to imposing higher assessed charges for the debt originators than would a Prudent/Deliberate incident, the charges for which might be zero.

5.       Require that deliberately incurred technical debt be secured

In the financial realm, secured debt is debt for which repayment is guaranteed by specifically pledged assets. By analogy, secured technical debt is technical debt for which resources have been allocated (possibly in a forward time period) to guarantee the debt’s retirement and possibly its associated MICs.

This policy implies that deliberately incurred technical debt, whether incurred strategically or recklessly, must be secured. If anyone involved in a development or maintenance effort feels that technical debt has been incurred, a dispassionate third party, unaligned with any function involved in the effort, reviews project deliverables for the presence of technical debt. However, allocating future resources might require securing commitments of resources for fiscal periods beyond the current one. For many organizations, such forward commitments might require modifying the management accounting system.

Last words

Controlling incremental technical debt requires changes well beyond the behavior and attitudes of engineering staff, or the technologies they employ. Achieving control of incremental technical debt formation requires engagement with enterprise culture to alter the behavior and attitudes of most of the people of enterprise.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Nine indicators of wickedness

Last updated on December 1st, 2018 at 09:04 am

Several properties of the problem of designing technical debt retirement projects tend to make those design problems more likely to be wicked problems—that is, more likely to satisfy all ten of the criteria of Rittel and Webber [Rittel 1973]. I call these properties indicators of wickedness.

The interchange between Interstate 35 and U.S. Route 30, just outside Ames, Iowa
The interchange between Interstate 35 and U.S. Route 30, just outside Ames, Iowa. The new flyover ramp is indicated in red. It replaces the cloverleaf ramp in the upper right quadrant of the cloverleaf. A construction error has forced a delay, while the piers of the flyover ramp bridge are corrected. The cloverleaf, which was designed with curves a bit too tight, was a high-accident area, and thus constituted a technical debt, with the ongoing vehicle accidents comprising the metaphorical interest charges. The construction error in the flyover ramp piers necessitated a rollback. Rollbacks often indicate wickedness in the projects to design technical debt retirement projects, but in this case, the indicated wickedness is not much greater than was anticipated by the project designers. Construction drawing by Iowa Department of Transportation  [Iowa DOT 2016].
Although we usually have some notion of the degree of wickedness of a given design effort for a technical debt retirement project, actually executing the debt retirement project can reveal unanticipated issues and complexity. Some of what’s revealed can cause us to adjust our estimate of the degree of wickedness of the design effort. If we know in advance what kinds of revelations are most likely to cause such adjustments, we can reduce the incidence of unanticipated revelations.

As I noted in my post, “Degrees of wickedness,” we can regard all problems as lying on a Tame/Wicked spectrum, with wicked problems lying at the extreme Wicked end of the spectrum, and the tamest of the tame lying at the opposite end. As for the ten criteria of wickedness developed by Rittel and Webber, I proposed that they could be satisfied in degrees, with the most wicked problems satisfying all ten criteria absolutely.

As a quick review, here are the attributes of wicked problems as Rittel and Webber see them [Rittel 1973], rephrased for brevity:

  1. There is no clear problem statement
  2. There’s no way to tell when you’ve “solved” it
  3. Solutions aren’t right/wrong, but good/bad
  4. There’s no ultimate test of a solution
  5. You can’t learn by trial-and-error
  6. There’s no way to describe the set of possible solutions
  7. Every problem is unique
  8. Every problem can be seen as a symptom of another problem
  9. How you explain the problem determines what solutions you investigate
  10. The planner (or designer) is accountable for the consequences of trying a solution

Below is a sample of conditions or situations that tend to increase the wickedness of the problem of designing a technical debt retirement project. I have no data to support these conjectured effects. But the principles I used to generate them are three:

  • If a phenomenon expands the set of stakeholders in a debt retirement project, it tends to enhance the wickedness of the design problem.
  • If a phenomenon increases the number or heterogeneity of the assets or processes that must be considered, it tends to enhance the wickedness of the design problem.
  • If a phenomenon creates a need for a rollback of work performed as part of the debt retirement project, and that rollback creates a need to re-design the debt retirement project, it tends thereby to enhance the wickedness of the design problem.

In what follows, I use the term “DRP” to indicate the Debt Retirement Project itself, as distinguished from the effort to design the DRP, which I refer to as “DDRP.” The problem whose wickedness we’re considering is not the DRP itself, but the DDRP. Also, let DBA (for debt-bearing assets) be the set of assets undergoing modification in the context of the DRP, and let IA (for interacting assets)  be the set of assets, excluding the DBA assets, that interact directly or indirectly with assets in the DBA.

With all this in mind, I offer the following nine examples of indicators of wickedness of the DDRP.

1.   A previous attempt to retire this debt was abandoned

Perhaps the most significant indicator of the wickedness of the DDRP is either the failure of a previous attempt to execute a DRP with similar objectives, or the failure of a previous attempt to execute a DDRP for a DRP with those objectives. There are two reasons why such failures are significant indicators of wickedness.

First, it’s reasonable to assume that these previous attempts weren’t founded on any recognition of the wickedness of the DRP or the DDRP. Few such efforts are.  (A Google search for the two phrases “technical debt” and “wicked problem” yields less than 1000 results) (update 12 Nov 2018: 1160 results) Consider first the DDRP. If it is a wicked problem, proceeding as if it were not would very likely fail. If the designers of the previous DDRP did assume that it was a wicked problem, investigating their approach could prove invaluable, and save much time and effort. An analogous argument applies for the DRP itself.

Second, if the previous attempt to execute a DRP with similar objectives has left traces of itself in the DBA, and if those traces must be taken into account while executing the DDRP, they might complicate the DRP, and they might be incompletely addressed in the DDRP. To the extent that these conditions prevail, Criterion 5 is satisfied, and the DDRP exhibits wickedness.

2.   Some revenue streams need to be interrupted

If the work of the DRP entails temporary interruption of revenue streams, executing the DRP can have significant and long lasting effects on the organization. In estimating the cost of the DRP, it’s clearly necessary to account for the financial impact of any revenue shifted into the future, and any revenue irretrievably lost as well. And in some cases, market share might also suffer. All of these factors tend to increase the wickedness of the DDRP.

When these effects are expected, political opposition to the DRP can develop. Senior management can prevent this opposition from halting the DDRP inappropriately by requiring that the business case for the DRP include these financial factors and demonstrate clearly the need to proceed despite them. Involving potential political opponents of the DRP in business case development can be an effective means of ensuring the strength of the business case.

The ability to model all these financial effects is an important organizational asset that can be developed and maintained, for deployment across multiple DDRPs. The organization can monitor DRPs, gathering actual experience data for comparison to the effects projected in the respective business cases of the DRPs. Those comparisons are useful for enhancing the modeling capability.

3.   Some assets not directly touched need to be re-tested

The DDRP is more likely to be a wicked problem if, as a result of the changes executed in the DRP, any of the assets in IA need to be re-tested after or during DRP execution. The need to re-test any assets in IA typically arises when there’s some risk that the DRP’s changes in the DBA could somehow affect the performance of the assets in IA, and when the consequences of such a risk event are severe.

This scenario enhances of the wickedness of the DDRP for at least five possible reasons.

  • Baseline testing of IA is necessary to enable the DRP team to recognize the effects of the DRP on IA behavior. But this baseline testing can reveal pre-existing and unaddressed faults. Because leaving those faults in place can seriously complicate interpretation of anomalies that appear in IA assets after DRP work has begun, the DDRP team might insist that the owners of the IA assets in question address some of these faults. With regard to these issues, political differences between the DDRP team and the owners of IA assets are possible.
  • The additional testing of IA assets tends to expand dramatically the set of stakeholders affected by the DRP, to include the owners, users, and maintainers of the IA assets.
  • The additional testing of IA assets can increase the need to interrupt revenue streams temporarily, and increase the number, duration, and frequency of such interruptions.
  • The additional testing of IA assets can require expertise and staffing beyond the DRP project team, which can disrupt other elements of the organization as the people needed are temporarily assigned to IA testing.
  • The additional testing of IA assets can reveal unanticipated consequences of the DRP alterations, which can trigger re-planning or re-design of the DRP during its execution. That re-planning or re-design, in turn, can trigger alterations in the DDRP.

The need to re-test assets not directly touched in the DRP is more likely when the DRP alters the external behavior of any of the DBA assets. The goal of many DRPs is improvement of the internals of assets without altering their external behavior, except possibly for performance improvements. This goal is desirable because it limits the need for re-testing and re-certification of IA assets. However, some kinds of technical debt appear in the externals of the DBA assets, including their architecture, behavior, appearance, or interfaces. Compared to DRPs that do not alter the externals of DBA assets, retiring the kinds of technical debt that alter the externals of DBA assets is inherently more difficult and more risky because it requires more extensive re-testing and re-certification of both DBA and IA assets.

4.   Multiple sites are directly touched

When the DRP entails modification of technological assets of geographically dispersed organizations, one consequence is a tendency to increase the wickedness of the DDRP. This comes about because of factors including the following:

  • Sites might be dispersed not only geographically, but they might also be separated by language boundaries, legal jurisdictions, cultural divides, time zones, and much more. The required work can vary from site to site for technical reasons and because of variations arising from these non-technical factors.
  • The multiple sites might have different landlords, with different lease agreements governing the organization’s occupancy of the property. This is just one of many factors that increase the numbers of stakeholders and exacerbate their heterogeneity. And the leases might constrain the kind of work that can be performed according to the day of the week or time of day.
  • If local vendors provide services such as communications or Internet connections to some of the sites, and if the work of the DRP involves these technologies and the local vendors, the task of coordinating all the different players can be complex and riddled with unanticipated obstacles.

For example, if the work involves networking hardware and software, work that we might prefer to perform at night or on a weekend might need to be carried out during business hours for some of the sites. For a global enterprise, there might not be a time of day when no sites are conducting regular business.

As a second example, consider a network upgrade for the retail branch offices of a global bank. If that upgrade requires trenching for new cable connections, the project design must take into account local regulations governing the trenches, including how they must be permitted, dug, re-filled, covered, and marked while still open. These regulations vary with national and sometimes local jurisdiction. The complexity causes most organizations to rely on local vendors, but even then, the vendor selection process must include reliable vendor assessment and evaluation. Scheduling becomes a complex and risky endeavor.

For these reasons, a DDRP that involves technological assets housed at multiple sites geographically dispersed has an elevated probability of exhibiting the properties of a wicked problem.

5.   Government agencies and/or industrial standards organizations must re-certify assets

Another driver of stakeholder expansion is the need for re-certification of assets after they’ve been modified in the course of executing the DRP. The certification agencies can range from local and municipal regulators to national regulators and pan-industrial standards organizations. The sheer number of possibilities is itself a contributor to increased wickedness, but the nature of the operating style of these organizations merits special notice.

For the most part, these agencies operate without competitors, whether they are government elements or private organizations. Perhaps for this reason, “customer service” might not be their strength, and gaining timely cooperation from them might be a challenging undertaking. Even though re-certification might be a small part of the DRP, it can easily become a blocking obstacle. Researching these requirements and their associated lead times, and maintaining a current knowledge base about them, can be a non-trivial task of the DDRP.

6.   Non-technical stakeholders must change their behavior

Generally, people don’t like to change how they do their jobs. There are exceptions, of course, if they recognize a benefit that arrives in some immediate and visible way. But unless there is a recognizable benefit, requiring people to change their work patterns as part of a DRP is likely to increase the wickedness of the DDRP. And the difficulty is more problematic if the people affected are technically unsophisticated, because they’re less likely to appreciate the value of managing technical debt, and less likely to accept explanations of that value when those explanations are offered.

DDRP wickedness increases in this case because, in addition to retiring the technical debt, it must address the tasks of motivating and training the affected population. That requires preparing materials, scheduling and accounting for the time spent in training, and monitoring training effectiveness. The business case must also address these issues, but in addition, it must provide evidence required to defuse any political opposition that might otherwise develop.

7.   Discovery of major unanticipated complexity triggers re-design

Unanticipated complexity happens in almost every project of almost any kind. But for DDRPs, unanticipated complexity significant enough to trigger significant adjustment or re-design of the DRP during execution of the DRP is another matter. Such a discovery can mean that the DBA assets or their connections to the IA assets have changed since the plan was developed. Or it can mean that the design team had an incomplete or incorrect understanding of the problem they were trying to solve. These events can occur for a number of reasons.

Technical causes are perhaps more easily imagined, so I’ll focus on non-technical causes, which can actually be more serious. For example, suppose that a political alliance enabled the VP of Sales and the VP of Engineering to reach a deal that enabled the DRP team to work on some DBA assets critical to the Sales function, by taking them off line for defined periods. If that political alliance weakens, or if the deal between the two VPs collapses for some other reason, the scheduled downtime of those assets might vanish or be shortened dramatically. This pattern is more likely to arise in situations in which the DDRP team is not a party to such agreements. The DDRP team must be a party to any agreements regarding access to assets by the DRP team.

As a second example, consider what happens when the enterprise undertakes an acquisition that wasn’t revealed to the DDRP team during their design effort. Because chances are good that the DDRP would have a significant amount of re-work to do in such cases, the DDRP team must be kept informed of any organizational changes that could affect the DRP, for the active life of the DRP.

Re-designing the DDRP can take time. Elements of the DDRP that have short shelf lives must be revisited during re-design. And the need to re-design can also indicate gaps in the DDRP team’s understanding of the problem.

All of these conditions tend to move the DDRP in the direction of increased wickedness.

8.   Weekend or middle-of-the-night work periods are required

The need to perform critical operations on weekends or in nighttime hours suggests three things. First, the work is risky in the sense that undetected faults that go into production can lead to costly operational errors. Second, the organization lacks a simulated operating environment that emulates the actual operating environment faithfully enough to enable detection of errors before deployment. Third, and finally, the organization lacks a rapid rollback mechanism that can restore the original state of an asset if the new modified state proves problematic when deployed.

These last two factors—the lack of a simulated operating environment and the lack of a rapid rollback mechanism—should be corrected if multiple DRPs are anticipated. Cost is usually the blocking issue. However, that cost must be compared against the cost of retarding all future DRPs, and the cost of any operational failures arising from deploying faulty systems.

Continued refusal to provide a simulated operating environment with rapid rollback increases the wickedness of this and any future DDRPs.

9.   Rollback(s) of attempted changes triggers re-design

A rollback is an incident in which, in the course of executing the DRP, it has become necessary to revert some (or all) of the work that has already been performed. Minor rollbacks do happen. But a major rollback, or a rollback the necessity of which is discovered long after completion of the work in question, could be an indication of a deep misunderstanding of the consequences of the work involved. Because that misunderstanding could have consequences not yet recognized, such a rollback could suggest that the wickedness if the DDRP had been underestimated.

Let DBAf (faulty DBA) be the set of assets in the DBA that formerly contained some of the debt being retired, and which had been altered as part of the DRP, and whose alterations contained or led to exposure of some kind of fault(s) that forced a rollback after they were deployed. Let DBAfw (wicked-faulty DBA) represent the subset of DBAf for which that rollback did trigger a re-design of the DDRP. Then wickedness of the DDRP is correlated with the size of DBAfw and the extent of the DDRP re-design that the rollback triggered.

For example, let Efw be a member of DBAfw. And suppose that Efw is a modular element of a system that monitors the click-through behavior of users of a Web site. It records data for later analysis, and because of the fault it does so incorrectly. When the errors are discovered, the module is withdrawn and replaced by the original, unaltered, debt-bearing form. Because Efw contaminated the original database, data rollback is impossible. The error was discovered, but there is no way to re-capture the data that has been lost. That’s why the DDRP (and after that, possibly the DRP) must be re-designed. This scenario is an example of Criterion 5. If there are political consequences for the loss of data, it could be an example of Criterion 10.

This example suggests how the frequency of incidents that trigger re-design of the DDRP can be an indicator of the wickedness of the DDRP.

Example of a non-indicator: the I-35 SR-30 interchange near Ames, Iowa

Just outside Ames, Iowa, is an interchange between Interstate 35 (a four-lane, divided, limited-access roadway) and U.S. Route 30 (also four-lane, divided, but not limited-access). The interchange is a conventional cloverleaf design. The “leaves” are rather tight, though, and consequently, there have been numerous rollovers and crashes at this interchange. We can regard these tight cloverleaf ramps as technical debt in the highway system, and the rollovers and crashes as metaphorical interest charges on that debt.

In 2016, construction began on a new “flyover” exit ramp from northbound Interstate 35 onto westbound U.S. Route 30. The objective was to reduce the number of accidents at the interchange by replacing the current tight-curvature cloverleaf ramp with a flyover exit ramp with a longer radius of curvature. We can regard this project as a Debt Retirement Project (DRP). The project that planned that DRP was an effort to Design a Debt Retirement Project (DDRP).

Completion of the DRP was scheduled for November 2018. When completed, the new ramp will replace the northeast leaf of the cloverleaf. Like most civil engineering projects, this project does have some elements of wickedness, but they were dealt with effectively. Nevertheless, a construction error is delaying completion [Magel 2018] [Iowa DOT 2018]. The error involves the height and position of the bolt anchors where steel bridge beams will connect to the concrete piers of the new flyover ramp. Six piers have been constructed to support the flyover. Those piers are being corrected by jackhammering the concrete tops, leaving the steel reinforcement in place. Then the beam anchors are positioned correctly, and concrete re-poured. The length of the delay in completion hasn’t yet been announced.

This effort, which constitutes a rollback and re-deployment, is a significant project in itself. It requires scheduling the work to be performed, but it also requires scheduling highway lane closures and lane shifts, working around high-volume traffic periods, and possibly pouring concrete in winter conditions. And after the piers are corrected, the bridge beam placement and bridge roadbed work must proceed on a new schedule.

Consequently, the construction error triggered a redesign of the flyover project’s DRP. But it probably did not trigger a significant re-design of the DDRP. The construction error is therefore unlikely to be an indicator of significant additional wickedness for the DDRP.

Last words

You can become better managers of the risk of unanticipated wickedness. If your organization is embarking upon a long-term program of technical debt retirement, you’ll be executing many DDRPs and DRPs. Gathering data about incidents of unanticipated wickedness in DDRPs can be a useful practice, if you use that data when you design new technical debt retirement projects.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Accounting for technical debt

Last updated on September 16th, 2018 at 03:28 pm

With all the talk of technical debt these days, it’s a bit puzzling why there’s so little talk in the financial community about how to go about accounting for technical debt [Conroy 2012]. Perhaps one reason for this is the social gulf that exists between the financial community and the community most keenly aware of the effects of technical debt — the technologists. But another possibility is the variety of mechanisms compelling technologists to leave technical debt in place and move on to other tasks.

Accounting for technical debt isn’t the same as measuring it
Accounting for technical debt isn’t the same as measuring it. We usually regard our accounting system as a way of measuring and tracking the financial attributes of the enterprise. As such, we think of those financial attributes as representations of money. Technical debt is different. It isn’t real, and it isn’t a representation of money — it’s a representation of resources. Money is just one of those resources. Money is required to retire technical debt, and money is consumed when we carry technical debt, but other kinds of resources are also required. Sometimes we forget that when we account for technical debt.

Here’s an example. One common form of technical debt is the kind first described by Cunningham [Cunningham 1992]. Essentially, when we complete a project, we often find that we’ve advanced our understanding of what was actually needed to accomplish our goals. And we’ve advanced our understanding to such an extent that we recognize that we should have taken a different approach. Fowler described this kind of technical debt as, “Now we know how we should have done it.” [Fowler 2009] At this point, typically, we disband the team and move on to other things, leaving the technical debt outstanding, and often, undocumented and soon to be forgotten.

A (potentially) lower-cost approach involves immediate retirement of the debt and a re-release of the asset — an “echo release” — in which the asset no longer carries the technical debt we just incurred and immediately retired. But because echo releases usually offer no immediate, evident advantage to the people and assets that interact with the asset in question, decision-makers have difficulty allocating resources to echo releases.

This problem is actually due, in part, to the effects of a shortcoming in management accounting systems. Most enterprise management accounting systems track effectively the immediate costs associated with technical debt retirement projects. They do a much less effective job of representing the effects of failing to execute echo releases, or failing to execute debt retirement projects in general. The probable cause of this deficiency is the distributed nature of the MICs — the metaphorical interest charges associated with carrying a particular technical debt. The MICs appear in multiple forms: lower productivity, increased time-to-market, loss of market share, elevated voluntary turnover in the ranks of technologists, and more (see “MICs on technical debt can be difficult to measure”). These phenomena are poorly represented in enterprise accounting systems.

Decision-makers then adopt the same bias that afflicts the accounting system. In their deliberations regarding resource allocation, they emphasize only the cost of debt retirement, often omitting from consideration altogether any mention of the cost of not retiring the debt, which can be ongoing.

If we do make long-term or intermediate-term projections of costs related to carrying technical debt or costs related to debt retirement releases, we do so in the context of a cost/benefit computation as part of the proposal to retire the debt. Methods vary from proposal to proposal. Many organizations lack a standard method for making these projections. And because there’s no standard method for estimating costs, comparing the benefits of different debt retirement proposals is difficult. This ambiguity and variability further encourages decision-makers to base their decisions solely on current costs, omitting consideration of projected future benefits.

Dealing with accounting for technical debt

Relative to technical debt, the accounting practice perhaps most notable for its absence is accounting for outstanding technical debts as liabilities. We do recognize outstanding financial debt as such, but few balance sheets — even those for internal use only — mention outstanding technical debt. Ignorance of the liabilities imposed by outstanding technical debt can cause decision-makers to believe that the enterprise has capacity and resources that it doesn’t actually have. Many of the problems associated with high levels of technical debt would be alleviated more readily if we began to track our technical debts as liabilities — even if we did so for internal purposes only.

But other shortcomings in accounting practices can create additional problems almost as severe.

Addressing the technical-debt-related shortcomings of accounting systems requires adopting enterprise-wide patterns for proposals, which gives decision-makers meaningful comparisons between different technical debt retirement options and between technical debt retirement options and development or maintenance options. One area merits focused and immediate attention: estimating MPrin and estimating MICs.

Standards for estimating MPrin are essential for estimating the cost of retiring technical debt. Likewise, standards for estimating MICs, at least in the short term, are essential for estimating the cost of not retiring technical debt. Because both MPrin and MICs can include contributions from almost any enterprise component, merely determining where to look for contributions to MPrin or MICs can be a complex task. So developing a checklist of potential contributions can help proposal writers develop a more complete and consistent picture of the MICs or MPrin associated with a technical debt. Below are three suggestions of broad areas worthy of close examination.

Revenue stream disruption

Technical debt can disrupt revenue streams either in the course of being retired, or when defects in production systems need attention. When those systems are taken out of production for repairs or testing, revenue capture might undergo short disruptions. Burdens of technical debt can extend those disruptions, or increase their frequency.

For example, a technical debt consisting of the absence of an automated test can lengthen a disruption while the system undergoes manual tests. A technical debt consisting of a misalignment between a testing environment and the production environment can allow defects in a repair or enhancement to slip through, creating new disruptions as those new defects get attended to. Even a short disruption of a high-volume revenue stream can be expensive.

Some associations between classes of technical debt and certain revenue streams can be discovered and defined in advance of any debt retirement effort. This knowledge is helpful in estimating the contributions to MICs or MPrin from revenue stream disruption.

Extended time-to-market

Although technologists are keenly aware of productivity effects of technical debt, these effects can be small compared to the costs of extended time-to-market. In the presence of outstanding technical debt, time-to-market expands not only as a result of productivity reduction, but also from resource shortages and resource contention. Extended time-to-market can lead to delays in realizing revenue potential, and persistent and  irreparable reductions in market share. To facilitate comparisons between different technical debt retirement proposals, estimates of these effects should follow standard patterns.

Data flow disruption

All data flow disruptions are not created equal. Some data flow processes can detect their own disruptions and backfill when necessary. For these flows, the main consequence that might contribute to MICs or MPrin is delay. And the most expensive of these are delays in receipt of orders, delays in processing orders, and delays in responding to anomalous conditions. Data flows that cannot detect disruptions are usually less critical, but they nevertheless have costs too. All of these consequences can be modeled and estimated, and we can develop standard packages for doing so that we can apply repeatedly to MICs or MPrin estimates for different kinds of technical debt.

Last words

Estimates of MICs or MPrin are helpful in estimating the costs of retiring technical debt. They’re also helpful in estimating the costs of not retiring technical debt. In either case, they’re only estimates, and as such, they have error bars and confidence limits. The accounting systems we now use have no error bars. That, too, is a shortcoming that must be addressed.

References

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

The Principal Principle: Focus on MICs

Last updated on December 11th, 2018 at 06:56 am

When some organizations first realize that their performance is being limited by technical debt, they begin by chartering a “technical debt inventory.” Their goals are to determine just how much technical debt they’re carrying, where it is, how much retiring it all will cost, and how fast they can retire it. That’s understandable. It’s not too different from how one would approach an out-of-control financial debt situation. Understandable, but in most cases, ineffective. With technical debt we need a different approach, because technical debt is different from financial debt. With technical debt, we must be guided by what I call the Principal Principle, which is:

The Metaphorical Principal (MPrin) of a technical debt, which is the cost of retiring it, isn’t what matters most. What usually matters most is the Metaphorical Interest Charges—the MICs.

The door to a bank vault
The door to a bank vault. One way to know that technical debt differs from financial debt is that banks aren’t involved in any way. Treating technical debt as if it had anything in common with financial debt—beyond our own sense of obligation—is a shortcut to real trouble. Remember the Principal Principle.

With technical debt, MICs can vary dramatically. For assets that aren’t being maintained or enhanced, the MICs can be Zero for extended periods. For retiring assets, their technical debt can vanish when the asset is retired. For other assets, MICs can be dramatically higher—beyond the total cost of replacing the asset.

Most people regard MICs as being restricted to productivity problems among engineers. I take a different approach. I include in MICs anything that depresses net income—lost or delayed revenue, increased expenses, anything. For instance, if technical debt causes a two-month delay in reaching a market, its effect on revenues can be substantial for years to come. I regard all of that total effect as contributing to MICs.

So the Principal Principle is that a focus on Principal can be your undoing. Focus on MICs. Drive them to Zero and keep them there.

Other posts in this thread

Technical debt use disorder

The American Psychiatric Association has identified a disorder they call Substance Use Disorder (SUD), which includes alcoholism, drug addiction, and other patterns related to substance use [APA 2013]. The research they’ve done can serve well as a model for understanding organizational behavior related to technical debt. In this post I’ll show how to use that model to describe a disorder of organizations that we could call Technical Debt Use Disorder (TDUD). It’s a pattern in which the organization can’t seem to retire much of its technical debt, or can’t stop incurring increasing burdens of technical debt, even though everyone in the organization realizes—or almost everyone realizes—that technical debt is harming the organization.

A ball and chain, with shackle
A ball and chain, with shackle.  Attaching this device to the legs of prisoners or slaves limits their ability to run, just as technical debt limits the ability of organizations to exploit new opportunities, or even to maintain their current market positions.

A brief description of the American Psychiatric Association’s publication, DSM-5®, might help explain the connection between SUD and TDUD. DSM-5 is the fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM). It’s used by health care professionals in the United States and much of the rest of the world as a guide to diagnosing mental disorders, and as a framework for further research. First published in 1952, the current revision, DSM-5, was released in 2013.

So what does DSM-5 have to do with technical debt?

What distinguishes responsible use of technical debt from irresponsible use is a topic that has generated many papers, conference presentations, and hallway debates over the years. Although there is consensus about the distinction in many cases, the debate continues. Sometimes, though, research in one field suggests paths forward in seemingly unrelated fields. So much thought and study has been invested in DSM-5 that it’s worth a look to see if the technical debt community can harvest something useful from the research in psychiatry.

I looked at DSM because I noticed that organizations that carry significant volumes of technical debt seem to have difficulty retiring it. In some cases, they also have difficulty halting accumulation of technical debt, or even slowing the rate of accumulation. This struck me as similar to the substance use problems some people encounter. I began to wonder whether there might be parallels between the substance use disorders that afflict people—alcoholism, drug addiction, and so on—and the technical debt problems that afflict organizations.

In the table below is one set of parallels I’ve found. The left column of the table is the list of diagnostic criteria for Substance Use Disorder provided in the DSM. The right column is my rewording of those criteria in an attempt to make them apply to how organizations deal with technical debt. I had thought initially that the rewording exercise might be difficult—that it might be a stretch. And here and there, it was a bit of a stretch. But overall, the SUD framework is a very good fit.

Diagnosing technical debt use disorder

Have a look at the table, and then check out the comments below it about how health care professions use the criteria.

DSM-5 Criteria for Substance Use Disorder (SUD)Criteria for Enterprise Technical Debt Use Disorder (TDUD)
1. Taking the substance in larger amounts or for longer than you’re meant to.
1. Incurring technical debt in larger amounts than you intended and carrying it for longer than you intended.
2. Wanting to cut down or stop using the substance but not managing to.2. Wanting to retire your technical debt or reduce the rate of incurring it but not managing to.
3. Spending a lot of time getting, using, or recovering from use of the substance.3. Spending a lot of time dealing with the consequences of the technical debt you’ve already incurred.
4. Cravings and urges to use the substance.4. Insistent demands on precious resources, causing the enterprise to incur “just a little more” technical debt.
5. Not managing to do what you should at work, home, or school because of substance use.5. Not managing to attend to the needs of existing products, services, or technological infra­structure because of the demands resulting from metaphorical interest charges on technical debt.
6. Continuing to use, even when it causes problems in relationships.6. Continuing to carry technical debt, or continuing to incur yet more technical debt, even though it causes toxic conflict among employees, and problems in customer relationships and strategic partnerships.
7. Giving up important social, occupational, or recreational activities because of substance use.7. Giving up developing important new products or services, or upgrading critical infrastructure, or pursuing new initiatives because of resource deficits traceable to technical debt service.
8. Using substances again and again, even when it puts you in danger.8. Incurring technical debt again and again, even when it puts the enterprise in fiscal danger or danger of losing market position.
9. Continuing to use, even when you know you have a physical or psychological problem that could have been caused or made worse by the substance.9. Continuing to incur technical debt, even when you know you have a fiscal or market leadership problem that could have been caused or made worse by technical debt.
10. Needing more of the substance to get the effect you want (tolerance).10. Needing to incur more technical debt to get the fiscal effect you need—a product delivered or a contract completed.
11. Development of withdrawal symptoms, which can be relieved by taking more of the substance.11. Upon attempting to retire existing technical debt, or halting incurring yet more technical debt, fiscal or market position problems develop in short order, which can be relieved only by incurring yet more debt.

In health care, two or three symptoms indicate a mild substance use disorder; four or five symptoms indicate a moderate substance use disorder, and six or more symptoms indicate a severe substance use disorder. Have a look at the right-hand column. How would you score your organization? Can we categorize the severity of an organization’s problem with technical debt using a scale similar to the one health care professionals use for SUD?

Conclusion

Technical debt isn’t inherently evil. Its existence among technological assets isn’t proof of engineering malpractice. For example, we can decide—responsibly—to deliver a system that carries technical debt, provided that we’ve carefully weighed the consequences of incurring that debt against the consequences of delayed delivery, and provided that we have a workable plan for retiring that debt, or for carrying the burden of its MICs.

But organizations can nevertheless become trapped in a cycle of technical debt, unable to make much progress in reducing it. In some cases, business as usual won’t work. Drastic action is required.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Using SMART goals for technical debt reduction

Last updated on December 24th, 2018 at 01:42 pm

Attempting to reduce the enterprise burden of technical debt by setting so-called “SMART goals” in the obvious way can often produce disappointing results. SMART, due to George T. Doran [Doran 1981], is a widely used approach for expressing management goals. “SMART” is an acronym for “Specific, Measurable, Attainable, Realistic, and Time-boxed,” though the last three words (the “ART”) are chosen in various alternative ways. Doran himself used “assignable, realistic, and time-related.”

Prof. George T. Doran (1939-2011), creator of the S.M.A.R.T acronym for setting management objectives
Prof. George T. Doran (1939-2011), creator of the S.M.A.R.T acronym for setting management objectives. Watch a 2010 interview of Prof. Doran at YouTube.
SMART is so embedded in management culture that many assume without investigation that expressing technical debt reduction goals directly using the SMART pattern will produce the desired results. Also embedded in management culture is the aphorism, “You get what you measure.” [Ariely 2010]  [Bouwers 2010] Combining these two ideas in a straightforward way, one might express the technical debt reduction goal as, “Reduce the burden of technical debt by 20% per year for each of the next five years.”

There is ample support for a claim that this “direct” approach to applying the SMART technique will be ineffective. The fundamental issue is that so much of employee behavior affects technical debt indirectly that it overwhelms the effects of employee behaviors that affect technical debt directly. The result is that although the direct approach does cause some employees to adopt desirable behaviors, their impact is not significant enough compared to the effects of the behaviors of employees who see little connection between their own activities and the burden of technical debt, or who are subject to competing constraints on their behaviors that then cause them to act in ways that increase technical debt.

That’s why it’s necessary for management to develop a series of SMART goals that affect behaviors that have indirect effects on technical debt. In the first part of this post, “Setting a direct SMART goal for technical debt reduction is problematic,” I explore the problems inherent in the direct approach. In the second part, “How to set SMART goals for technical debt,” I provide examples of SMART goals that touch on behaviors that have indirect effects on technical debt.

Setting a direct SMART goal for technical debt reduction is problematic

Let’s begin by exploring some of the problems with the direct approach. In this section, I assume that management has set a SMART goal for the enterprise in the form, “Reduce the burden of technical debt by 20% per year for each of the next five years.” But there’s nothing special about the numbers. My comments below apply to the form of the goal, rather than the specific numbers.

The direct approach assumes measurability

To attain a goal of a 20% reduction in technical debt in a given year, we must be able to measure the level of technical debt at the beginning of the year and the level at the end of the year, presumably with confidence in the 90% range or better. Such a measurement with the precision required might not be possible. Moreover, in most cases the probability that such a measurement is possible is low. For these reasons, periodic goals for total technical debt is not a useful management tool.

Consider a simple example. One form of technical debt—and it’s a common form—is missing or incompletely implemented capability. In some instances, the metaphorical principal (MPrin) of a given instance of this debt in the current year can change spontaneously to a dramatically larger value in the following year (or even the following week), due to changes in the underlying asset unrelated to the technical debt, or due to debt contagion, or due to any number of other reasons. When this happens, the technical debt retirement effort for that year can appear to have suffered a serious setback, even though the technical debt retirement teams might have been performing perfectly well.

The direct approach assumes a static principal

With most financial debts, the principal amount is specified at the time of loan origination. Moreover, we can compute the principal at any time given the mathematical formulas specified in the loan agreement.

By contrast, in many cases, the metaphorical principal amount of a technical debt might be neither fixed nor readily computable. We can estimate the MPrin of a given kind of technical debt at a given time, and we can even make forward projections of those estimates. But they are merely estimates, and their error bars can be enormous. See “Policy implications of the properties of MPrin” and “Useful projections of MPrin might not be attainable.”

The direct approach focuses on MPrin, not MICs

Objectives expressed in terms of the volume of technical debt—the total MPrin—are at risk of missing the point. Total MPrin is not what matters most. What matters is MICs—the total cost of carrying the debt. Even more important is the timing of arrival of the MICs.

And like MPrin, MICs can vary in wild and unpredictable ways. For example, the MICs for a piece of technical debt borne by an asset that isn’t undergoing maintenance or enhancement can be zero; in a later time period, when that asset is undergoing enhancement, the MICs can be very high indeed. See  “MICs on technical debt can be unpredictable” for a detailed discussion.

Priority setting for technical debt retirement is most effective when it takes into account the timing of MICs. For example, if we know that we must enhance a particular asset by FY21 Q3, and if we know that it bears technical debt that adds to the cost of the enhancement, retiring that debt in FY20 would be advisable. On the other hand, if that form of technical debt has no effect on MICs for the foreseeable future, retiring that technical debt might not be worth the effort.

The direct approach fails to distinguish legacy technical debt from incremental technical debt

Unless policies are already in place governing the formation of new technical debt—what I call incremental technical debt—technical debt retirement programs might encounter severe difficulty meeting their goals. The technical debt retirement program might simply be unable to keep up with the formation of new technical debt resulting from new development or from ongoing maintenance and enhancement of existing assets.

The direct approach fails to anticipate the formation of enterprise-exogenous technical debt

Technical debt can sometimes form as a result of innovations, changes in standards, or changes in regulations that occur entirely external to the enterprise. I call such technical debt enterprise-exogenous. When this happens, the technical debt retirement effort can appear to have suffered a serious setback, even though the technical debt retirement teams might have been performing perfectly well. Before initiating a technical debt reduction program, it’s wise to first deploy a program that’s capable of retiring technical debt at a pace that at least equals the pace of formation of enterprise-exogenous technical debt.

Incurring technical debt is sometimes the responsible thing to do

At times, incurring technical debt is prudent. In some situations, accepting the debt you’ve incurred—even for the long term—might be called for. Because strict goals about total technical debt can lead to reluctance to incur debt that has a legitimate  business purpose, whatever goals are set for total technical debt must be nuanced enough to deal with these situations. Goals for total technical debt that adhere strictly to the SMART goal pattern sometimes lack the necessary level of nuance.

How to set SMART goals for technical debt

SMART goals can work for technical debt management, but we must express them in ways that are more closely related to behavioral choices. Here are some examples of SMART goals that can be effective elements of the technical debt management program. Some of these examples are admittedly incomplete. For example,  I offer no proof of assignability, attainability, or realism, because they can vary from organization to organization, or because the goal in question must be distributed across multiple organizational elements in ways peculiar to the organization.

At least 30% of incremental technical debt will be secured technical debt at the end of Year 1; 60% by the end of Year 2

Incremental technical debt is technical debt that’s incurred in the course of work currently underway or just recently completed. Because it’s so well understood, its MPrin can be estimated with higher precision than is usually possible with legacy technical debt. That precision is needed for defining the collateral and resources used to secure the debt.

A secured technical debt, like a secured financial debt, is one for which the enterprise reserves the resources needed to retire the debt. However, unlike a financial debt, the resources required to retire a technical debt might not be purely financial. Beyond financial resources, they might include particular staff, equipment, test beds, and downtime. The commitment might extend beyond the current fiscal period. Secured technical debt is a powerful means of driving down total technical debt burden, but it might require modification of internal budget management processes and fiscal reporting. Policymakers can help in designing and deploying the necessary changes.

Within one year, at least 50% of all incremental technical debt will be retired within one year of its origination; 70% within 18 months

This goal also exploits the fact that incremental technical debt can be estimated with relatively high precision. As a goal, it builds on the goal above by requiring that the resources pledged to retire incremental debts actually be expended as intended.

Within one year, all engineers and their direct supervisors will be educated in basic technical debt concepts

The educational materials will be developed in the next five months and piloted with 10% of the technical staff within seven months. The material will include an online proficiency test that 90% of engineers will have successfully passed within a year.

Within one year, 90% of project plans will include projections of the MPrin of the incremental technical debt they expect to generate for each delivery cycle

This information is useful for making forward projections of resources needed to secure incremental technical debt. Tracking the accuracy of these projections helps project planners improve their estimates.

Within one year, initiate a practice of identifying the top five forms of legacy technical debt, ranked by the volume of the contagion

Debt contagion is the propagation of a given form of technical debt by creating new system elements or assets in forms compatible with elements already identified as technical debt. By examining the body of incremental technical debt created enterprise-wide in a given time period (say, by fiscal quarter), we can determine the portion of that incremental debt that results from contagion, for each type of contagious legacy technical debt. This data is needed to identify the most contagious forms of legacy technical debt. They are prime candidates for debt retirement.

Within one year, initiate an industrial intelligence practice that is responsible for projecting the formation of enterprise-exogenous technical debt

This group must have a sophisticated grasp of the technologies in use within the enterprise that already bear enterprise-exogenous technical debt, or which could be subject to the formation of enterprise-exogenous technical debt. Its responsibilities cover enterprise products and services, as well as enterprise infrastructure. It issues advisories as needed, and an annual forecast. The group is also responsible for recommending and monitoring participation in industrial standards organizations. The group reports to the CIO or CTO.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.

Available: here; Retrieved: June 4, 2018

Cited in:

[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.

Available: here; Retrieved: June 4, 2018

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Doran 1981] George T. Doran. “There’s a S.M.A.R.T. Way to Write Management’s Goals and Objectives”, Management Review, 70:11, pp. 35-36, 1981.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Malfeasance can be a source of technical debt

Last updated on April 29th, 2018 at 06:36 am

Although creating and deploying policies to manage technical debt is a necessary step, it isn’t sufficient for achieving control in every case. Even if training and communication programs are effective, the possibility of intentional circumvention of technical debt management policy remains. Malfeasance can lead to incurring new technical debt by circumventing any policy. And malfeasance can be an obstacle to retiring—or even identifying—existing technical debt. Moreover, indirect effects of forms of malfeasance seemingly unrelated to technical debt can nevertheless incur technical debt or extend the lifetime of existing technical debt.

Examples of malfeasance as a source of technical debt

Elizabeth Holmes Backstage at TechCrunch Disrupt San Francisco 2014
Elizabeth Holmes Backstage at TechCrunch Disrupt San Francisco 2014. She is the founder, chairman, and CEO of Theranos, a startup that grew to a total valuation of $9 billion in 2015, and has since dramatically declined in value, now on the edge of its second bankruptcy. Theranos, through Holmes, claimed to have developed a technology that enabled blood testing with small amounts of blood—0.1% to 1% of the amount of blood required by conventional blood testing technologies. These claims proved false. After a series of collisions with U.S. government agencies, the U.S. Securities and Exchange Commission sued Holmes and Theranos. In March 2018, a settlement was reached in which Holmes accepted severe financial penalties, loss of voting control of Theranos, and a ban from serving as an officer or director of any public company for ten years. Photo (cc) Max Morse for TechCrunch.

Consider an example from software engineering. To save time, an engineer might intentionally choose to use a deprecated approach. When the malfeasance is discovered, one question naturally arises: in what other places and on what other projects has this individual (or other individuals) been making such choices? In a conventional approach to controlling this form of technical debt, we might be concerned only with the engineer’s current assignment. But a more comprehensive investigation might uncover a trail of technical debt artifacts in the engineer’s previous assignments.

Allman relates a hardware-oriented example [Allman 2012]. He describes an incident involving the University of California at Berkeley’s CalMail system, which failed catastrophically in November 2011, when one disk in a RAID (Redundant Array of Inexpensive Disks) failed due to deferred maintenance. Allman regards this incident as traceable to the technical debt consisting of the deferred RAID maintenance. While this particular case isn’t an example of malfeasance, it’s reasonable to suppose that decisions to defer technical maintenance on complex systems frequently are arguably negligent.

History provides us with many clear examples of malfeasance leading to technical debt indirectly. Consider the Brooklyn Bridge. Many of the suspension cables of the bridge contain substandard steel wire, which was provided to the bridge constructors by an unscrupulous manufacturer. When the bridge engineer discovered the malfeasance, he recognized that the faulty wire that had already been installed could not be removed, or even inspected. So he compensated for the faulty wire by adding additional strands to the affected cables. For more, see “Non-technical precursors of non-strategic technical debt.”

What kinds of malfeasance deserve special attention and why

Malfeasance that leads to incurring technical debt or which extends the life of existing technical debt has the potential to expose the enterprise to uncontrolled increases in operating expenses and unknown obstacles to revenue generation. The upward pressures on operating expenses derive from the MICs associated with technical debt. Although MICs can include obstacles to revenue generation, considering these obstacles separately helps to clarify of the effects of malfeasance.

Malfeasance deserves special attention because the financial harm to the enterprise can dramatically exceed the financial benefit the malfeasance confers on its perpetrators. This property of technical-debt-related malfeasance is what makes its correction, detection, and prevention so important.

For example, when hiring engineers, some candidates claim to have capabilities and experience that they actually don’t have. Once they’re on board, they expose the enterprise to the risk of technical debt creation through substandard work that can escape notice for indefinite periods. The malfeasance here consists of the candidate’s misrepresentation of his or her capabilities. Although the candidate, once hired, does receive some benefit arising from the malfeasance, the harm to the enterprise can exceed that benefit by orders of magnitude.

As a second example, consider the behavior of organizational psychopaths [Babiak 2007] [Morse 2004]. Organizational psychopathy can be a dominant contributing factor to technical debt formation when the primary beneficiary of a proposed strategy is the decision-maker or the advocate who takes credit for the short term effects of the decision, and when he or she intends knowingly to move on to a new position or to employment elsewhere before the true long term cost of the technical debt becomes evident. This behavior is malfeasance of the highest order. And although it’s rare, its impact can be severe. For more, see “Organizational psychopathy: career advancement by surfing the debt tsunami.”

What’s required to control malfeasance

When a particular kind of malfeasance can incur technical debt or extend the life of existing technical debt, it merits special attention. Examples like those above suggest three attributes that technical debt management programs must have if they are to deal effectively with malfeasance.

Corrective measures

Corrective measures can be undertaken in a straightforward manner when inadvertent policy violations occur. For example, unexpected difficulties in setting priorities for technical debt retirement efforts might be the result of individual performance metrics that conflict with the technical debt control program. Such conflicts can be inadvertent and can be resolved collaboratively.

But with regard to malfeasance, difficulties arise when policy violations are discovered or reported. When the violations are intentional, corrective action usually entails investigation of the means by which the infraction was achieved, and the means by which it was concealed. When these activities involve many individuals attached to multiple business units, some means of allocating the cost of corrective action might be needed. Allocating the cost of corrections can also be difficult when one party has reaped extraordinary benefits by taking steps that led to incurring significant technical debt. In some cases, corrective measures might include punitive actions directed at individuals.

Detection measures

When intentional violations are covert, or those who committed the violations claim that they’re unintentional, investigation is necessary to determine whether or not a pattern of violations exists. Technical debt forensic activities require resources, including rigorous audits and robust record-keeping regarding the decisions that led to the formation or persistence of technical debt. Automated detection techniques might be necessary to control the cost of detection efforts, and to ensure reliable detection.

Preventative measures

Successful prevention of policy violations requires education, communication, and effective enforcement. The basis of effective policy violation prevention programs includes widespread understanding of the technical debt concept and the technical debt management policies, and the certainty of discovery of intentional infractions. These factors require commitment and continuing investment.

Policy frameworks are at risk of depressed effectiveness if they pay too little attention to malfeasance and other forms of misconduct. Such misbehavior deserves special attention because it’s often accompanied both by attempts to conceal any resulting technical debt, and attempts to mislead investigators and managers about its existence. These situations do arise, though rarely, and when they do, they must be addressed in policy terms.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Allman 2012] Eric Allman. “Managing Technical Debt: Shortcuts that save money and time today can cost you down the road,” ACM Queue, 10:3, March 23, 2012.

Available: here; Retrieved: March 16, 2017

Cited in:

[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.

Available: here; Retrieved: June 4, 2018

Cited in:

[Babiak 2007] Paul Babiak and Robert D. Hare. Snakes in Suits: When Psychopaths Go to Work. New York: HarperCollins, 2007. ISBN:978-0-06-114789-0

Order from Amazon

Cited in:

[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.

Available: here; Retrieved: June 4, 2018

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Doran 1981] George T. Doran. “There’s a S.M.A.R.T. Way to Write Management’s Goals and Objectives”, Management Review, 70:11, pp. 35-36, 1981.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Morse 2004] Gardiner Morse. “Executive psychopaths,” Harvard Business Review, 82:10, 20-22, 2004.

Available: here; Retrieved: April 25, 2018

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

How budget depletion leads to technical debt

Some projects undergo budget depletion exercises when their budgets are reduced, or when there’s evidence that the funds remaining will be insufficient to complete the work originally planned. Formats vary, but the typical goal of these exercises is downscoping — removing, relaxing, deferring, or suspending some requirements. Since funds are limited, downscoping is often executed in a manner that leads to technical debt.

The Old River Control Complex on the Mississippi River
The Old River Control Complex on the Mississippi River. Built and operated by the US Army Corps of Engineers (USACE) the Old River Control Complex is used for controlling the flow from the Mississippi into a distributary known as the Atchafalaya River. Were it not for this facility, the Mississippi would long ago have rerouted itself into the Atchafalaya, which has a much steeper gradient to the ocean. Since that change would have deprived New Orleans and all the industrial facilities along the lower Mississippi of access to the water and navigational channels they now enjoy, USACE maintains a complex of flow control facilities to prevent Nature taking its course, and to prevent flooding along the lower Mississippi.
The industrial facilities of the lower Mississippi constitute a technical debt, in the sense that their existence is no longer compatible with the “update” Nature is trying to deploy, in the form of rerouting the flow of water from the Mississippi to the Atchafalaya. Because our national budget doesn’t allow for repositioning the city of New Orleans and all the industrial facilities from the lower Mississippi to somewhere along the Atchafalaya, we’ve elected to redirect the flow of water from the course Nature prefers to a course more compatible with the existing industrial base. Operating and maintaining the Old River Control Complex, together with a multitude of levees, dredging projects, and gates throughout lower Louisiana, are the MICs we pay for the technical debt that is the outdated position of the city of New Orleans and its associated industrial base.
For more about Atchafalaya, see the famous New Yorker article by John McPhee [MacFee 1987]. Photo courtesy USACE
Here’s an illustrative scenario. At the time downscoping begins, the work product might contain incomplete implementations of items that are to be removed from the list of objectives. A most insidious type of debt, and most difficult to detect, consists of accommodations contained in surviving artifacts that are no longer needed because the items they were intended to support have been cancelled. This class of technical debt is difficult to detect because the affected system components appear to be merely overly complicated. Recognizing it as a residual of a cancelled capability requires knowledge of its history. Unless these artifacts are documented at the time of the downscoping, that knowledge may be lost.

Other items of technical debt that arise from budget depletion include tests that no longer need to be executed, or documentation that’s no longer consistent with the rest of the work product, or user interface artifacts no longer needed. When budgets become sufficiently tight, funds aren’t available for documenting these items of technical debt as debt, and the enterprise can lose track of them when team members move on or are reassigned.

In some instances, budget depletion takes effect even before the work begins. This happens, for example, when project champions unwittingly underestimate costs in order to obtain approval for the work they have in mind. The unreasonableness of the budget becomes clear soon after the budget is approved, and the effects described above take hold soon thereafter.

Budget depletion can also have some of the same effects as schedule pressure. When the team devises the downscoping plan, it must make choices about what to include in the revised project objectives. In some cases, the desire to include some work can bias estimates of the effort required to execute it. If the team underestimates the work involved, the result is increased pressure to perform that work. With increased pressure comes technical debt. See “With all deliberate urgency” for more.

A policy that could limit technical debt formation in response to budget depletion would require identifying the technical debt such action creates, and later retiring that debt. Because these actions do require resources, they consume some of the savings that were supposed to accrue from downscoping. In some cases, they could consume that amount in its entirety, or more. Most decision-makers probably over-estimate the effectiveness of the downscoping strategy. Often, it simply reduces current expenses by trading them for increased technical debt, which raises future expenses and decreases opportunities in future periods.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Allman 2012] Eric Allman. “Managing Technical Debt: Shortcuts that save money and time today can cost you down the road,” ACM Queue, 10:3, March 23, 2012.

Available: here; Retrieved: March 16, 2017

Cited in:

[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.

Available: here; Retrieved: June 4, 2018

Cited in:

[Babiak 2007] Paul Babiak and Robert D. Hare. Snakes in Suits: When Psychopaths Go to Work. New York: HarperCollins, 2007. ISBN:978-0-06-114789-0

Order from Amazon

Cited in:

[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.

Available: here; Retrieved: June 4, 2018

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Doran 1981] George T. Doran. “There’s a S.M.A.R.T. Way to Write Management’s Goals and Objectives”, Management Review, 70:11, pp. 35-36, 1981.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[MacFee 1987] John MacFee. “Atchafalaya,” The New Yorker, February 23, 1987.

Available: here; Retrieved: February 5, 2018.

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Morse 2004] Gardiner Morse. “Executive psychopaths,” Harvard Business Review, 82:10, 20-22, 2004.

Available: here; Retrieved: April 25, 2018

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Separating responsibility for maintenance and acquisition

Last updated on July 24th, 2018 at 08:49 pm

Separating responsibility for maintenance and acquisition of technical assets can lead to uncontrolled growth of technical debt. When the performance of the business acquisition function or the performance of the development organization is measured without regard for the technical debt that arises as a consequence of their actions, technical debt is likely to expand unchecked. To limit such expansion, policymakers must devise performance measures that hold these organizations accountable for the technical debt arising from their actions.

Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010
Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010 [NOAA 2013]. The storms were so severe that at least one river flood gauge “flat-lined” — the floodwaters overtopped the gauge’s measurable range. Moreover, the National Weather Service (NWS) lacked a database of measurable ranges for flood gauges. Quoting the NWS report: “A lesson learned here was that maximum recordable river levels should be known by NWS staff, not only so staff aren’t surprised when this type of issue arises, but also to notify USGS personnel so that they can install a temporary gage and remove or elevate threatened equipment.” Technical debt, if ever I’ve seen it.
For systems consisting solely of software, separation of responsibility for system maintenance and system development or acquisition enables the acquiring organization to act with little regard for the consequences of its decisions vis-à-vis maintenance matters [Boehm 2016]. This is an unfortunate state of affairs that increases the rate of accumulation of new technical debt, and increases the lifetime of legacy technical debt, because the MICs associated with the technical debt aren’t borne by the acquiring organization.

For example, a focus on performance of the organization that’s responsible for acquisition biases them in favor of attending to the direct and immediate costs of the acquisition, with little or no regard for ongoing maintenance issues. The maintenance organization is then left to deal with whatever the acquired system contains (or lacks).

An analogous mechanism operates for organizations that develop, market, and maintain products or services with software elements in their respective infrastructures. In that case, separation of the development function from the maintenance function enables the development function to act independently of the consequences of its decisions for maintenance matters.

But the separation-of-responsibilities mechanism that leads to uncontrolled technical debt isn’t restricted to software. Any technological asset that has ongoing maintenance needs (and most of them do) can potentially present this problem.

For example, in the United States, and many other countries, two streams of resources support publicly-owned infrastructure [Blair 2017]. The funding stream covers construction, operations and maintenance, and repairs. Its usual sources are taxes, tolls, licenses, other user fees, sale of ad space, and so on. The financing stream covers up-front construction costs, to bridge the period from conception through construction, until the funding stream begins delivering resources. The financing stream usually comes from bond sales.

Although both streams are controlled by legislatures, or by agencies they establish, the effects of the two streams differ fundamentally. The financing stream is dominant in the early stages of the asset’s lifecycle — during construction. The funding stream is dominant after that — when maintenance and operations are most important. Legislators and agencies are generally reluctant to supply funding because of the impact on taxpayers and users. Legislators and agencies find financing much more palatable. For this reason, among others, U.S. infrastructure maintenance is generally under-resourced, and technical debt gradually accumulates.

So it is with technological assets in organizations. For accounting purposes, capital expenses are treated differently from operational expenses, and the result is that operational expenses can have a more significant impact on current financial results than capital expenses do. This leads organizations to underfund operations and maintenance, which contributes to the accumulation of technical debt.

Control of new technical debt accumulation and enhancement of technical debt retirement rates is possible only if the acquisition or development organizations can somehow be held accountable for the MICs that result from their actions. Securitization of the debt incurred, as I’ll address in a forthcoming post, is one possible means of imposing this accountability. But reserves are also required, because some of the debt incurred might not be known at the time the asset is acquired or created.

Separation of responsibility for system maintenance and system acquisition or system development is actually a form of stovepiping. See “Stovepiping can lead to technical debt” for more on stovepiping.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Allman 2012] Eric Allman. “Managing Technical Debt: Shortcuts that save money and time today can cost you down the road,” ACM Queue, 10:3, March 23, 2012.

Available: here; Retrieved: March 16, 2017

Cited in:

[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.

Available: here; Retrieved: June 4, 2018

Cited in:

[Babiak 2007] Paul Babiak and Robert D. Hare. Snakes in Suits: When Psychopaths Go to Work. New York: HarperCollins, 2007. ISBN:978-0-06-114789-0

Order from Amazon

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.

Available: here; Retrieved: June 4, 2018

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Doran 1981] George T. Doran. “There’s a S.M.A.R.T. Way to Write Management’s Goals and Objectives”, Management Review, 70:11, pp. 35-36, 1981.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[MacFee 1987] John MacFee. “Atchafalaya,” The New Yorker, February 23, 1987.

Available: here; Retrieved: February 5, 2018.

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Morse 2004] Gardiner Morse. “Executive psychopaths,” Harvard Business Review, 82:10, 20-22, 2004.

Available: here; Retrieved: April 25, 2018

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread

Zero tolerance and work-to-rule deliveries create an adversarial culture

Last updated on February 1st, 2018 at 07:29 am

Defining technical debt at the level of specificity needed for project objectives is difficult. Confronted with this difficulty, some internal customers of technologists adopt a zero-tolerance approach to technical debt, without specifically defining technical debt. Post-delivery — sometimes much, much, post — when technical debt is discovered or recognized, technologists are held responsible, even in cases when no one could have predicted that a specific artifact would eventually come to be regarded as technical debt. This sets up an adversarial dynamic between technologists and their internal customers.

Delayed or cancelled flights
Trouble at the airport. When airline pilots engage in work-to-rule actions, the immediate result can be large numbers of delayed or cancelled flights. The longer-term result might be beneficial to pilots, the airline, and the public, but only if labor peace can be restored, and the damage to the flying public can be overcome. So it is with work-to-rule deliveries as a way of dealing with zero-tolerance technical debt policies. The organization must overcome the adversarial culture that results from indiscriminate attempts to control technical debt. Technologists do gain some measure of protection by working to rule, but the longer-term benefit of the organization’s learning to manage technical debt arrives only if the adversarial culture can be overcome. Image (cc) Hotelstvedi courtesy Wikimedia.

And that’s when the trouble begins.

Within this adversarial dynamic, technologists try to protect themselves against future recriminations by “working to rule.” They perform only work that is specified by the internal customer. If they find something additional that must be done, they perform that work only if they successfully obtain the customer’s approval. Some customers continue to adhere to a zero-tolerance policy with respect to technical debt, but such a non-specific requirement cannot be met. Because technologists are “working to rule,” they use the ambiguity of the zero-tolerance requirement to assert that they performed all work that was sufficiently specified. This level of performance is analogous to the work-to-rule actions of some employees involved in labor disputes with their employers, and who are literally in compliance with the requirements of the employer, but only literally [LIBCom 2006].

Requiring deliverables to be totally free of technical debt contributes to formation of an adversarial culture, wherein the adversaries are the technologists and their internal customers. Shedding that adversarial culture, once it sets in, can be difficult. Compelling employees, vendors, or contractors to deliver work that’s free of all technical debt is therefore unlikely to succeed. Whether work is performed in-house by employees, or is outsourced, or is performed in-house by contractors, deliverables that meet the minimum possible interpretation of the objectives of the effort are almost certainly burdened with unacceptable levels of technical debt. What can we do to prevent this?

To avoid creating an adversarial culture, we can specify in project objectives some kinds of technical debt that must be removed in toto. To ensure steady progress in technical debt retirement, develop a statement of objectives that includes complete retirement of at least one well-defined class of technical debt, emphasizing debt classes that have the highest anticipated MICs in the near term. Other well-defined classes of technical debt can be addressed on a best-effort basis.

We must accept that any other forms of technical debt that remain at the end of a given project, or any constructions that later come to be recognized as technical debt, are just the “cost of doing business.” We’ll get to them, but unfortunately, not this time.

References

[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.

Order from Amazon

Cited in:

[Allman 2012] Eric Allman. “Managing Technical Debt: Shortcuts that save money and time today can cost you down the road,” ACM Queue, 10:3, March 23, 2012.

Available: here; Retrieved: March 16, 2017

Cited in:

[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.

Available: here; Retrieved: June 4, 2018

Cited in:

[Babiak 2007] Paul Babiak and Robert D. Hare. Snakes in Suits: When Psychopaths Go to Work. New York: HarperCollins, 2007. ISBN:978-0-06-114789-0

Order from Amazon

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.

Available: here; Retrieved: June 4, 2018

Cited in:

[Brenner 2017b] Richard Brenner. “Managing Technical Debt: Nine Policy Recommendations,” Cutter Consortium Executive Update 18:4, 2017.

Available: here; Retrieved: December 29, 2017

Cited in:

[Conroy 2012] Patrick Conroy. “Technical Debt: Where Are the Shareholders' Interests?,” IEEE Software, 29, 2012, p. 88.

Available: here; Retrieved: August 15, 2018.

Cited in:

[Cunningham 1992] Ward Cunningham. “The WyCash Portfolio Management System.” Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

Cited in:

[Doran 1981] George T. Doran. “There’s a S.M.A.R.T. Way to Write Management’s Goals and Objectives”, Management Review, 70:11, pp. 35-36, 1981.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Garnett 2013] Steve Garnett, “Technical Debt: Strategies & Tactics for Avoiding & Removing it,” RippleRock Blog, March 5, 2013.

Available: here; Retrieved February 12, 2017.

Cited in:

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

Cited in:

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

Cited in:

[LIBCom 2006] “Work-to-rule: a guide.” libcom.org.

Available: here; Retrieved: May 9, 2017.

Cited in:

[MacFee 1987] John MacFee. “Atchafalaya,” The New Yorker, February 23, 1987.

Available: here; Retrieved: February 5, 2018.

Cited in:

[McIntosh 2003] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, Premkumar Devanbu, ed., New York: ACM, 192-201

Cited in:

[Morse 2004] Gardiner Morse. “Executive psychopaths,” Harvard Business Review, 82:10, 20-22, 2004.

Available: here; Retrieved: April 25, 2018

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

[Weinberg 1985] Gerald M. Weinberg. The Secrets of Consulting. New York: Dorset House, 1985.

Ford’s Fundamental Feedback Formula. Order from Amazon

Cited in:

Other posts in this thread