A jumble of jigsaw puzzle pieces. Managing technical debt can be like solving a puzzle. Where do we begin? With jigsaw puzzles, we usually begin with two assumptions. First, we assume that we have all the pieces. Second, we assume that they fit together to make a coherent whole. These assumptions might not be valid for the puzzle of technical debt in any given organization.
Managing technical debt is something few organizations now do. And fewer do well. Several issues make managing technical debt difficult and they’re discussed elsewhere in this blog. This thread explores tactics for dealing with those issues from a variety of initial conditions. For example, tactics that work well for an organization that already has control of its technical debt, and which wants to keep it under control, might not work at all for an organization that’s just beginning to address a vast portfolio of runaway technical debt. The needs of these two organizations differ. The approaches they must take might then also differ.
What’s in this thread
The first three posts in this thread illustrate the differences among organizations in different stages of developing technical debt management practices. In “Leverage points for technical debt management,” I begin to address the needs of strategists working in an organization just beginning to manage its technical debt. They ask the question, “Where do we begin?” In “Undercounting nonexistent debt items,” I offer an observation about a risk that accompanies most attempts to assess the volume of technical debt. Such assessments are frequently undertaken in organizations at early stages of the technical debt management effort. In “Crowdsourcing debt identification,” I discuss a method for maintaining the contents of a database of technical debt items. Data maintenance is something we might undertake in the context of a more advanced technical debt management program.
Obstacles we must address
Whatever approach is adopted, it must address factors that include technology, business objectives, politics, culture, psychology, and organizational behavior. So what you’ll find in this thread are insights, observations, and recommendations that address one or more of the issues related to these fields. “Demodularization can help control technical debt” considers mostly technical strategies. “Undercounting nonexistent debt items” is an exploration of a psychological phenomenon. “Leverage points for technical debt management” considers the organization as a system and discusses tactics for altering it. And “Legacy debt incurred intentionally” explores how existing technical debt can grow as long as it remains outstanding.
Two shipping containers resting on a “spine car.” Spine cars are a kind of rail car used for shipping containers. The container on the left is a so-called tank container, used for bulk cargo. Various types of tank containers are available for transporting different types of cargo. Bulk cargoes include wine, oils, ammonia, and even cryogenic liquids. The steel frame around the tank provides compatibility with the standard container profile. That profile makes the tank compatible with equipment built to handle standard shipping containers. It functions as an interface between the tank and the container-handling equipment. Photo (cc) Mr Snrub at the English language Wikipedia
Modularity is a standard design approach for complex systems. But because modularity can be be a factor in the accumulation and persistence of technical debt, temporary demodularization can help control technical debt. And longer-term demodularization can reduce the rate of accumulation of technical debt.
Modularization is a standard design approach for complex systems
Since the 1970s, modular design of systems has been de rigueur in both software and hardware. More than that, modularization has demonstrated that it’s an essential feature of maintainable, adaptable, and extensible systems [Parnas 1979] [Sullivan 2001]. And we now understand that modularization is a foundational attribute of loose coupling in systems. Loose coupling enables system designers and maintainers to work in parallel, with independence. Independent parallelism renders systems economical at levels of complexity beyond what is achievable with tighter coupling [Orton 1990].
Eliminating duplication is one reason why modularization reduces maintenance and enhancement costs. Modularization enables system designers to create a single system element that provides needed functionality to many other parts of the system. Because there is then only one copy of the system element that provides that capability, adapting it in response to new needs, or to correct defects, need be done only once.
That’s a big deal. If we provided that capability through multiple system elements, and we needed to adapt it, adaptation would be necessary for each of those elements. Moreover, the multiplicity of elements opens the possibility that we might not perform that adaptation consistently. That could create further problems. Eliminating duplication is a most useful property of modularization.
Modularization provides many other advantages. For example, modularization shortens time-to-market for new capabilities. When extending the system by adding new capability, we sometimes need access to capabilities present in existing modules. We can access those capabilities easily in modularized systems, because those modules are already in a form that makes access easy. We have no need to recreate them for the new capability we’re implementing. They exist, and they’re already tested and ready to go. In this way, modularization shortens time-to-market for new capability.
Modularization has a dark side
It’s a bit of a story to show illustrate the dark side of modularization. Let’s give a name to the modular system element for which we’ve removed all duplicates. Since it’s now unique, I’ll call that modular system element “U.” Any system element that interacts with U is now indirectly coupled to every other system element that interacts with U. And that’s where the trouble comes in.
A scenario that illustrates the problem
When implementing a new capability N, and N needs access to U, we gain the advantages described above. But suppose N needs U to do something a bit differently from what U now does. Sometimes we can extend U to accommodate N without disturbing U’s existing “client base” of system elements that already depend on U. That’s no problem. But suppose what N now needs U to do would disturb U’s client base if we implemented the changes in the way we would do it if we were starting fresh. In that case, we would need to modify all of U’s existing clients. And then we would need to re-test everything. So let’s suppose that we don’t have time or resources to do all that work. We requested the time and resources, but we didn’t receive an approval.
Instead, we found a way to extend U in a less elegant, less maintainable, but still reliable way that doesn’t disturb U’s existing clients, and does meet N’s needs. We do that instead, promising ourselves that we’ll go back someday, when we have approval for the time and resources we need. Then we would “fix” U so that it serves both its existing clients and N in the “correct” way.
A natural question
That’s one form—exactly—of what we call technical debt. In this scenario we’ve illustrated one way in which modularization leads to technical debt formation.
So a natural question arises: would it make sense instead to create a new system element—call it U2—that meets N’s needs, and also meets the needs of U’s existing client base? It would make sense, in many cases, if only they knew about U2 and if only they had the time and resources to make the change. To create such a U2 would be demodularization—that is, a violation of modularity—and that is indeed heresy. It also creates a different technical debt: the obligation to convert U’s clients to become U2 clients someday, and then to delete U. But it might be the right approach.
When would demodularization help?
Under what conditions would demodularization be sensible? Here are three possibilities.
When a new and necessary adaptation is incompatible with existing forms
The scenario above is one situation in which demodularization can help. Demodularization helps when adding new capability, or adapting to a new need, requires a change to a shared module, and that change is incompatible with the existing uses of that module. Demodularization is than a useful technique, provided that we retire with due dispatch the technical debt that results.
When retiring technical debt requires an incompatible adaptation
A second situation arises during technical debt retirement operations. During technical debt retirement, it might be necessary to alter a shared module in a way that would be incompatible with the needs of its existing client base. In that case, the approach used above can be useful. First, create a successor (“U2”) to the original shared module (“U”) in a form that isn’t burdened with the technical debt that’s being retired. Then, at the same time, or over an extended period, convert all the clients of U to use the successor U2. In the meantime, the demodularization comprises a technical debt. When the conversion is complete, the you will have retired the original technical debt. Finally delete the original shared module U, thereby retiring the demodularization.
This approach entails some risk. In the interim period before you retire U, when demodularization is still in place, U and U2 are both in use. If you need changes in U, you might need to replicate them in U2. When that happens, duplication of effort can occur. This approach is useful, though, provided the interim period of demodularization is short compared to the anticipated intervals between incidents that require alterations to both U and U2. There is risk, of course, that the resources committed to finally retiring U might become unavailable after U2 is in place. In that case, the technical debt portfolio will have expanded to no good end. To manage this risk, the artifice of secured technical debt can prove useful.
Partial demodularization helps when adaptations are focused
In some instances, portions of a shared system element—call it “U”—evolve very rapidly, while most of the rest of U remains stable. Technical debt can accumulate rapidly if the element remains unitary—that is, in one piece. However, in some cases we can segregate the rapidly evolving portion of U into a smaller unit—call it “S.” If we provide S as a separate shared system element, those portions of the system that are experiencing rapid evolution can access S separately, without disturbing the system elements that require access only to the stable portions of U.
Such segregation might require a bit of duplication There might be pieces of S that U needs, and which must therefore appear in both U and S. Likewise, there might be pieces of U that S needs, and which must therefore appear in both U and S.
But the segregation might be worthwhile, because changes in S usually require testing S and S’s clients. Testing can be expensive in time and resources. Because test coverage isn’t always 100% (read: test coverage is rarely 100%), changes in S entail some operational risk. Segregating S reduces that risk by protecting U’s clients from changes in S.
Later, when the rapidly evolving S stabilizes, you can re-integrate it into its former residence in U. Until that point, its segregation—and the attendant duplications—might constitute a technical debt.
Last words
Accepting modularization as an inviolable design principle is one cause of unnecessary accumulation of technical debt. It makes retiring legacy technical debt more difficult. Be prepared to violate modularity, but do so judiciously.
References
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
By carefully observing what happens when we actually try to retire some kinds of technical debt, we can better understand the degree of the degree of wickedness of the effort. That understanding helps manage risk in technical debt retirement projects, reducing costs and speeding execution.
A subset of wicked problems can be viewed as super wicked, if time is running out, if the central authority fails to act, and if the people who cause them influence the solution. Some technical debt retirement situations qualify as super wicked problems.
Retiring technical debt isn’t just a problem, it’s a wicked problem. And that explains many of the difficulties we associate with retiring technical debts that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.
The tendency to underestimate technical debt retirement costs is due, in part to the interaction between the consequences of the reification error and the consequences of confirmation bias. Here are some recommendations for mitigating these effects.
Managing technical debt is something few organizations now do, and fewer do well. This thread explores tactics for dealing with a variety of technical debt situations.
I’ve often expressed the view that the people of the organization know where much of their technical debt is. Or they can find it quickly. To exploit this resource, what’s needed is a systematic method for gathering what they know. If we do, we can produce a database that can serve as a starting point for further investigation. We might call this part of the debt identification process “crowdsourcing debt identification.”
A crowd. Crowds are powerful when they coordinate their actions.
When an organization first undertakes to manage its technical debt, one of the many initial tasks is identifying its existing technical debt. There are tools for executing some of this task, at least for software assets, and they are useful. But because they’re in an early stage of development, and because many non-software assets also carry technical debt, human assistance is required. And that’s the place where crowdsourcing can help.
An example of crowdsourcing debt identification
For example, if you ask engineers for examples of technical debt in the assets they work on regularly, they can rattle off a few examples without hesitation. But a few days later, while working on whatever task has focus that day, they’ll realize that they could have mentioned another painful item. And they’ll want to report it. Gathering that kind of information is very helpful to the debt identification effort. That’s crowdsourcing in action.
But investment is required for crowdsourcing to be effective. We must educate the people who will be doing the reporting, and we must give them tools to make reporting quick and easy.
Reporting issues
Crowdsourcing debt identification will produce a stream of “incident reports” by Debt Reporters (DRs). People we might call Debt Report Administrators (DRAs) could then interpret the reports. Then they could recast the reports for later investigation by experts in the assets involved. Common difficulties that add to workload of DRAs include the items below.
Inconsistent definitions of technical debt
Lack of uniformity in understanding what technical debt is and isn’t can cause DRs to report as potential debt items some artifacts that aren’t manifestations of technical debt. Worse, they might fail to report items that are technical debt.
Only educating the DRs about the organizational definition of technical debt can enhance consistency.
Repeated reporting of previously reported debt items
Unaware that a previous report has identified a debt item, DRs might file reports unnecessarily. Some of these duplications are obvious. But if the language of the report is different enough, identifying duplicates can take time.
<
p class="left-indent">We can reduce duplication by making available descriptions of previously reported items in multiple forms.
Inconsistent descriptions of debt items
DRA must be able to recognize when two different DRs use different language to describe the same debt item. If they do not, then the debt report database will contain an unrecognized duplication.
The asset expert must then address this situation.
Failure to report known debt items
Some people, pressed by the urgency of their “own work,” might not report debt items they know about, or might hurriedly file low-quality reports. A high incidence of this behavior is an indicator of a deeper organizational issue: namely, that some people do not regard technical debt management as a worthy activity.
Tracking report quality and report frequency is one way to determine how much regard the people of the organization have for the debt management effort.
Report format and content
Reporting a potential technical debt item must not be burdensome. It must be easy. A Web-based form is a minimum. Users must be able to prefill some fields common to all their reports, and save the result as a template. Fields they might want to prefill include their personal identity and the asset identity. DRs might need several templates, depending upon the number of assets with which they interact. Switching from one template to another must also be easy.
Several authors have proposed report templates, Below is one due to Foganholi, et al. [Foganholi 2015]. (TD is technical debt)
ID
TD identification number
Date
Date of TD identification
Responsible
Person or role who should fix this TD item
Type
Design, documentation, defect, testing, or other type of debt
Project
Name of project or software application
Location
List of files/classes/methods or documents/pages involved
Description
Describes the anomaly and possible impacts on future maintenance
Estimated principal
How much work is required to pay off this TD item on a three-point scale: High/Medium/Low
Estimated interest amount
How much extra work will need to be performed in the future if this TD item is not paid off now on a three-point scale: High/Medium/Low
Estimated interest probability
How likely is it that this item, if not paid off, will cause extra work to be necessary in the future on a three-point scale: High/Medium/Low
Intentional
Yes/No/Don’t Know
Fixed by
Person or role who really fix this TD item
Fixed date
Date of TD conclusion
Realized principal
How much work was required to pay off this TD item on a three-point scale: High/Medium/Low
Realized interest amount
How much extra work was needed to be performed if this TD item was not paid off at moment of detection, on a three-point scale: High/Medium/Low
While this template might be useful for tracking the technical debt item, it contains fields that aren’t needed for crowdsourcing debt identification. A simplified template for crowdsourcing debt identification might look like this:
Identifying Report Title
Your identifier for this report
Date
Date of report (prefilled)
Type
Drop down menu of debt types, including “other”
Project
Name of the project sponsoring the work which led to your observation of the debt item
Location of debt item
List of assets involved, including specific location within complex assets
Description
Describe the debt item including
Whether your current effort has created it and if so, how
Possible impact on present or future maintenance or enhancement efforts
Whether it has led to, or is a result of, contagion
How it’s affecting your work
Intentional
Yes/No/Don’t Know
Asset experts then receive these reports and take one or more of the following actions:
Seek further information from the DR.
Reject the report as not involving technical debt. Rejection data is part of the basis for assessing the effectiveness of the education program.
Attach the report to a new or existing debt item, incorporating relevant information from the report into the debt item’s data.
The asset experts produce contains information like that suggested by Foganholi, et al.. It can be the basis of further analysis and eventual retirement of the debt item.
Last words
Investment in ease-of-use for the reporting process is essential for at least three reasons:
Some might regard reporting as an additional burden beyond the current workload.
In many organizations, some might regard reporting as a secondary responsibility.
Unless technical debt retirements rapidly become common occurrences, some might regard reporting as a waste of effort. The reporting itself must therefore be easy.
These phenomena all exert negative pressure on report quality. They tend to suppress report frequency. Ease-of-use can mitigate these effects.
References
[Foganholi 2015] Lucas Borante Foganholi, Rogério Eduardo Garcia, Danilo Medeiros Eler, Ronaldo Celso Messias Correia, and Celso Olivete Junior. “Supporting technical debt cataloging with TD-Tracker tool,” Advances in Software Engineering 2015, 4.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
By carefully observing what happens when we actually try to retire some kinds of technical debt, we can better understand the degree of the degree of wickedness of the effort. That understanding helps manage risk in technical debt retirement projects, reducing costs and speeding execution.
A subset of wicked problems can be viewed as super wicked, if time is running out, if the central authority fails to act, and if the people who cause them influence the solution. Some technical debt retirement situations qualify as super wicked problems.
Retiring technical debt isn’t just a problem, it’s a wicked problem. And that explains many of the difficulties we associate with retiring technical debts that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.
The tendency to underestimate technical debt retirement costs is due, in part to the interaction between the consequences of the reification error and the consequences of confirmation bias. Here are some recommendations for mitigating these effects.
Managing technical debt is something few organizations now do, and fewer do well. This thread explores tactics for dealing with a variety of technical debt situations.
Sherlock Holmes and Doctor Watson, in an illustration by Sidney Paget, with the caption, “Holmes gave me a sketch of the events.” In 1892 The Strand magazine published this illustration to accompany a story called “The Adventure of Silver Blaze” by Sir Arthur Conan Doyle. It’s in this story that the following dialog occurs:
Gregory (Scotland Yard detective): “Is there any other point to which you would wish to draw my attention?”
Holmes: “To the curious incident of the dog in the night-time.”
Gregory: “The dog did nothing in the night-time.”
Holmes: “That was the curious incident.”
From this, Holmes deduces that the dog’s master was the villain. This is an example of looking for what isn’t there, and failing to notice it. It’s an example of absence blindness.
People and companies are developing technologies for assessing the nature and volume of technical debt borne by enterprise assets. The key word is developing. Some tools do exist, and they can be helpful. But they can’t do it all. Most assessments also rely on surveys and interviews of engineers and their managers. But these tools have limitations, too. Among these limitations is undercounting nonexistent debt items in surveys about technical debt.
It’s well known that survey results can exhibit biases. Collectively, these biases are known as response biases [Furnham 1986]. Sources of response bias include phrasing of questions, the demeanor of the interviewer, the desires of the participants to be good experimental subjects, attempts by subjects to respond with the “right answers,” selection of subjects, and more. These sources of bias are real, and we must address them when we design surveys.
Selection bias and absence blindness
But I have in mind here a set of biases more specific to technical debt. For example, when we ask subjects for examples of technical debt, they’re more likely to recall and provide examples of artifacts that exist than they are to provide examples of artifacts that don’t exist. This happens because of a cognitive bias called selection bias. The effect isn’t intentional, and it can dramatically skew results.
Selection bias is an example of a cognitive bias. In this case, selection bias acts to skew the data in such a way as to interfere with proper randomization, which ensures that the sample data doesn’t accurately represent the actual population of technical debt artifacts. Specifically, the data will tend to under-represent technical debt artifacts that don’t exist. Related phenomena are absence blindness and survivorship bias.
For example, regression testing is an essential step used in refactoring systems. When regression tests are unavailable, and we try to refactor a system to retire some of its technical debt, we face a problem. We can’t be certain that we haven’t changed something important. And so, when a survey design doesn’t mitigate the effects of selection bias, we can expect an elevated probability of failing to note any missing regression tests.
Mitigating the risk of undercounting nonexistent debt items
It’s helpful for surveys to include questions that specifically ask subjects to report technical debt items that don’t exist, but which would be helpful if they did exist—like missing regression tests. Even more helpful: conduct brainstorming sessions for engineers in which the goal is to list missing artifacts, tools, or processes that comprise technical debt precisely because they’re missing.
References
[Foganholi 2015] Lucas Borante Foganholi, Rogério Eduardo Garcia, Danilo Medeiros Eler, Ronaldo Celso Messias Correia, and Celso Olivete Junior. “Supporting technical debt cataloging with TD-Tracker tool,” Advances in Software Engineering 2015, 4.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
The door to a bank vault. One way to know that technical debt differs from financial debt is that banks aren’t involved in any way. Treating technical debt as if it had anything in common with financial debt—beyond our own sense of obligation—is a shortcut to real trouble. Remember the Principal Principle.
When some organizations first realize that technical debt is limiting their performance, they begin by chartering a “technical debt inventory.” They try to determine how much technical debt they’re carrying, where it is, and how much retiring it would cost. They really want to know how fast they can retire it. That’s understandable. It’s not too different from how one would approach an out-of-control financial debt situation. It might be understandable, but in most cases, inventorying all technical debt is ineffective. For technical debt we need a different approach, because technical debt is different from financial debt. With technical debt, we must be follow what I call the Principal Principle, which is:
The Metaphorical Principal (MPrin) of a technical debt, which is the cost of retiring it, isn’t what matters most. What usually matters most is the Metaphorical Interest Charges—the MICs.
Why MICs matter more
MICs on technical debt can vary dramatically. For assets that aren’t undergoing maintenance or enhancement, the MICs can be Zero for extended periods. And for retiring assets, any technical debt they carry can vanish when the asset passes out of service. For other assets, MICs can be dramatically higher—beyond the total cost of replacing the asset.
Most people regard the sole effect of MICs as reduction in engineering productivity. I take a different approach. I include in MICs anything associated with technical debt and which depresses net income. That would include lost or delayed revenue, increased expenses—anything. For example, suppose technical debt causes a two-month delay in reaching a market. IN that case, its effect on revenues can be substantial for years to come. I regard all of that total effect as contributing to MICs.
So the Principal Principle is that a focus on Principal can be your undoing. Focus on MICs instead. Drive them to Zero and keep them there.
McMurdo Station, Antarctica, as seen from nearby Observation Hill. The United States Antarctic Program, a unit of the National Science Foundation, operates the station. It can house as many as 1258 people in Summer. Photo (cc) Gaelen Marsden courtesy McMurdo Station in Antarctica.
Adopting a technical debt management programs entails significant organizational change. The problem can seem so daunting that we don’t know where to begin. The places to begin are the places where the change agents have greatest leverage—what systems analysts call leverage points. Consider this scenario:
You’re sitting in the kickoff meeting of the new Technical Debt Management Task Force. The CEO is talking about how she realized that the company had a technical debt problem. It was when the Marigold project went through delay after delay, and was finally declared done, with multiple objectives waived. She’s saying something about, “we were trying to do backflips with millstones around our necks. So I want this task force to show us how to get rid of the millstones, and then get rid of them.”
OK, you think. But how? We’re a global enterprise with thousands of engineers and operations on every continent. Except maybe Antarctica. No wait, we’re there, too. McMurdo I think. We have software we don’t even know much about, acquired long ago along with the companies that built it. And we’re building new systems or modifying old ones all the time, trying to move everything to the cloud while enhancing data security. Where do we begin to look for the millstones of technical debt?
Have you been in that meeting? If not, can you imagine being in that meeting? Meetings like that are happening around the globe. We’re all in the same soup.
Leverage points: how to get rid of the millstones
It turns out that the answers to the millstone questions are available, but the pioneers and deep thinkers who have shown the way aren’t working on technical debt. Their field is called systems analysis. They work on problems like the collapse of the North Atlantic fishery, urban deterioration, unemployment, poverty, climate change, and the causes of the Great Recession of 2008—really difficult problems. Although the technical debt problem isn’t quite that challenging, it’s challenging enough to justify taking a look at the methods of systems analysis.
And when we do that, we immediately encounter a concept many call leverage points.
What are leverage points?
Leverage points are places in complex systems where a small change in one thing can produce big changes in system behavior. In a brilliant 1997 article, Donella Meadows describes what she calls “places to intervene in a system.” [Meadows 1997] She followed this article, making improvements each time, in 1999 [Meadows 1999] and 2008 [Meadows 2008]. Let me summarize Meadows’ work here.
To alter the behavior of a complex system, intervene at one or more of 12 categories of leverage points. For example, one category is called “Rules.” It consists of the incentives, punishments, and constraints that govern the behavior of the people and institutions that comprise the system. By adjusting the system’s rules, we can alter overall system behavior.
One more thing: the leverage points form an ordered hierarchy, ordered by effectiveness. Acting at a higher-level leverage point is more effective than acting at a lower-level leverage point. And more difficult, too. The ordering of the categories is a bit fuzzy, because every situation has its own quirks, but generally, the order is as given in the list below.
The twelve leverage points
In a moment I’ll give an example of using leverage point #9, Delays, to bring about change in the way the enterprise deals with technical debt. But first, here’s a brief summary of the leverage points in increasing order of leverage; not enough to truly understand what they are, but probably enough to pique your interest. As I write posts that illustrate interventions at these leverage points, I’ll link to them from here.
Numbers: Constants and parameters such as subsidies, taxes, and standards
Buffers: The sizes of stabilizing stocks relative to their flows
Stock-and-Flow Structures: Physical systems and their nodes of intersection
Delays in feedback loops
Balancing Feedback Loops: The strength of the feedbacks relative to the impacts they are trying to correct
Reinforcing Feedback Loops: The strength of the gain of driving loops
Information Flows: The structure of who does and does not have access to information
Rules: Incentives, punishments, and constraints
Self-Organization: The power to add, change, or evolve system structure
Goals: The purpose or function of the system
Paradigms: The mind-set out of which the system—its goals, structure, rules, delays, parameters—arises
Transcending Paradigms
Delays in feedback loops
When we use feedback to control systems, and there are delays in the feedback, we can potentially create destructive system behavior. And that can happen when we try to control technical debt.
Whenever we try to control a quantity in an enterprise process, we must (a) set a target value for that quantity; then (b) measure its current value; and then (c) take action as appropriate to move the current value toward the target value. Systems analysts (and control theorists) call that arrangement a feedback loop. The action taken to move the current value to the target value is sometimes called the control signal. Under certain conditions, the feedback works as expected.
For example, to control the profitability of the enterprise, we can examine its net income, say, quarterly. And at the end of each quarter we can make adjustments if net income isn’t in the target range.
Feedback loops generally work pretty well, but under some conditions, oscillations can develop. One of those troublesome situations occurs when there’s a delay in the loop that’s of the same order as (or longer than) the time the system takes to respond to adjustments. Meadows uses the example of adjusting the water temperature of a shower when there’s a long delay between making the adjustment and feeling its effects. Overcorrection is almost inevitable, and that’s what causes system oscillation.
How controlling technical debt can create feedback loops
So let’s suppose that we’re trying to control the rate of accumulation of technical debt. One approach is to set a target for TDnew, the new technical debt generated in a project. To be fair to all projects, we decide to normalize this quantity according to the project budget B. So we set targets for each project’s N = TDnew/B, and we require that projects estimate N, on an ongoing basis, with a goal of having N in some target range when the project is complete.
Identifying technical debt isn’t straightforward
One problem with this approach is that we rarely identify accurately all the technical debt we’ve incurred until some time has passed after project delivery. With time, as the newly produced assets go into production and learning accumulates, we acquire the wisdom needed to identify more of the technical debt we created. This is one source of delay in this feedback loop.
So let’s assume that this happens for several projects, and management decides that delayed recognition of incurred technical debt is a common occurrence. To account for this, management lowers the target ranges for N for future projects. This causes project managers and project sponsors to include in their project plans additional effort directed at retiring more of their incremental technical debt before their projects complete, to enable them to project lower values of N. They must therefore identify as much of the incremental technical debt as they can, and retire it, to meet the lower targets for N.
How oscillations set in
But recall that technical debt identification sometimes requires time and experience using the newly produced asset. And the reverse process also occurs. Technical artifacts that we thought were technical debt prove to be useful in unexpected ways, and actually turn out not to be debt items after all. As a result, some of the incremental technical debt that got retired before the project was completed actually should not have been retired. Eventually, people realize that this happens with uncomfortable frequency, and so the targets for N are raised once more.
Oscillations thus set in. Long delays inevitably cause them. To prevent oscillations, shorten the delays.
How to shorten delays in feedback controlling technical debt
When we use feedback to control a system, delays in that feedback can lead to instability. Trying to control technical debt is no exception. With technical debt we can shorten delays in several ways.
If the asset is meant for human use, involve representatives of the user population in the development and design process as soon as practical. Have them exercise the asset, or prototypes, early. Listen to their suggestions. Observe how they use the asset.
If the asset must interact with non-human assets, exercise it early and often. Don’t think of this as testing, though it might look very much like testing. What you’re actually doing is searching for shortcomings in how the asset interacts with non-human assets, in design and implementation in an asset that already works.
Subject the asset to multiple reviews all along the development trajectory. Don’t wait for final release to review it.
These practices expose technical debt items early—potentially, during initial design—thereby reducing delays in identifying what is and what isn’t technical debt. They help to advance the date at which we uncover missing capabilities or capabilities designed or implemented in awkward ways. No surprise, I’m sure, but these practices are consistent with Agile approaches to technological development.
Indirect effects can add to delayed recognition of technical debt
Most of the argument above assumed that the incremental technical debt associated with the project was incurred within the asset undergoing development or maintenance. But technical debt can occur in other assets as well. When the development team is unaware of such “remote” or “indirect” incremental technical debt, recognition of that new incremental technical debt can be significantly delayed. The project’s N (the ratio of incremental technical debt to project budget) will appear to be smaller that it actually is, until that remote incremental technical debt is recognized.
This form of delay is likely to occur when the debt incurred is asset-exogenous. Recall the example of line extension of mobile phones. In that example, the enterprise incurs technical debt in one set of products as a result of the introduction of a different product. In some cases, the newly incurred technical debt is immediately evident. When it is not, delays can be substantial.
This effect is by no means rare. Any organizational change can potentially add to the technical debt portfolio—reorganizations, acquisitions, expansions, wholly new products, and much more.
Last words
Interventions at the leverage points of an organization can produce the changes we want with a minimum of effort. Some subtlety is involved, because Meadows’ leverage points are expressed at a high level of abstraction. But applying them to the problem of technical debt management is a promising approach.
Bookmark this post. I’ll be linking to more examples of using leverage points to manage technical debt. So far:
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
A ball and chain, with shackle. Attaching this device to the legs of prisoners or slaves limits their ability to run. Similarly, technical debt limits the organization’s ability to exploit new opportunities, or even to maintain their current market positions.
The American Psychiatric Association (APA) has identified a disorder called Substance Use Disorder (SUD). It includes alcoholism, drug addiction, and other patterns related to substance use [APA 2013]. Their research can serve as a model for understanding organizational behavior related to technical debt. In this post I show how to use that model to describe a disorder of organizations that we could call Technical Debt Use Disorder (TDUD). In the grip of TDUD, the organization can’t retire much of its technical debt. It can’t stop incurring new debt, even though almost everyone in the organization realizes that technical debt is harming the organization.
A brief description of APA’s publication, DSM-5®, might explain the connection between SUD and TDUD. DSM-5 is the fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM). Health care professionals in the United States and much of the rest of the world use it as a guide to diagnosing mental disorders. It’s also a framework for further research. First published in 1952, the current revision, DSM-5, was released in 2013.
The connection to technical debt
So what does DSM-5 have to do with technical debt?
What distinguishes responsible use of technical debt from irresponsible use is a topic that has generated many papers, conference presentations, and hallway debates over the years. Although there is consensus about the distinction in many cases, the debate continues. Sometimes, though, research in one field suggests paths forward in seemingly unrelated fields. So much thought and study has been invested in DSM-5 that it’s worth a look to see if the technical debt community can harvest something useful from the research in psychiatry.
I looked at DSM because I noticed that organizations that carry significant volumes of technical debt seem to have difficulty retiring it. In some cases, they also have difficulty halting accumulation of technical debt, or even slowing the rate of accumulation. This struck me as similar to the substance use problems some people encounter. I began to wonder whether there might be parallels between the substance use disorders that afflict people—alcoholism, drug addiction, and so on—and the technical debt problems that afflict organizations.
In the table below is one set of parallels I’ve found. The left column of the table is the list of diagnostic criteria for Substance Use Disorder provided in the DSM. The right column is my rewording of those criteria in an attempt to make them apply to how organizations deal with technical debt. I had thought initially that the rewording exercise might be difficult—that it might be a stretch. And here and there, it was a bit of a stretch. But overall, the SUD framework is a very good fit.
Diagnosing technical debt use disorder
Have a look at the table, and then check out the comments below it about how health care professions use the criteria.
DSM-5 Criteria for Substance Use Disorder (SUD)
Criteria for Enterprise Technical Debt Use Disorder (TDUD)
1. Taking the substance in larger amounts or for longer than you’re meant to.
1. Incurring technical debt in larger amounts than you intended and carrying it for longer than you intended.
2. Wanting to cut down or stop using the substance but not managing to.
2. Wanting to retire your technical debt or reduce the rate of incurring it but not managing to.
3. Spending a lot of time getting, using, or recovering from use of the substance.
3. Spending a lot of time dealing with the consequences of the technical debt you’ve already incurred.
4. Cravings and urges to use the substance.
4. Insistent demands on precious resources, causing the enterprise to incur “just a little more” technical debt.
5. Not managing to do what you should at work, home, or school because of substance use.
5. Not managing to attend to the needs of existing products, services, or technological infrastructure because of the demands resulting from metaphorical interest charges on technical debt.
6. Continuing to use, even when it causes problems in relationships.
6. Continuing to carry technical debt, or continuing to incur yet more technical debt, even though it causes toxic conflict among employees, and problems in customer relationships and strategic partnerships.
7. Giving up important social, occupational, or recreational activities because of substance use.
7. Giving up developing important new products or services, or upgrading critical infrastructure, or pursuing new initiatives because of resource deficits traceable to technical debt service.
8. Using substances again and again, even when it puts you in danger.
8. Incurring technical debt again and again, even when it puts the enterprise in fiscal danger or danger of losing market position.
9. Continuing to use, even when you know you have a physical or psychological problem that could have been caused or made worse by the substance.
9. Continuing to incur technical debt, even when you know you have a fiscal or market leadership problem that could have been caused or made worse by technical debt.
10. Needing more of the substance to get the effect you want (tolerance).
10. Needing to incur more technical debt to get the fiscal effect you need—a product delivered or a contract completed.
11. Development of withdrawal symptoms, which can be relieved by taking more of the substance.
11. Upon attempting to retire existing technical debt, or halting incurring yet more technical debt, fiscal or market position problems develop in short order, which can be relieved only by incurring yet more debt.
In health care, two or three symptoms indicate a mild substance use disorder; four or five symptoms indicate a moderate substance use disorder, and six or more symptoms indicate a severe substance use disorder. Have a look at the right-hand column. How would you score your organization? Can we categorize the severity of an organization’s problem with technical debt using a scale similar to the one health care professionals use for SUD?
Conclusion
Technical debt isn’t inherently evil. Its existence among technological assets isn’t proof of engineering malpractice. For example, we can decide responsibly to deliver a system that carries technical debt. But if we do, we must carefully weigh the consequences of incurring that debt against the consequences of delayed delivery. And we must have a workable plan for retiring that debt, or for carrying the burden of its MICs.
But organizations can nevertheless trap themselves in cycles of technical debt, unable to make much progress in reducing it. In some cases, business as usual won’t work. In some cases, only drastic action can break the cycle.
References
[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
“The Young Chief Uncas,” 1869 chromolithograph by John Mix Stanley (1814–1872). Photo courtesy Wikimedia Commons.Uncas then deduces amazing details about the man who left the track he examined—150 years ahead of Sherlock Holmes. Natty Bumppo, Cooper’s Hawkeye character, calls the signs Uncas uses tell-tales.
Technical debt smell would be a useful indicator of the presence of a severe problem of technical debt in an organization. Unfortunately, technical debt doesn’t usually have a smell, as such. But metaphorically, it might have indirect indicators, just as we might say, “I smell a rat” when probing a mystery. Actually, the idea of indirect indicators has a long and storied tradition. In one scene midway through James Fenimore Cooper’s The Last of the Mohicans, Uncas (who is the actual last of the Mohicans) demonstrates his tracking skills [Cooper 1857]:
The young Mohican bent over the track, and removing the scattered leaves from around the place, he examined it with much of that sort of scrutiny that a money-dealer, in these days of pecuniary doubts, would bestow on a suspected due-bill. At length he arose from his knees, satisfied with the result of the examination.
Tell-tales in Nature
Nature abounds with examples of such skill at noticing tell-tales. Lions, tigers, bears, and all sorts of fauna use their olfactory senses to detect food, predators, mates, offspring, weather, and even the change of seasons. Smell gives them access to information they need, often before sight or hearing can.
That’s probably a part of why smell has become a useful metaphor in software engineering. The technical literature about code smells is vast and growing [Haque 2018]. In a blog post titled “CodeSmell,” Martin Fowler defines code smell as, “…a surface indication that usually corresponds to a deeper problem in the system.” [Fowler 2006] Code smells are traits that are easy to recognize, and often—but not always—indicators of problems.
Enter the “red flag”
The concept is also useful in the business domain, though there we use a different metaphor and a different term. In the business context, we call smells red flags. Investopedia defines a red flag as, “…an indicator of potential problems with a security, such as any undesirable characteristic that stands out to an analyst as it pertains to a company’s stock, financial statements or negative news reports.” But I’ve heard the term red flag used in the context of evaluating proposals, assessments, status reports, personnel, and intelligence of all kinds.
Whether called tell-tales, smells, red flags, or just indicators, their value is that they suggest the outlines of something we haven’t yet seen clearly enough to identify with certainty. Their principal attributes are that they’re available at the surface of the domain we’re surveying, they’re relatively cheap to obtain, and, if found, they suggest trouble, and deeper investigation might be worthwhile.
Cultural smells
In the software engineering community, technical debt is regarded as a smell that indicates trouble in the system’s software. So we might ask, “Among policymakers, what are the smells that indicate trouble in the organization?” If technical debt is the trouble we’re looking for, what are the cultural smells that indicate that technical debt might be a problem?
Said differently, can we find, or can we develop, a set of attributes of enterprise culture that indicate the degree of severity of an organization’s problems with technical debt?
Here are some possible “technical debt smells”—aspects of enterprise culture that could indicate problems with technical debt:
There is a general belief that technical debt afflicts some organizations, but not ours
We’re a new startup—just a year old—so we have no technical debt.
We don’t build software, therefore no technical debt
Nontechnical members of our executive team have little if any knowledge of the concept of technical debt
No enterprise resources are allocated to educating nontechnical employees about technical debt
The VP of Marketing doesn’t believe that anything she does could possibly contribute to technical debt
There is a general belief that if we have technical debt, it’s due solely to malpractice on the part of engineers
We’ve tried to assess the total cost of eliminating all of our technical debt. But we found the estimates so unreliable that we decided to leave well enough alone.
We do believe that technical debt does have costs. But because it only affects the productivity of engineers, we just hired more engineers and decided to live with it.
Last words
Clearly we could assemble a list of technical debt smells—beliefs about technical debt and behaviors that affect it—and check for their presence in a given organization. But fortunately, some of that work has already been done, albeit in a very different context; That context is a malady psychiatrists call “Substance Use Disorder.” More about that next time.
References
[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.
[Haque 2018] Md Shariful Haque, Jeff Carver, and Travis Atkison. "Causes, impacts, and detection approaches of code smell: a survey." Proceedings of the ACMSE 2018 Conference. ACM, 2018.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
Attempting to reduce technical debt by setting so-called “SMART goals” in the obvious way can often disappoint. SMART, due to George T. Doran [Doran 1981], is widely used for expressing management goals. “SMART” is an acronym for “Specific, Measurable, Attainable, Realistic, and Time-boxed.” The last three words are available in various alternative ways. Doran himself used “assignable, realistic, and time-related.”
SMART is deeply embedded in management culture. Many assume without investigation that expressing technical debt goals using the SMART pattern will produce desired results. Also embedded in management culture is the aphorism, “You get what you measure.” [Ariely 2010] [Bouwers 2010] A typical technical debt reduction goal: “Reduce technical debt by 20% per year for the next five years.”
SMART goals in their simplest form are ineffective for technical debt
Prof. George T. Doran (1939-2011), creator of the S.M.A.R.T acronym for setting management objectives. Watch a 2010 interview of Prof. Doran at YouTube.There’s ample support for a claim that applying the SMART technique in direct ways will be ineffective. Much employee behavior affects technical debt indirectly. It can overwhelm the effects of employee behaviors that affect technical debt indirectly. The direct approach does cause some employees to adopt desirable behaviors. But their impact isn’t significant enough compared to the effects of the behaviors that affect technical debt indirectly. Employees who see little connection between their own activities and the burden of technical debt can unwittingly have enormous impact. Moreover, many are subject to competing constraints on their behaviors that then cause them to act in ways that increase technical debt.
That’s why it’s necessary for management to develop a series of SMART goals that affect behaviors that have indirect effects on technical debt. In the first part of this post, “Setting a direct SMART goal for technical debt reduction is problematic,” I explore the problems inherent in the direct approach. In the second part, “How to set SMART goals for technical debt,” I provide examples of SMART goals that touch on behaviors that have indirect effects on technical debt.
Setting a direct SMART goal for technical debt reduction is problematic
Let’s begin by exploring some of the problems with the direct approach. In this section, I assume that management has set a SMART goal for the enterprise in the form, “Reduce technical debt by 20% for each of the next five years.” But there’s nothing special about the numbers. My comments below apply to the form of the goal, rather than the specific numbers.
The direct approach assumes measurability
To attain a goal of a 20% reduction in technical debt in a given year, we must be able to measure the level of technical debt. We measure it at the beginning of the year and at the end of the year. Presumably we do so with confidence in the 90% range or better. Such a measurement with the precision required might not be possible. Moreover, in most cases the probability that such a measurement is possible is low. For these reasons, setting periodic goals for total technical debt isn’t a useful management tool.
Consider a simple example. One common form of technical debt is missing or incompletely implemented capability. In some instances, the metaphorical principal (MPrin) of a given instance of this debt in the current year can change spontaneously to a dramatically larger value in the following year (or even the following week). This can happen due to changes in the underlying asset unrelated to the technical debt. Ot it can happen due to debt contagion. Or it can happen due to any number of other reasons. When this happens, the technical debt retirement effort for that year can appear to have suffered a serious setback. Setbacks like this can happen even though the technical debt retirement teams have been performing perfectly well.
The direct approach assumes a static principal
With most financial debts, a loan agreement sets the principal amount. Moreover, we can compute the principal at any time given the mathematical formulas specified in the loan agreement.
By contrast, in many cases, the metaphorical principal amount of a technical debt might be neither fixed nor readily computable. We can estimate the MPrin of a given kind of technical debt at a given time, and we can even make forward projections. But they are merely estimates, and their error bars can be enormous. See “Policy implications of the properties of MPrin” and “Useful projections of MPrin might not be attainable.”
The direct approach focuses on MPrin, not MICs
Objectives expressed in terms of the volume of technical debt—the total MPrin—are at risk of missing the point. Total MPrin isn’t what matters most. What matters is MICs—the total cost of carrying the debt. Even more important is the timing of arrival of the MICs. See “The Principal Principle: Focus on MICs.”
And like MPrin, MICs can vary in wild and unpredictable ways. For example, the MICs for a piece of technical debt borne by an asset that isn’t undergoing maintenance or enhancement can be zero; in a later time period, when that asset is undergoing enhancement, the MICs can be very high indeed. See “MICs on technical debt can be unpredictable” for a detailed discussion.
Priority setting for technical debt retirement is most effective when it accounts for the timing of MICs. For example, suppose we know that we must enhance a particular asset by FY21 Q3. And suppose we know that it bears technical debt that adds to the cost of the enhancement. Then retiring that debt in FY20 would be advisable. But if that technical debt has zero MICs for the foreseeable future, retiring it might not be worth the effort.
Unless policies are already in place governing the formation of incremental technical debt, technical debt retirement programs might encounter severe difficulty. New development and maintenance and enhancement of existing assets are ongoing. They generally produce technical debt in various forms. The technical debt retirement program might simply be unable to keep up with new debt formation.
The direct approach fails to anticipate the formation of enterprise-exogenous technical debt
Technical debt can sometimes form as a result of innovations, changes in standards, or changes in regulations that occur entirely external to the enterprise. I call such technical debt enterprise-exogenous. When this happens, the technical debt retirement effort can appear to have suffered a serious setback, even though the technical debt retirement teams might have been performing perfectly well. Before initiating a technical debt reduction program, it’s wise to first deploy a program that’s capable of retiring technical debt at a pace that at least equals the pace of formation of enterprise-exogenous technical debt.
Incurring technical debt is sometimes the responsible thing to do
At times, incurring technical debt is prudent. In these situations, accepting the debt you’ve incurred—even for the long term—might be appropriate. Strict goals about total technical debt can lead to reluctance to incur debt that has a legitimate business purpose. To prevent this, goals for total technical debt must be nuanced enough to deal with these situations. Goals for total technical debt that adhere strictly to the SMART goal pattern sometimes lack the necessary level of nuance.
How to set SMART goals for technical debt
SMART goals can work for technical debt management, but we must relate them to behavioral choices. Here are some examples of SMART goals that can be effective elements of a technical debt management program. Some of these examples are admittedly incomplete. For example, I offer no proof of assignability, attainability, or realism. Such attributes can vary from organization to organization. And we must allocate the goal in question across multiple organizational elements in ways peculiar to the organization.
At least 30% of incremental technical debt will be secured technical debt at the end of Year 1; 60% by the end of Year 2
Incremental technical debt is technical debt that’s incurred in the course of work currently underway or just recently completed. Because it’s so well understood, its MPrin can be estimated with higher precision than is usually possible with legacy technical debt. That precision is needed for defining the collateral and resources used to secure the debt.
A secured technical debt, like a secured financial debt, is one for which the enterprise reserves the resources needed to retire the debt. However, unlike a financial debt, the resources required to retire a technical debt might not be purely financial. Beyond financial resources, they might include particular staff, equipment, test beds, and downtime. The commitment might extend beyond the current fiscal period. Secured technical debt is a powerful means of driving down total technical debt burden, but it might require modification of internal budget management processes and fiscal reporting. Policymakers can help in designing and deploying the necessary changes.
Within one year, at least 50% of all incremental technical debt will be retired within one year of its origination; 70% within 18 months
This goal also exploits the fact that we can estimate incremental technical debt with relatively high precision. As a goal, it builds on the goal above by requiring that the organization actually expend as intended the resources pledged to retire incremental debts.
Within one year, all engineers and their direct supervisors will be educated in basic technical debt concepts
The educational materials will be developed in the next five months and piloted with 10% of the technical staff within seven months. The material will include an online proficiency test that 90% of engineers will have successfully passed within a year.
Within one year, 90% of project plans will include projections of the MPrin of the incremental technical debt they expect to generate for each delivery cycle
This information is useful for making forward projections of resources needed to secure incremental technical debt. Tracking the accuracy of these projections helps project planners improve their estimates.
Within one year, initiate a practice of identifying the top five forms of legacy technical debt, ranked by the volume of the contagion
Debt contagion is the propagation of a given form of technical debt by creating new system elements or assets in forms compatible with elements already identified as technical debt. By examining the body of incremental technical debt created enterprise-wide in a given time period (say, by fiscal quarter), we can determine the portion of that incremental debt that results from contagion, for each type of contagious legacy technical debt. This data is needed to identify the most contagious forms of legacy technical debt. They are prime candidates for debt retirement.
Within one year, initiate an industrial intelligence practice that is responsible for projecting the formation of enterprise-exogenous technical debt
This group must have a sophisticated grasp of the technologies in use within the enterprise that already bear enterprise-exogenous technical debt, or which could be subject to the formation of enterprise-exogenous technical debt. Its responsibilities cover enterprise products and services, as well as enterprise infrastructure. It issues advisories as needed, and an annual forecast. The group is also responsible for recommending and monitoring participation in industrial standards organizations. The group reports to the CIO or CTO.
References
[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.
[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.
This article is probably the source of the adage “You are what you measure.” Personally, I believe it’s overstated. That is, it’s true in the large, perhaps, but not in detail. Moreover, there are some things that we are that can’t be measured. But it’s important to understand the content of this article because so many people take it as dogma. Available: here; Retrieved: June 4, 2018
[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.
[Haque 2018] Md Shariful Haque, Jeff Carver, and Travis Atkison. "Causes, impacts, and detection approaches of code smell: a survey." Proceedings of the ACMSE 2018 Conference. ACM, 2018.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
Exogenous technical debt is debt that arises from causes not directly related to the asset that bears the debt. Mastering understanding of exogenous technical debt is essential to controlling technical debt formation. Exogenous technical debt is particularly troublesome to those who work on the affected assets. They can’t control its formation, and they’re rarely responsible for creating it. But their internal customers and those who control resources often fail to understand this. Indeed, those who work on the affected assets often bear blame for the formation of exogenous technical debt even though they had no role in its formation, and could have done nothing to prevent its formation.
Exogenous technical debt and endogenous technical debt
Technical debt is exogenous when it’s brought about by an activity not directly related to the assets in which the debt appears. The word exogenous comes from the Greek exo– (outside) + –genous (related to producing). So exogenous technical debt is that portion of an asset’s debt that comes about from activities or decisions that don’t involve the asset directly.
Why we must track exogenous technical debt
Asbestos with muscovite. Asbestos is a family of minerals occurring naturally in fibrous form. The fibers are all known carcinogens. Until 1990, asbestos was a common ingredient of building materials, including insulation, plaster, and drywall joint compound. It’s now banned, but it’s present in existing homes and offices. The ban caused these structures to incur exogenous technical debt. Photo by Aramgutang courtesy Wikipedia.
Because so much technical debt arises indirectly, controlling its direct formation is insufficient to achieve control. To control technical debt formation, we must track which activities produce it. We must track both direct and indirect effects. Allocating technical debt retirement costs to the activities that brought that debt about is useful. It’s useful even if the allocation doesn’t affect budget authority for those activities. Knowledge about which past activities created technical debt, and how much, is helpful for long-term reduction in the rate of technical debt formation.
When we think of technical debt, we tend to think of activities that produce it relatively directly. We often imagine it as resulting solely from engineering activity, or from decisions not to undertake engineering activity. In either case the activity involved, whether undertaken or not, is activity directly involving the asset that carries—or which will be carrying—the technical debt. This kind of technical debt is endogenous technical debt. The word endogenous comes from the Greek endo– (within or inside) + –genous (related to producing). So endogenous technical debt is that portion of an asset’s debt that comes about from activities or decisions that directly involve the asset.
More about endogenous technical debt in future posts. For now, let’s look more closely at exogenous technical debt, and its policy implications.
Examples of exogenous technical debt
In “Spontaneous generation,” I examined one scenario in which technical debt formation occurs spontaneously—that is, in the absence of engineering activity. Specifically, I noted how the emergence of the HTML5 standard led to the formation of technical debt in some (if not all) existing Web sites. This happened because those sites didn’t exploit capabilities that had become available in HTML5. Moreover, some sites needed rehabilitation to remove emulations of the capabilities of the new standard. Those emulations needed to be replaced with use of facilities in the HTML5 standard. All of these artifacts—including those that existed, and those that didn’t—comprised technical debt. This scenario thus led to the formation of exogenous technical debt.
In a second example, AMUFC, A Made-Up Fictitious Corporation, incurs technical debt when the vendor that supplies the operating system (OS) for AMUFC’s desktop computers announces the date of the end of extended support for the version of the OS in use at AMUFC. Because the end of extended support brings an end to security updates, AMUFC must retire that debt by migrating to the next version of that vendor’s OS before extended support actually ends.
In both examples, the forces that lead to formation of exogenous technical debt are external to the enterprise and the enterprise’s assets. But what makes technical debt exogenous is that the forces that led to its formation are unrelated the engineering work being performed on the asset. This restriction is loose enough to also include technical debt that arises from any change or activity external to the asset, but within the enterprise.
Exogenous technical debt arising from actions within the enterprise
Exogenous technical debt can arise from activities or decisions that take place entirely within the enterprise.
For example, consider the line of mobile devices of AMUFC (A Made-Up Fictitious Corporation). Until this past year, AMUFC has been developing ever more capable devices. These efforts extended its line of offerings at the high end—the more expensive and capable members of the line. But this past quarter, AMUFC developed a low-end member of the line.
As often happens, price constraints for the low-cost device led to innovations. Those innovations could produce considerable savings in manufacturing costs if used all across the line. In effect, the designs of the previously developed higher-end models have incurred exogenous technical debt. The debt is exogenous because the activity that led to debt formation wasn’t performed on the assets that carry the debt. The debt is real, even though the activity that led to debt formation occurred within the enterprise. This kind of exogenous technical debt is asset-exogenous. Exogenous technical debt of the kind that results from activity beyond the enterprise is enterprise-exogenous.
Exogeneity versus endogeneity
For asset-exogenous technical debt, ambiguity between endogeneity and exogeneity can arise. The example above regarding the line of mobile devices produced by AMUFC provides an illustration.
For convenience, call the team that developed one of the high-end devices Team High. Call the team that developed the low-end device Team Low. From the perspective of Team High, the technical debt due to the innovations discovered by Team Low is exogenous. But from the perspective of the VP Mobile Devices, that same technical debt might be regarded as endogenous. The debt can be endogenous at VP level because it’s possible to regard the entire product line as a single asset, and that might actually be the preferred perspective of VP Mobile Devices.
This ambiguity can lead to some nasty toxic conflict. Team High and VP Mobile Devices might attack each other as they try to defend themselves proactively against claims that they are incurring technical debt. Avoiding this kind of conflict requires educating everyone as to the origins of technical debt.
Exogeneity and legacy technical debt
The technical debt portfolio of a given asset can contain a mix a technical debt that arose from various past incidents. In assessing the condition of the asset, it’s useful to distinguish this existing debt from debt that’s incurred as a consequence of any current activity or decisions. Call this pre-existing technical debt legacy technical debt.
The legacy technical debt an asset carries is technical debt associated with the asset, and which existed in any form before undertaking work on that asset. For example, consider planning a project to renovate the hallways and common areas of a high-rise apartment building. Suppose workers discover beneath the existing carpeting a layer of asbestos floor tile. Then Management might decide to remove the tile. In this context, we can regard the floor tile as legacy technical debt. It isn’t directly related to the objectives of the current renovation. But removing it will enhance the safety of future renovations. It will also enable certification of the building as asbestos-free, increase the property value, and reduce the cost of eventual demolition. In this situation asbestos removal is retirement of legacy technical debt. Accounting for it as part of the common-area renovation would be misleading.
Exogeneity is relevant when allocating resources for legacy technical debt retirement efforts. If the debt in question is enterprise-exogenous, we can justifiably budget the effort from enterprise-level accounts. For other cases, other resources become relevant, depending on what actions created the debt. For example, suppose that the technical debt arose from a change in enterprise standards. Then we can justifiably allocate retirement costs to the standard-setting initiative. If the exogenous technical debt arose from innovations in other members of the asset’s product line, we can can justifiably allocate those debt retirement costs to the product line.
Policy insights
Understanding the properties of exogenous technical debt can be a foundation for policy innovations that enhance enterprise agility.
Culture transformation
Widespread understanding of the distinction between exogenous and endogenous technical debt is helpful in controlling interpersonal conflict. For example, it can reduce blaming behavior that targets the engineering teams responsible for developing and maintaining technological assets.
Understanding asset-exogenous technical debt helps non-engineers understand how their actions and decisions can lead to technical debt formation. The concept clarifies the import of their actions even when there is no apparent direct connection between those actions or decisions and the assets in question.
Resource allocation
Data about the technical debt creation effects of enterprise activities is helpful in allocating technical debt retirement costs. For example, suppose that we know all the implications of reorganization, including its impact on internal data about the enterprise itself. Then we can charge data-related activity to the reorganization instead of to general accounts of the Information Technology function. This helps the enterprise understand the true costs of reorganization.
Similarly, data about enterprise-exogenous technical debt helps planners understand how to deploy resources to gather external intelligence about trends that can affect internal assets. Such data is also useful for setting levels of support and participation in industrial standards organizations or in lobbying government officials.
Last words
Knowing the formation history of exogenous technical debt provides useful guidance for those charged with allocating the costs of retiring technical debt or preventing its formation.
References
[APA 2013] American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association Publishing, 2013.
[Ariely 2010] Dan Ariely. “You are what you measure,” Harvard Business Review 88:6, p. 38, 2010.
This article is probably the source of the adage “You are what you measure.” Personally, I believe it’s overstated. That is, it’s true in the large, perhaps, but not in detail. Moreover, there are some things that we are that can’t be measured. But it’s important to understand the content of this article because so many people take it as dogma. Available: here; Retrieved: June 4, 2018
[Bouwers 2010] Eric Bouwers, Joost Visser, and Arie van Deursen. “Getting What You Measure: Four common pitfalls in using software metrics for project management,” ACM Queue 10: 50-56, 2012.
[Haque 2018] Md Shariful Haque, Jeff Carver, and Travis Atkison. "Causes, impacts, and detection approaches of code smell: a survey." Proceedings of the ACMSE 2018 Conference. ACM, 2018.
[Orton 1990] J. Douglas Orton and Karl E. Weick. “Loosely Coupled Systems: A Reconceptualization,” The Academy of Management Review, 15:2, 203-223, 1990.
[Parnas 1979] David L. Parnas. “Designing Software for Ease of Extension and Contraction,” IEEE Transactions on Software Engineering, vol. SE-5, no. 2, March 1979, 128-138.
[Sullivan 2001] Kevin J. Sullivan, William G. Griswold, Yuanfang Cai, and Ben Hallen. “The structure and value of modularity in software design,” in ACM SIGSOFT Software Engineering Notes, 26:5, 99-108, 2001.
