Exogenous technical debt

Last updated on July 24th, 2018 at 07:30 pm

Mastering understanding of exogenous technical debt—debt that arises from causes not directly related to the asset that bears the debt—is essential to controlling technical debt formation. Exogenous technical debt is particularly troublesome to those who work on the affected assets. They can’t control its formation, and they’re rarely responsible for creating it. But their internal customers and those who control resources often fail to understand this. Indeed, those who work on the affected assets are often blamed for the formation of exogenous technical debt even though they had no role in its formation, and could have done nothing to prevent its formation.

Asbestos with muscovite.
Asbestos with muscovite. Asbestos is a family of six minerals that occur naturally in fibrous form. The fibers are all known carcinogens. Until 1990, it was widely used in many common building materials, including insulation, plaster, and drywall joint compound. It is now banned, but it’s present in many existing structures, including homes and offices. The installation of the ban caused these structures to incur exogenous technical debt. Photo by Aramgutang courtesy Wikipedia.

Technical debt is exogenous when it’s brought about by an activity not directly related to the assets in which the debt appears. The word exogenous comes from the Greek exo– (outside) + –genous (related to producing). So exogenous technical debt is that portion of an asset’s debt that comes about from activities or decisions that don’t involve the asset directly.

Because so much technical debt is produced indirectly, controlling its direct formation—for example, by engineering teams—isn’t sufficient for achieving enterprise control of technical debt formation. To control technical debt formation, we must track which activities produce it, including both direct and indirect effects. Allocating technical debt retirement costs to the activities that brought that debt about, even if the allocation doesn’t affect budget authority for those activities, is therefore a useful practice. Knowledge about which past activities created technical debt, and how much, is helpful for long-term reduction in the rate of technical debt formation.

When we think of technical debt, we tend to think of activities that produce it relatively directly. We often imagine it as resulting solely from engineering activity, or from decisions not to undertake engineering activity. In either case the activity involved, whether undertaken or not, is activity directly involving the asset that carries—or will be carrying—the technical debt. This kind of technical debt is endogenous technical debt. The word endogenous comes from the Greek endo– (within or inside) + –genous (related to producing).  So endogenous technical debt is that portion of an asset’s debt that comes about from activities or decisions that directly involve the asset.

More about endogenous technical debt in future posts. For now, let’s look more closely at exogenous technical debt, and its policy implications.

Examples of exogenous technical debt

In “Spontaneous generation,” I examined one scenario in which technical debt formation occurs spontaneously—that is, in the absence of engineering activity. Specifically, I noted how the emergence of the HTML5 standard led to the formation of technical debt in some (if not all) existing Web sites, in the sense that they didn’t exploit capabilities that had become available in HTML5. Moreover, some sites whose developers had elected to emulate capabilities of the new standard by exploiting alternative technologies needed rehabilitation to remove the emulation and replace it with use of facilities in the HTML5 standard. All of these artifacts—including those that existed, and those that didn’t—comprised technical debt. This scenario thus led to the formation of exogenous technical debt.

In a second example, AMUFC, A Made-Up Fictitious Corporation, incurs technical debt when the vendor that supplies the operating system (OS) for AMUFC’s desktop computers announces the date of the end of extended support for the version of the OS in use at AMUFC. Because the end of extended support brings an end to security updates, AMUFC must retire that debt by migrating to the next version of that vendor’s OS before extended support actually ends.

In both of these examples, the forces that lead to formation of exogenous technical debt are external to both the enterprise and the enterprise’s assets. But what makes technical debt exogenous is that the forces that led to its formation are unrelated to any of the engineering work being performed on the asset that carries the debt. This restriction is loose enough to also include technical debt that arises from any change or activity external to the asset, but within the enterprise.

Exogenous technical debt arising from actions within the enterprise

Exogenous technical debt can arise from activities or decisions that take place entirely within the enterprise.

For example, consider a line of mobile devices developed and marketed by AMUFC (A Made-Up Fictitious Corporation). Until this past year, AMUFC has been developing ever more capable devices, thereby extending its line of offerings at the high end—the more expensive and capable members of the line. But this past quarter, AMUFC developed a low-end member of the line, and as often happens, price constraints led to innovations that could produce considerable savings in manufacturing costs if those innovations were applied to all members of the line. In effect, then, the designs of the previously developed models in this line of devices have incurred exogenous technical debt. The debt is exogenous because the activity that led to debt formation was not performed on the assets that now carry the debt, even though the activity that led to debt formation occurred within the enterprise. This kind of exogenous technical debt might be termed asset-exogenous. Exogenous technical debt of the kind that’s incurred by activity beyond the enterprise might be termed enterprise-exogenous.

Exogeneity versus endogeneity

For asset-exogenous technical debt, ambiguity between endogeneity and exogeneity can arise. The example above regarding the line of mobile devices produced by AMUFC provides an illustration.

For convenience, call the team that developed one of the high-end devices Team High. Call the team that developed the low-end device Team Low. From the perspective of Team High, the technical debt due to the innovations discovered by Team Low is exogenous. But from the perspective of the VP Mobile Devices, that same technical debt might be regarded as endogenous. The debt can be endogenous at VP level because it’s possible to regard the entire product line as a single asset, and that might actually be the preferred perspective of VP Mobile Devices.

Exogeneity and legacy technical debt

The technical debt portfolio of a given asset can contain a mix a technical debt that arose from various past incidents. In assessing the condition of the asset, it’s useful to distinguish this existing debt from debt that’s incurred as a consequence of any current activity or decisions. Call this pre-existing technical debt legacy technical debt.

The legacy technical debt carried by an asset is technical debt associated with that asset, and which exists in that asset in any form prior to undertaking work on that asset. For example, in planning a project to renovate the hallways and common areas of a high-rise apartment building, workers discover that beneath the existing carpeting is a layer of floor tile containing asbestos. Management has decided to remove the tile. In this context, the floor tile can be viewed as legacy technical debt. It isn’t directly related to the objectives of the current renovation, but removing it will enhance the safety of future renovations, enable certification of the building as asbestos-free, increase the property value, and reduce the cost of eventual demolition. In this situation asbestos removal amounts to retirement of legacy technical debt, and accounting for it as part of the common-area renovation would be misleading.

When contemplating efforts to retire legacy technical debt, exogeneity becomes a factor in allocating the necessary resources. If the debt in question is enterprise-exogenous, then we can justifiably budget the effort from enterprise-level accounts if appropriate. For other cases, other pools of resources become relevant depending on what actions created the debt. For example, if the exogenous technical debt arose because of a departmental change in standards, debt retirement costs can justifiably be allocated to the standards effort. If the exogenous technical debt arose from innovations in other members of the asset’s product line, those debt retirement costs can justifiably be allocated to the product line.

Policy insights

Understanding the properties of exogenous technical debt can be a foundation for policy innovations that enhance enterprise agility.

Culture transformation

Widespread understanding the distinction between exogenous and endogenous technical debt is helpful in controlling blaming behavior that targets the engineering teams responsible for developing and maintaining technological assets.

Understanding of asset-exogenous technical debt helps non-engineers understand how their actions and decisions can lead to technical debt formation, even when there is no apparent direct connection between those actions or decisions and the assets in question.

Resource allocation

Data about the technical debt creation effects of enterprise activities is helpful in allocating technical debt retirement costs. For example, when we know all the implications of reorganization, including its impact on internal data about the enterprise itself, we can charge data-related activity to the reorganization instead of to general accounts of the Information Technology function. This helps the enterprise understand the true costs of reorganization.

Similarly, data about enterprise-exogenous technical debt helps planners understand how to deploy resources to gather external intelligence about trends that can affect internal assets. Such data is also useful for setting levels of support and participation in industrial standards organizations or in lobbying government officials.

Knowing the formation history of exogenous technical debt provides useful guidance for those charged with allocating the costs of retiring technical debt or preventing its formation.

Other posts in this thread

Spontaneous generation

Last updated on June 3rd, 2018 at 04:09 pm

Technical debt needn’t result from anyone’s conscious decision. In some instances, technical debt seems to appear as if by spontaneous generation. And that creates problems for technical debt management programs that assume that technical debt results from employee decisions in the form of the intentions of engineers or others.

U.S. Army soldiers, along with volunteers from the community, install roof trusses for a Habitat for Humanity home in Brainerd, Minn., July 13, 2010
U.S. Army soldiers, along with volunteers from the community, install roof trusses for a Habitat for Humanity home in Brainerd, Minnesota, July 13, 2010. Hurricane ties are in place at the top of the wall as the roof trusses are being placed. The Florida state building code was strengthened in 2002 to require hurricane ties to strengthen the connection between the roofs and walls of buildings. Homes built before 2002, and which lack hurricane ties, are therefore carrying technical debt. Retiring that debt is difficult. It involves retrofitting hurricane ties which usually requires cutting holes in the home’s siding—one for each tie—and then repairing the holes. Photo by Sgt. Nicholas Olson, courtesy  Wikimedia Commons, where you can find a larger version of this image.
Although knowing author or engineer intention relative to technical debt artifacts can be helpful when organizations plan or execute technical debt retirement programs, sound technical debt management policy must address situations in which author or engineer intention wasn’t a contributing factor in creating the debt, or intention can’t be determined, or intention is concealed. Classifications of technical debt must therefore consider business strategy and resource availability as well as author intention.

This difference in priorities contributes to tension between technologists and policymakers with respect to their definitions of technical debt.

Within enterprises of significant size, classifying technical debt is an essential step in designing programs for reducing the cost of carrying technical debt. Although the choice of classification scheme depends on one’s objectives, most classification schemes explored so far in the literature of technical debt are more suitable for use by technologists than by policymakers. But, unsurprisingly, the assistance they provide to policymakers relates mostly to policies that affect technologist behavior or resource allocation within the space of technical activities.

An example may clarify the issue. Technologists tend to create classifications of technical debt that emphasize author intentions. For example, Fowler has created a widely accepted two-dimensional [Lowy 2004] classification [Fowler 2009] that characterizes technical debt according to the Degree of Wisdom in incurring it (he calls this dimension Reckless/Prudent), and Degree of Intentionality in incurring it (he calls this dimension Deliberate/Inadvertent). See “Technical debt in software engineering” for more.

This classification, and another due to McConnell [McConnell-slides 2013] are widely accepted in the technical literature—widely, but not universally. For example, some believe that no artifact can be deemed technical debt unless its presence (or absence) was the result of a conscious decision [Adobe Blogs 2014]. Some adherents of this view would reject all of Fowler’s “Inadvertent” technical debt.

This focus on engineering intention likely arises, in part, for two reasons. First, technologists tend to have good access to their own intentions, and to the intentions of other technologists. Second, knowledge of the intentions of the people who created (or omitted) the artifacts in question can be helpful to technologists as they develop plans for retiring particular classes of technical debt.

For policymakers, both of these widely accepted classifications, while helpful, are inadequate. Intentionality with respect to technical debt formation is indeed a valuable consideration in developing technical debt policy, but because technical debt can arise for reasons unrelated to engineers’ intentions—indeed, it can arise for reasons unrelated to any enterprise activity—intention-based classifications provide inadequate guidance for policy formation.

Consider technological advancement that arises from sources external to the enterprise. For example, with the emergence of the HTML5 standard, many Web sites became obsolete, in the sense that they didn’t exploit capabilities that had become available. These sites were in need of updating to remain competitive in their markets. And capabilities that emulated the new standard, but which exploited alternative technologies, needed to be replaced. All of these artifacts—including those that existed, and those that didn’t, comprise technical debt.

Relative to technical debt management, an enterprise that devotes resources to monitoring external technology trends would have an advantage over competitors that tend to focus solely on employee behavior.

Technological advancement that occurs outside the enterprise can thus create technical debt within the enterprise. This is just one example of spontaneous generation of technical debt. Thinking about technical debt this way, you can probably identify other sources of spontaneous generation. Together, they create a need for policies that can address their management.

References

[Adobe Blogs 2014] Adobe Blogs. “What is Technical Debt?,” Adobe Blogs, September 8, 2014.

Available: here; Retrieved February 26, 2017.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Lowy 2004] A. Lowy and P. Hood. The Power of the 2x2 Matrix: Using 2x2 Thinking to Solve Business Problems and Make Better Decisions. Jossey-Bass, 2004.

Order from Amazon

Cited in:

[McConnell-slides 2013] Steve McConnell. “Managing Technical Debt”, ICSE 2013.

Available: here; Retrieved November 11, 2017

Cited in:

Other posts in this thread

Adopt an enterprise-wide definition of technical debt

An enterprise-wide definition of technical debt is essential because effective technical debt management requires cooperation from almost everyone. Absent a shared definition of technical debt, controversy can develop, especially among those who have previously encountered the concept—namely, among technologists. Policymakers can make invaluable contributions to the design of the cultural transformation that will enable control of technical debt.

A physical expression of shared commitment, essential to adopting an enterprise-wide definition of technical debt
A physical expression of shared commitment. Effective management of technical debt requires both a shared understanding of what it is and a shared commitment to do what’s required to get control of it.

Li et al. [Li 2015] found that defining what is and what is not technical debt remains a question at issue in the software engineering literature. Even if we restrict the discussion to software constructed in-house, opinions about what constitutes technical debt do differ. The authors found that in the literature of technical debt, “The term ‘debt’ has been used in different ways by different people, which leads to ambiguous interpretation of the term.”

This finding is perhaps the most significant for policymakers, because it suggests that implementing a technical debt management regime will require forging an organizational consensus about the meaning of the term technical debt. The people of most organizations come from a broad array of different backgrounds. Some have little knowledge of technical debt, and therefore have no preconceptions. But those who are aware of the issue, who are mainly technologists and their managers, probably interpret the term technical debtin a variety of ways. Because some of those who do have awareness of the term are likely to have strong opinions about its meaning, one can anticipate a need to resolve these differences of opinion early in the effort to gain control of technical debt.

Some technical terms, like RAID,byte, compiler, and kilowatt,have standard definitions that are widely accepted. Although the term technical debthas found wide use, there is no standard definition for it. What some people categorize as technical debt, others do not. Those who are accustomed to working with terms that have precise, widely accepted definitions might tend to assume that the term technical debtdoes have (or should have) one as well. This assumption can create some difficulty for people who do not realize that others might not share their views as to the definition of the term.

Policymakers must be aware that there is a lack of consensus about the definition of technical debt. Our definition, crafted specially for the use of policymakers, might seem unusually broad to technologists and engineers. For that reason alone, it’s advisable to become familiar with the various ways the term is used by technologists, because understanding their perspective is essential to formulating sound policy deserving of their respect.

References

[Adobe Blogs 2014] Adobe Blogs. “What is Technical Debt?,” Adobe Blogs, September 8, 2014.

Available: here; Retrieved February 26, 2017.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Li 2015] Z. Li, P. Avgeriou, and P. Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[Lowy 2004] A. Lowy and P. Hood. The Power of the 2x2 Matrix: Using 2x2 Thinking to Solve Business Problems and Make Better Decisions. Jossey-Bass, 2004.

Order from Amazon

Cited in:

[McConnell-slides 2013] Steve McConnell. “Managing Technical Debt”, ICSE 2013.

Available: here; Retrieved November 11, 2017

Cited in:

Other posts in this thread

Tension between policymakers and technologists

Last updated on July 24th, 2018 at 08:17 pm

Tension between policymakers and technologists, which can manifest as misalignment of their respective priorities, is a significant contributor to uncontrolled growth of technical debt. In this thread I explore sources of this tension and introduce concepts that can assist policymakers and technologists in their efforts to control the growth of technical debt.

Effective, sustainable control of technical debt is the objective of technical debt management policy. In an enterprise that has achieved this objective, technical debt serves as a strategic tool that assists in attaining and maintaining market leadership. In such an organization, technical debt does exist, and some legacy technical debt might remain in place, but technical debt growth is managed strategically, if growth occurs at all. Any technical debt that carries significant MICs, and which compromises productivity and enterprise agility, is addressed and retired with appropriate priority. In short, technical debt is addressed not solely as a technological issue, but as a component of business strategy.

Raindrops on a grapevine
Raindrops on a grapevine. We often think of tension as a negative, destructive force. But tension—in the case of raindrops, surface tension—is what holds a raindrop together. Tension gives the raindrop structure and integrity. The tension between technologists and policymakers can also be constructive. It can ensure that the enterprise manages its technical debt in ways that balance the political imperatives of both technology and strategic health.

This stance is at odds with the historical position most enterprises have adopted vis-à-vis technical debt. Historically, technical debt has been seen as a technical problem, if it has been recognized at all. Most enterprises have left the management of technical debt to the technologists. Frequently, then, the policymaker who enters the discussion about technical debt might be seen by technologists as an interloper, arriving late to the discussion, or as a less-than-knowledgeable invader attempting to seize control of a piece of the technologists’ domain. Tensions can arise between policymakers and technologists. Such tensions complicate the problem of managing technical debt.

One possible source of this tension is revealed in a study of the literature of technical debt, which is evolving so rapidly that it has itself become a focus for research. Li et al. [Li 2015] have produced a review of the software engineering technical debt literature, from which we can extract insights useful to policymakers. Although they studied only the literature relating to technical debt in software engineering, their conclusions are, at least in part, applicable to any field in which the components of the finished product are executed within software tools before being committed to operational form. This covers a wide array of knowledge-intensive endeavors, including mechanical system design, electronic design, framing of legislation, process design, architecture, and even book authorship.

In this thread, I explore the sources of the tension between the modern reality of technical debt as an enterprise issue, and the historical situation of technical debt as a technological issue. This can serve as a guide for policymakers in reframing technical debt from a technological issue dependent for resolution on enterprise resources, to an enterprise issue dependent for resolution on technological resources.

References

[Adobe Blogs 2014] Adobe Blogs. “What is Technical Debt?,” Adobe Blogs, September 8, 2014.

Available: here; Retrieved February 26, 2017.

Cited in:

[Fowler 2009] Martin Fowler. “Technical Debt Quadrant.” Martin Fowler (blog), October 14, 2009.

Available here; Retrieved January 10, 2016.

Cited in:

[Li 2015] Z. Li, P. Avgeriou, and P. Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[Lowy 2004] A. Lowy and P. Hood. The Power of the 2x2 Matrix: Using 2x2 Thinking to Solve Business Problems and Make Better Decisions. Jossey-Bass, 2004.

Order from Amazon

Cited in:

[McConnell-slides 2013] Steve McConnell. “Managing Technical Debt”, ICSE 2013.

Available: here; Retrieved November 11, 2017

Cited in:

Other posts in this thread