Tag: metrics

Posted on

Metrics for technical debt management: the basics

Last updated on July 11th, 2021 at 02:19 pm

Measuring tools needed to follow a recipe in the kitchen
Measuring tools needed to follow a recipe in the kitchen. The word “measurement” evokes images that relate to our earliest understanding of the word as children. For most of us, that involves determining the attributes of a physical thing. In most cases, the physical thing of most concern is our own body—its height and weight. But we also measure everyday things, as when following recipes. So the strongest associations we have with the word “measurement” involve physical things. “Measuring” attributes of an abstract construct like technical debt can be perilous. We must make allowances for its lack of physicality. Those allowances must guide us as we develop metrics for technical debt management.

Whether it’s wise to use metrics for technical debt management is an open question. But whether it will become a widespread practice does seem to be a settled question. Using metrics for technical debt management now does appear to be inevitable. So let’s explore just what we mean by metrics, and what traps might lie ahead when we use metrics for technical debt management.

The reification error

Skepticism about the effectiveness of using metrics for technical debt management is reasonable. Technical debt isn’t a physically measurable thing. “Measuring” technical debt is therefore susceptible to what psychologists call the reification error [Levy 2009]. Philosophers call it the Fallacy of Misplaced Concreteness [Whitehead 1948].

The logical fallacy of reification occurs when we treat an abstract construct as if it were a concrete thing. Although reification can provide helpful mental shorthand, it can produce costly cognitive errors. For example, advising someone who’s depressed to get more self-esteem is unlikely to work, because self-esteem isn’t something one can order from Amazon, or anywhere else. (I checked; all I could find were books and ebooks) One can enhance self-esteem through counseling, reflection, or many other means, but it isn’t a concrete object one can “get.” Self-esteem is an abstract construct.

Likewise, technical debt is an abstract construct. We can discuss “measuring” it, but attempts to specify measurement procedures will eventually confront the inherently abstract nature of technical debt. Those attempts lead to debates about both the definitions of technical debt and the measurement process.

Metrics inherently require some kind of collection of numeric data. That’s why skepticism about using metrics for technical debt management is a reasonable position. Reasonable or not, though, metrics will be used. We’d best be prepared to use them responsibly. That’s the focus of this post—and a few to come. If we use metrics for technical debt management, how can we do it responsibly? How can we manage the risks of reification?

This post is about metrics in general. In coming posts I’ll apply this line of thinking to specific examples of metrics for managing technical debt, and suggest approaches that could mitigate reification risk.

Foundations for behavior guidance decisions

The objective of technical debt management is support for behavior guidance decisions. Behavior guidance decisions are decisions that guide the behavior and choices of employees. In this case, the goal is controlling the volume of technical debt. Although many frameworks exist for supporting behavior guidance decisions, they generally consist of four elements:

Quantifiers

A quantifier is a specification for a measurement process designed to yield a numeric representation of some attribute of an asset or process. With respect to technical debt, we use quantifiers to prescribe how we produce data that represents the state of technical debt of an asset. We also use quantifiers to generate data that captures other related items, such as budgets, the cost or availability of human effort, revenue flows—almost anything that interacts with the assets whose debt burden we want to control.

An example of a quantifier is the process for estimating the MPrin of a particular kind of technical debt an asset carries. The MPrin quantifier definition includes an explicit procedure for measuring it. That is, it defines a procedure for estimating the size of the MPrin in advance of actually retiring that debt. After retirement, we know its value without estimating, because the MPrin is what we actually spent to complete the retirement.

Measures

A measure is the result of determining the value of a quantifier. For example, we might use a quantifier’s definition to determine how much human effort has been expended on an asset in the past fiscal quarter. Or we might use another quantifier’s definition to determine the current size of the MPrin the asset now carries.

Metrics

A metric is an arithmetic formula expressed in terms of constants and a set of measures. One of the simpler metrics consists of a single ratio of two measures. For example, the metric that captures the average cost of acquiring a new customer in the previous fiscal quarter is the ratio of two measures, namely, the investment made in acquiring new customers, and the number of new customers acquired.

Associated with some metrics is a defined set of actors (actual people) who have authority to take steps to affect the value of the metric in some desirable way. They might also have authority to direct others to take similar steps. Metrics that have defined sets of actors are usually Key Performance Indicators (see below). If more than one individual is a designated actor for a metric, there is a process for resolving differences among the designated actors about what action to take, if any. In some cases, this process is as simple as determining which designated actor has the highest organizational rank.

An example of a technical debt metric

An example of a technical debt metric is the ratio MPrin(i)/MPrin(r). MPrin(i) is the total of incremental technical debt incurred in the given time period. MPrin(r) is the total of MPrin retired in that same time period. In periods during which this ratio exceeds 1.0, the organization is accumulating incremental technical debt faster than it is retiring technical debt. Computing it as a ratio, as opposed to a difference, has the effect of expressing the increases (or decreases) in technical debt portfolio size in units of the size of the debt retired. This enables the organization to take on some incremental technical debt responsibly.

This metric also has the virtue of displaying meaningful trends in an easily recognized way. In this case, a steady upward trend means a steadily increasing debt burden, even if in some time periods the debt doesn’t increase much. In other words, the ratio removes some of the “choppiness” that might plague a metric expressed in terms of absolute values.

Key Performance Indicators

A Key Performance Indicator (KPI) is a metric that provides meaningful insight for guiding business decisions. All KPIs are metrics; not all metrics are KPIs.

The value of a KPI depends on one or more metrics. It represents how successful the business is in reaching a given business objective. A metric, on the other hand, represents only the degree of success in reaching a targeted value for that metric. The relationship between the target value of a metric and any given business objective can be complicated. It can also potentially involve other metrics. For these reasons, metrics that aren’t KPIs are less useful for indicating the degree of success in reaching business objectives.

MPrin(i)/MPrin(r) is a metric that could also serve as a KPI, if the business objective is steady declines in overall technical debt.

Dimensions of measure vs. dimensions of metrics

Some metrics, such as MPrin(i)/MPrin(r), are dimensionless. Their values are pure numbers. And some metrics have dimensions—units of measure. For example, let MPrin(i) be the volume of incremental technical debt a deliverable carries, and let Tdelay be the number of days delivery was late. Consider the metric the metric MPrin(i)·Tdelay. The dimension of this metric is Currency·Days. This metric is particularly interesting, because a common assertion about technical debt is that we incur it as a means of advancing project delivery.

The evidence for this assertion is mostly anecdotal. But actually determining the value of this metric over a number of projects might reveal useful information about the effectiveness of a common strategy. That strategy is to accept incurring technical debt as a means of advancing project delivery. Thus, we would expect to see small time delays associated with increased incremental technical debt. In other words, all projects of similar scale should lie along roughly the same hyperbola in a plot of this metric in a space in which one axis is debt and the other is the number of days of delayed delivery.

Units of measures are often different from the units of the metrics that those measures support. For example, measures of technical debt in code include test coverage, documentation asynchrony, documentation omissions, code duplication, code complexity, dependency cycles, rule violations, or interface violations. The units of these measures are all different.

Last words

To those who must make strategic technical debt management decisions by comparing the costs of retiring different kinds of debt, these detailed measures are awkward to use. MPrin is more directly related to the issues they must address. MPrin provides a unit of comparison among debt retirement options, and between retirement options and available resources. Beyond the level of the particular debt we are retiring, MPrin is the dimension of greatest utility. [Brown 2010]

In a future post I’ll describe the properties of metrics that are needed for technical debt management.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Other posts in this thread

Posted on

MICs can change when other debts are retired

Last updated on July 8th, 2021 at 12:49 pm

The MICs and MPrin of a particular class of technical debt can change when we retire other seemingly unrelated classes of technical debt. In most cases, we need engineering expertise to determine technical debt retirement strategies that can exploit this property.

Financial debts usually have associated interest rates that are used to compute the periodic interest charges. Typically, the interest charge on a financial debt for a given period is the periodic interest rate multiplied by the principal, and then scaled for the length of the time period. But there are no “rates” for technical debt. Their existence would imply that MICs were proportional to the analog of “principal,” which, in the case of technical debt, is the cost of retiring the debt—the MPrin. MICs depend only weakly on the cost of retiring the debt. Instead, they depend more strongly on the impact of the debt on ongoing operations.

How financial sophistication can be a handicap

Decision makers who understand the world of financial instruments at a very sophisticated level might tend to make an understandable error. They might overvalue arguments favoring technical debt management in ways analogous to how we manage financial debts. Financial sophisticates might find appealing any argument for technical debt management that parallels financial approaches. Such programs are unlikely to work, for two reasons. First, the uncertainties associated with estimating MPrin and MICs are significant. They make technical debt management decisions more dependent on engineering and project management judgment than they are on the results of calculations and projections (see MPrin uncertainties and MICs uncertainties).

Second, the familiar concept of interest rate doesn’t apply to technical debt. For technical debt, the interest charges depend on the interaction between ongoing activities and the debt, rather than the cost of retiring the debt. And that means that MICs (and MPrin) of one class of debt can change when another class is retired.

Implications of this effect

The possibility that retiring one class of technical debt can alter the financial burdens presented by another has both favorable and unfavorable implications.

MICs can change when other debts are retired
An example illustrating one way in which MICs on one kind of technical debt can change as a result of retiring another. The structure at the left represents the situation before any debt retirement occurs. The balloons labeled “A” represent instances of asset A. The balloon labeled “B” represents asset B. The orange circles represent instances of technical debts D1 and D2. The arrows connecting the As to B indicate that asset A depends on Asset B. The structure at the right represents the situation after debt retirement.
As an example of a favorable implication, consider software remodularization. Suppose we have a software asset A that depends on another software asset B. As shown in the left image of the figure, asset A, of which there are many copies, bears two classes of technical debt, D1 and D2. As shown, there is only one copy of asset B. Suppose further that an asset that bears debt D2 also bears debt D1, but an asset that bears D1 might or might not bear debt D2.

To retire D2, engineers modified B by assigning it responsibility for the tasks that formerly bore debt D2. After this change, B bears debt of type D1. Next, they retired debt D2. The right half of the figure shows the result. The system still bears debt D1. But now it’s inside B instead of A. Those modifications relieve all instances of A of both types of debt. The result is that D2 vanishes, and only a single instance of D1 remains. In this way, retiring debt D2 has reduced the MICs and MPrin for D1.

Policymakers can help

Exploiting the salutary opportunities of this property of technical debt provides an example of the risks of adhering too closely to the financial model of debt.

Many different scenarios have the property that retiring one kind of technical debt can reduce the MICs associated with other kinds. Technologists understandably tend to be more concerned with technical debt retirement strategies that emphasize short-term improvement of their own productivity. That’s why it’s so important for policymakers to provide guidance that steers the organization in the direction of enterprise benefits.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Related posts

  • The Principal Principle: Focus on MICs
    The Principal Principle is that a focus on the metaphorical principal of a technical debt can be your undoing. Focus on the metaphorical interest charges. Drive them to Zero and keep them there.
  • Where the misunderstandings about MICs come from
    Misunderstandings about the metaphorical interest charges on technical debt are costly. They prevent us from exploiting the properties of technical debt that reduce carrying costs and retirement costs. And the misunderstandings arise from the fact that the technical debt metaphor is only a metaphor—technical debt and financial debt are different.
  • MICs on technical debt can be difficult to measure
    Unlike financial debt, for technical debt there are no legally required reports or disclosures. We can sometimes estimate MICs, but most organizations don’t track the data necessary to estimate MICs with useful precision. Indeed, developing useful estimates is often technically impossible.
  • MICs can sometimes be deferred or advanced without penalty
    Rescheduling interest payments on financial debt is possible only by prearrangement or in bankruptcy, but MICs on technical debt can often be rescheduled by rescheduling work that might incur them. This is useful when we plan to retire assets bearing technical debt, because their technical debt vanishes.
  • MICs can differ for different instances of the same kind of technical debt
    For financial debts, the interest charges associated with a unit of debt are (usually) the same for every unit of debt incurred under the same loan agreement. But for technical debt, the MICs associated with a given instance of a class of technical debt might differ substantially from any other instance of the same class of technical debt, even if those instances of technical debt were incurred at the same time as a result of a single decision or sequence of events.
  • How financial interest charges differ from interest charges on technical debt
    The common understanding of interest on financial debts doesn’t correspond to the reality of technology-based systems, which are the targets of the technical debt metaphor. Formulating sound technical debt policy depends on understanding the nature of the difference between interest on financial debt and the metaphorical interest charges associated with technical debt.
Posted on

MICs on technical debt can be difficult to measure

Last updated on July 8th, 2021 at 12:48 pm

For financial debts, creditors regularly inform debtors of interest charges, principal remaining, and other loan parameters. Laws usually require reports and explicit statements about interest charges. By contrast, for technical debt, MICs can be difficult to compute with useful precision, even if we know they’re accumulating. Many decision makers are actually unaware that MICs are accumulating. For an organization to appreciate the full financial consequences of carrying technical debt, everyone in the enterprise must appreciate the concept of MICs.

The heart of the problem

A stack of floppy disks
A stack of floppy disks. You don’t see many of these around much anymore. Very little of the software or hardware we use is as obsolete as these floppies. But much of it is obsolete, and it therefore comprises technical debt. It still works, but it’s slow and probably out of support life. On the basis of speed alone, the MICs it incurs can easily justify replacement. And some of it is vulnerable to cyberattack. One significant breach can ruin a brand.
Unlike financial debt, for technical debt there are no legally required reports or disclosures. We can sometimes estimate MICs, but most organizations don’t track the data necessary to estimate MICs with useful precision. Indeed, developing useful estimates is often technically impossible.

The difficulty of measuring MICs arises from three sources. First, people whose productivity is most directly affected by technical debt—usually engineers—often have difficulty determining with precision the extent of the impact of technical debt on their efforts.

Second, many people are unaware of the impact technical debt has on their results. For example, if a product arrives late to market, we could compute the financial costs attributable to technical debt if we knew that technical debt is partially—or wholly—responsible for the delay. Too often, those who could perform such calculations aren’t sufficiently familiar with the concept of MICs. In any case, the data they would need for calculating a useful estimate is rarely available.

This brings us to the third source of difficulty.

The terrifying opportunity

Finally, a more insidious form of the consequences of technical debt is what we might call the terrifying opportunity. This situation arises when the organization rejects (or fails to recognize) a market opportunity because exploiting it would involve modifying an existing asset. The modification is terrifying because the asset harbors a heavy load of technical debt. The debt causes decision makers to assess that the probability of success is so low that the opportunity seems terrifying. They therefore reject the opportunity. Typically, terrifying opportunities would be exploitable if the debt-bearing assets didn’t exist at all. Then we would be starting fresh. But when opportunities require modifying assets that bear technical debt, commitment requires faith that we can address the technical debt successfully.

The sense of risk isn’t a reflection on the capabilities of the technical organization. Rather, it’s a result of the challenges involved in working with assets that bear high levels of technical debt. Given past performance of the technical organization relative to these debt-bearing assets, success can seem unlikely.

Computing the cost of a terrifying opportunity requires estimating the cost of not exploiting the opportunity, a difficult task in the best of circumstances. But whatever that cost is, it’s a form of MICs that we rarely recognize.

Last words

Building expertise in estimating MICs in all their forms is advantageous to any organization that seeks to make its technical debt more manageable. By making MICs visible, we can bring about better recognition of the cost of carrying technical debt, thereby providing an appropriate motivator for retiring technical debt.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Related posts

  • The Principal Principle: Focus on MICs
    The Principal Principle is that a focus on the metaphorical principal of a technical debt can be your undoing. Focus on the metaphorical interest charges. Drive them to Zero and keep them there.
  • Where the misunderstandings about MICs come from
    Misunderstandings about the metaphorical interest charges on technical debt are costly. They prevent us from exploiting the properties of technical debt that reduce carrying costs and retirement costs. And the misunderstandings arise from the fact that the technical debt metaphor is only a metaphor—technical debt and financial debt are different.
  • MICs can change when other debts are retired
    The metaphorical interest charges (MICs) and metaphorical principal (MPrin) of a particular class of technical debt can change as a result of retiring other seemingly unrelated classes of technical debt. In most cases, engineering expertise is required to determine technical debt retirement strategies that can exploit this property of technical debt.
  • MICs can sometimes be deferred or advanced without penalty
    Rescheduling interest payments on financial debt is possible only by prearrangement or in bankruptcy, but MICs on technical debt can often be rescheduled by rescheduling work that might incur them. This is useful when we plan to retire assets bearing technical debt, because their technical debt vanishes.
  • MICs can differ for different instances of the same kind of technical debt
    For financial debts, the interest charges associated with a unit of debt are (usually) the same for every unit of debt incurred under the same loan agreement. But for technical debt, the MICs associated with a given instance of a class of technical debt might differ substantially from any other instance of the same class of technical debt, even if those instances of technical debt were incurred at the same time as a result of a single decision or sequence of events.
  • How financial interest charges differ from interest charges on technical debt
    The common understanding of interest on financial debts doesn’t correspond to the reality of technology-based systems, which are the targets of the technical debt metaphor. Formulating sound technical debt policy depends on understanding the nature of the difference between interest on financial debt and the metaphorical interest charges associated with technical debt.
Show Buttons
Hide Buttons