Last updated on July 11th, 2021 at 02:19 pm
Whether it’s wise to use metrics for technical debt management is an open question. But whether it will become a widespread practice does seem to be a settled question. Using metrics for technical debt management now does appear to be inevitable. So let’s explore just what we mean by metrics, and what traps might lie ahead when we use metrics for technical debt management.
The reification error
Skepticism about the effectiveness of using metrics for technical debt management is reasonable. Technical debt isn’t a physically measurable thing. “Measuring” technical debt is therefore susceptible to what psychologists call the reification error [Levy 2009]. Philosophers call it the Fallacy of Misplaced Concreteness [Whitehead 1948].
The logical fallacy of reification occurs when we treat an abstract construct as if it were a concrete thing. Although reification can provide helpful mental shorthand, it can produce costly cognitive errors. For example, advising someone who’s depressed to get more self-esteem is unlikely to work, because self-esteem isn’t something one can order from Amazon, or anywhere else. (I checked; all I could find were books and ebooks) One can enhance self-esteem through counseling, reflection, or many other means, but it isn’t a concrete object one can “get.” Self-esteem is an abstract construct.
Likewise, technical debt is an abstract construct. We can discuss “measuring” it, but attempts to specify measurement procedures will eventually confront the inherently abstract nature of technical debt. Those attempts lead to debates about both the definitions of technical debt and the measurement process.
Metrics inherently require some kind of collection of numeric data. That’s why skepticism about using metrics for technical debt management is a reasonable position. Reasonable or not, though, metrics will be used. We’d best be prepared to use them responsibly. That’s the focus of this post—and a few to come. If we use metrics for technical debt management, how can we do it responsibly? How can we manage the risks of reification?
This post is about metrics in general. In coming posts I’ll apply this line of thinking to specific examples of metrics for managing technical debt, and suggest approaches that could mitigate reification risk.
Foundations for behavior guidance decisions
The objective of technical debt management is support for behavior guidance decisions. Behavior guidance decisions are decisions that guide the behavior and choices of employees. In this case, the goal is controlling the volume of technical debt. Although many frameworks exist for supporting behavior guidance decisions, they generally consist of four elements:
Quantifiers
A quantifier is a specification for a measurement process designed to yield a numeric representation of some attribute of an asset or process. With respect to technical debt, we use quantifiers to prescribe how we produce data that represents the state of technical debt of an asset. We also use quantifiers to generate data that captures other related items, such as budgets, the cost or availability of human effort, revenue flows—almost anything that interacts with the assets whose debt burden we want to control.
An example of a quantifier is the process for estimating the MPrin of a particular kind of technical debt an asset carries. The MPrin quantifier definition includes an explicit procedure for measuring it. That is, it defines a procedure for estimating the size of the MPrin in advance of actually retiring that debt. After retirement, we know its value without estimating, because the MPrin is what we actually spent to complete the retirement.
Measures
A measure is the result of determining the value of a quantifier. For example, we might use a quantifier’s definition to determine how much human effort has been expended on an asset in the past fiscal quarter. Or we might use another quantifier’s definition to determine the current size of the MPrin the asset now carries.
Metrics
A metric is an arithmetic formula expressed in terms of constants and a set of measures. One of the simpler metrics consists of a single ratio of two measures. For example, the metric that captures the average cost of acquiring a new customer in the previous fiscal quarter is the ratio of two measures, namely, the investment made in acquiring new customers, and the number of new customers acquired.
Associated with some metrics is a defined set of actors (actual people) who have authority to take steps to affect the value of the metric in some desirable way. They might also have authority to direct others to take similar steps. Metrics that have defined sets of actors are usually Key Performance Indicators (see below). If more than one individual is a designated actor for a metric, there is a process for resolving differences among the designated actors about what action to take, if any. In some cases, this process is as simple as determining which designated actor has the highest organizational rank.
An example of a technical debt metric
An example of a technical debt metric is the ratio MPrin(i)/MPrin(r). MPrin(i) is the total of incremental technical debt incurred in the given time period. MPrin(r) is the total of MPrin retired in that same time period. In periods during which this ratio exceeds 1.0, the organization is accumulating incremental technical debt faster than it is retiring technical debt. Computing it as a ratio, as opposed to a difference, has the effect of expressing the increases (or decreases) in technical debt portfolio size in units of the size of the debt retired. This enables the organization to take on some incremental technical debt responsibly.
This metric also has the virtue of displaying meaningful trends in an easily recognized way. In this case, a steady upward trend means a steadily increasing debt burden, even if in some time periods the debt doesn’t increase much. In other words, the ratio removes some of the “choppiness” that might plague a metric expressed in terms of absolute values.
Key Performance Indicators
A Key Performance Indicator (KPI) is a metric that provides meaningful insight for guiding business decisions. All KPIs are metrics; not all metrics are KPIs.
The value of a KPI depends on one or more metrics. It represents how successful the business is in reaching a given business objective. A metric, on the other hand, represents only the degree of success in reaching a targeted value for that metric. The relationship between the target value of a metric and any given business objective can be complicated. It can also potentially involve other metrics. For these reasons, metrics that aren’t KPIs are less useful for indicating the degree of success in reaching business objectives.
MPrin(i)/MPrin(r) is a metric that could also serve as a KPI, if the business objective is steady declines in overall technical debt.
Dimensions of measure vs. dimensions of metrics
Some metrics, such as MPrin(i)/MPrin(r), are dimensionless. Their values are pure numbers. And some metrics have dimensions—units of measure. For example, let MPrin(i) be the volume of incremental technical debt a deliverable carries, and let Tdelay be the number of days delivery was late. Consider the metric the metric MPrin(i)·Tdelay. The dimension of this metric is Currency·Days. This metric is particularly interesting, because a common assertion about technical debt is that we incur it as a means of advancing project delivery.
The evidence for this assertion is mostly anecdotal. But actually determining the value of this metric over a number of projects might reveal useful information about the effectiveness of a common strategy. That strategy is to accept incurring technical debt as a means of advancing project delivery. Thus, we would expect to see small time delays associated with increased incremental technical debt. In other words, all projects of similar scale should lie along roughly the same hyperbola in a plot of this metric in a space in which one axis is debt and the other is the number of days of delayed delivery.
Units of measures are often different from the units of the metrics that those measures support. For example, measures of technical debt in code include test coverage, documentation asynchrony, documentation omissions, code duplication, code complexity, dependency cycles, rule violations, or interface violations. The units of these measures are all different.
Last words
To those who must make strategic technical debt management decisions by comparing the costs of retiring different kinds of debt, these detailed measures are awkward to use. MPrin is more directly related to the issues they must address. MPrin provides a unit of comparison among debt retirement options, and between retirement options and available resources. Beyond the level of the particular debt we are retiring, MPrin is the dimension of greatest utility. [Brown 2010]
In a future post I’ll describe the properties of metrics that are needed for technical debt management.
References
[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.
Available: here; Retrieved: July 30, 2018
[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.
[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].
Other posts in this thread
- Managing technical debt
- Leverage points for technical debt management
- Undercounting nonexistent debt items
- Crowdsourcing debt identification
- Demodularization can help control technical debt
- Legacy debt incurred intentionally
- Accounting for technical debt
- Three cognitive biases
- The resilience error and technical debt
- Synergy between the reification error and confirmation bias
- Retiring technical debt can be a wicked problem
- Retiring technical debt can be a super wicked problem
- Degrees of wickedness
- Metrics for technical debt management: the basics
