Tag: metrics

Posted on

Metrics for technical debt management: the basics

Last updated on September 16th, 2018 at 05:13 pm

Whether it is wise to use metrics for technical debt management is an open question. But whether it will become a widespread practice does seem to be a settled question. Using metrics for technical debt management now does appear to be inevitable. So let’s explore just what we mean by “metrics,” and what traps might lie ahead when we use them for technical debt management.

Measuring tools needed to follow a recipe in the kitchen
Measuring tools needed to follow a recipe in the kitchen. The word “measurement” evokes images that relate to our earliest understanding of the word as children. For most of us, that involves determining the attributes of a physical thing. In most cases, the physical thing of most concern is our own body — its height and weight. But we also measure everyday things, as when following recipes. So the strongest associations with the word “measurement” involve physical things. “Measuring” attributes of an abstract construct like technical debt can be perilous, unless we make allowances for its lack of physicality.

Skepticism about the effectiveness of using metrics for technical debt management is reasonable, because technical debt isn’t a physically measurable thing. “Measuring” technical debt is therefor susceptible to what psychologists call the reification error [Levy 2009] or what philosophers call the Fallacy of Misplaced Concreteness [Whitehead 1948].

The logical fallacy of reification occurs when we treat an abstract construct as if it were a concrete thing. Although reification can provide helpful mental shorthand, it can produce costly cognitive errors. For example, advising someone who’s depressed to get more self-esteem is unlikely to work, because self-esteem isn’t something one can order from Amazon, or anywhere else. One can enhance self-esteem through counseling, reflection, or many other means, but it isn’t a concrete object one can “get.” Self-esteem is an abstract construct.

Technical debt is likewise an abstract construct. We can discuss “measuring” it, but attempts to specify measurement procedures will eventually confront the inherently abstract nature of technical debt, leading to debates about both definitions and the measurement process.

Metrics inherently require some kind of measurement. That’s why skepticism about using metrics for technical debt management is a reasonable position. Reasonable or not, though, metrics will be used. We’d best be prepared to use them responsibly. That’s the focus of this post — and a few to come. If we use metrics for technical debt management, how can we do it responsibly? How can we manage the risks of reification?

This post is about metrics in general. In coming posts I’ll apply this line of thinking to specific examples of metrics for managing technical debt, and suggest approaches that could mitigate reification risk.

Foundations for behavior guidance decisions

The objective of technical debt management is support for behavior guidance decisions — decisions that guide the behavior and choices of employees so as to control the volume of technical debt. Although many frameworks exist for supporting behavior guidance decisions, they generally consist of four elements:

Quantifiers

A quantifier is a specification for a measurement process designed to yield a numeric representation of some attribute of an asset or process. With respect to technical debt, we use quantifiers to prescribe how we produce data that represents the state of technical debt of an asset. We also use quantifiers to generate data that captures other related items, such as budgets, the cost or availability of human effort, revenue flows — almost anything that interacts with the assets whose debt burden we want to control.

An example of a quantifier is the process for estimating the MPrin of a particular kind of technical debt borne by an asset. The MPrin quantifier definition includes an explicit procedure for measuring it — that is, for estimating the size of the MPrin in advance of actually retiring that debt. After retirement, we know its value without estimating, because the MPrin is what we spent to complete the retirement.

Measures

A measure is the result of determining the value of a quantifier. For example, we might use a quantifier’s definition to determine how much human effort has been expended on an asset in the past fiscal quarter. Or we might use another quantifier’s definition to determine the current size of the MPrin carried by the asset.

Metrics

A metric is an arithmetic formula expressed in terms of constants and a set of measures. One of the simpler metrics consists of a single ratio of two measures. For example, the metric that captures the average cost of acquiring a new customer in the previous fiscal quarter is the ratio of two measures, namely, the investment made in acquiring new customers, and the number of new customers acquired.

Associated with some metrics is a defined set of actors (actual people) who are authorized to take steps — or who are authorized to direct others to take steps — designed to affect the value of the metric in some desirable way. Metrics that have defined sets of actors are usually Key Performance Indicators (see below). If more than one individual is a designated actor for a metric, a process is defined to resolve differences among the designated actors about what action to take, if any. In some cases, this process is as simple as determining which designated actor has the highest organizational rank.

An example of a technical debt metric is the ratio MPrin(i)/MPrin(r) = the total of incremental technical debt (MPrin(i)) incurred in the given time period divided by the total of MPrin retired (MPrin(r)) in that same time period. In periods during which this ratio exceeds 1.0, the organization is accumulating incremental technical debt faster than it is retiring technical debt. Computing it as a ratio, as opposed to a difference, has the effect of expressing the increases (or decreases) in technical debt portfolio size in units of the size of the debt retired. This enables the organization to take on some incremental technical debt responsibly.

This metric also has the virtue of displaying meaningful trends in an easily recognized way. In this case, a steady upward trend means a steadily increasing debt portfolio, even if in some time periods the debt doesn’t increase much. In other words, the ratio removes some of the “choppiness” that might plague a metric expressed in terms of absolute values.

Key Performance Indicators

A Key Performance Indicator (KPI) is a metric that provides meaningful insight that’s used to guide business decisions. All KPIs are metrics; not all metrics are KPIs.

A KPI is derived from one or more metrics. It represents how successful the business is in accomplishing a given business objective.  A metric, on the other hand, represents only the degree of success in reaching a targeted value for that metric. Because the relationship between the target value of a metric and any given business objective can be complicated, and potentially involve other metrics, metrics that aren’t KPIs are of less value in indicating the degree of success in achieving business objectives.

MPrin(i)/MPrin(r) is a metric that could also serve as a KPI, if the business objective is to achieve steady declines in overall technical debt.

Dimensions of measure vs. dimensions of metrics

Some metrics, such as MPrin(i)/MPrin(r), are dimensionless. Their values are pure numbers. And some metrics have dimensions — units of measure. For example, consider the metric MPrin(i)*Tdelay, where MPrin(i) is the volume of incremental technical debt borne by the deliverable, and Tdelay is the number of days after the target date that the project was delivered. The dimension of this metric is Currency*Days. This metric is particularly interesting, because a common assertion about technical debt is that we incur it as a means of advancing project delivery. Because the evidence for this assertion is mostly anecdotal, actually determining the value of this metric over a number of projects might reveal useful information about the effectiveness of the strategy of incurring technical debt as a means of advancing project delivery. Thus, we would expect to see small time delays associated with increased incremental technical debt. In other words, all projects of similar scale should lie along the same hyperbola in a plot of this metric in a space in which one axis is debt and the other is the number of days after the target date that the project was delivered.

Units of measures are often different from the units of the metrics that those measures support. For example, measures of technical debt in code include test coverage, documentation asynchrony, documentation omissions, code duplication, code complexity, dependency cycles, rule violations, or interface violations. The units of these measures are all different.

To those who must make strategic technical debt management decisions by comparing the costs of retiring different kinds of debt, these detailed measures are awkward to use. MPrin is more directly related to the issues they must address. MPrin provides a unit of comparison among debt retirement options, and between retirement options and available resources. Beyond the level of the particular debt being considered for retirement, MPrin is the dimension of greatest utility. [Brown 2010]

In a future post I’ll describe the properties of metrics that are needed for technical debt management.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Other posts in this thread

Posted on

MICs can change when other debts are retired

Last updated on December 22nd, 2017 at 12:17 pm

The metaphorical interest charges (MICs) and metaphorical principal (MPrin) of a particular can change as a result of retiring other seemingly unrelated classes of technical debt. In most cases, engineering expertise is required to determine technical debt retirement strategies that can exploit this property of some kinds of technical debt.

Financial debts usually have associated interest rates that are used to compute the periodic interest charges. Typically, the interest charge on a financial debt for a given period is the periodic interest rate multiplied by the principal, and then scaled for the length of the time period.

But there are no “rates” for technical debt. Their existence would imply that MICs were proportional to the analog of “principal,” which, in the case of technical debt, is the cost of retiring the debt — the MPrin. MICs depend only weakly on the cost of retiring the debt. Instead, they depend more strongly on the impact of the debt on ongoing operations.

Decision-makers who understand the world of financial instruments at a very sophisticated level might tend to overvalue arguments favoring technical debt management in ways analogous to the ways we manage financial debts. Financial sophisticates might find appealing any argument for a technical debt management program that parallels financial approaches. Such programs are unlikely to work, for two reasons. First, as we’ve already noted, the uncertainties associated with estimating MPrin and MICs make technical debt management decisions more dependent on engineering and project management judgment than they are on the results of calculations and projections (see MPrin uncertainties and MICs uncertainties).

Second, as noted above, the familiar concept of interest rate is inapplicable to technical debt, because the MICs depend on the degree of interaction between ongoing activities and the debt itself, rather than the cost of retiring the debt. And that means that MICs (and MPrin) of one class of debt can change when another class is retired.

Implications of this effect

The possibility that retiring one class of technical debt can alter the financial burdens presented by another class of technical debt has both favorable and unfavorable implications.

MICs can change when other debts are retired
An example illustrating one way in which MICs on one kind of technical debt  can change as a result of retiring a different kind of technical debt. The structure at the left represents the situation before any debt retirement occurs. The balloons labeled “A” represent instances of asset A. The balloon labeled “B” represents asset B. The orange circles represent instances of technical debt D1 and D2, respectively. The arrows connecting the As to B indicate that asset A depends on Asset B. The structure at the right represents the situation after debt retirement.

As an example of a favorable implication, consider software remodularization. Suppose we have a software asset A that depends on another software asset B. As shown in the left image of the figure, asset A, of which there are many copies, bears two classes of technical debt, D1 and D2. As shown, there is only one copy of asset B. Suppose further that an asset that bears debt D2 also bears debt D1, but an asset that bears D1 might or might not bear debt D2.

To retire D2, engineers have decided to modify B by having it assume responsibility for the tasks that formerly bore debt type D2. They do this even though, as a consequence of this change, B will now bear debt of type D1. Next, debt type D2 is retired. The right half of the figure shows the resulting implementation. The system still bears debt D1, but now it’s located in B instead of A. All instances of type A assets change, and those modifications relieve them of both types of debt. This is a sensible approach, because there are several assets of type A and only one of type B. The end result is that D2 vanishes, and only a single instance of D1 remains. In this way, retiring debt D2 has reduced the MICs and MPrin for D1.

Policymakers can help

Exploiting the salutary opportunities of this property of technical debt provides an example of the risks of adhering too closely to the financial model of debt.

Many different scenarios have the property that retiring one kind of technical debt can reduce the MICs associated with other kinds of debt. Because technologists understandably tend to be more concerned with technical debt retirement strategies that emphasize short-term improvement of their own productivity, policymakers can provide guidance that steers the organization in the direction of enterprise benefits.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Related posts

  • The Principal Principle: Focus on MICs
    The Principal Principle is that a focus on the metaphorical principal of a technical debt can be your undoing. Focus on the metaphorical interest charges. Drive them to Zero and keep them there.
  • Where the misunderstandings about MICs come from
    Misunderstandings about the metaphorical interest charges on technical debt are costly. They prevent us from exploiting the properties of technical debt that reduce carrying costs and retirement costs. And the misunderstandings arise from the fact that the technical debt metaphor is only a metaphor—technical debt and financial debt are different.
  • MICs on technical debt can be difficult to measure
    Unlike financial debt, for technical debt there are no legally required reports or disclosures. We can sometimes estimate MICs, but most organizations don’t track the data necessary to estimate MICs with useful precision. Indeed, developing useful estimates is often technically impossible.
  • MICs can sometimes be deferred or advanced without penalty
    Rescheduling interest payments on financial debt is possible only by prearrangement or in bankruptcy, but MICs on technical debt can often be rescheduled by rescheduling work that might incur them. This is useful when we plan to retire assets bearing technical debt, because their technical debt vanishes.
  • MICs can differ for different instances of the same kind of technical debt
    For financial debts, the interest charges associated with a unit of debt are (usually) the same for every unit of debt incurred under the same loan agreement. But for technical debt, the MICs associated with a given instance of a class of technical debt might differ substantially from any other instance of the same class of technical debt, even if those instances of technical debt were incurred at the same time as a result of a single decision or sequence of events.
  • How financial interest charges differ from interest charges on technical debt
    The common understanding of interest on financial debts doesn’t correspond to the reality of technology-based systems, which are the targets of the technical debt metaphor. Formulating sound technical debt policy depends on understanding the nature of the difference between interest on financial debt and the metaphorical interest charges associated with technical debt.
Posted on

MICs on technical debt can be difficult to measure

Last updated on July 3rd, 2018 at 08:02 am

For a financial debt, creditors regularly inform debtors of periodic interest charges, principal remaining, and other parameters of the loan. In many cases, laws require regular reports and explicit statements about interest charges when prospective creditors interact with prospective debtors. By contrast, for technical debt, metaphorical interest charges (MICs) can be difficult to compute with useful precision, even if we know that they’re accumulating. Many decision-makers are actually unaware that MICs are accumulating at all. For an organization to appreciate the full financial consequences of carrying technical debt, everyone in the enterprise must appreciate the concept of MICs.

A stack of floppy disks
A stack of floppy disks. You don’t see many of these around much anymore. Very little of the software or hardware we use is as obsolete as these floppies. But much of it is obsolete, and it therefore comprises technical debt. It still works, but it’s slow and probably no longer supported by its manufacturer. On the basis of speed alone, the MICs it incurs can easily justify replacement. And some of it is vulnerable to cyberattack. One significant breach can ruin a brand.

Unlike financial debt, for technical debt there are no legally required reports or disclosures. We can sometimes estimate MICs, but most organizations don’t track the data necessary to estimate MICs with useful precision. Indeed, developing useful estimates is often technically impossible.

The difficulty of measuring MICs arises from three sources. First, people whose productivity is most directly affected by technical debt — usually engineers — often have difficulty determining with precision the extent of the impact of technical debt on their efforts.

Second, many people are unaware of the impact technical debt has on their results. For example, if a product arrives late to market, the financial costs attributable to technical debt can be computed if we realize that technical debt is partially — or wholly — responsible for the delay. Too often, those who could perform such calculations aren’t sufficiently familiar with the concept of MICs, and in any case, the data they would need for calculating a useful estimate is rarely available.

Finally, a more insidious form of the consequences of technical debt is what we might call the terrifying opportunity. This situation arises when the organization rejects (or fails to recognize) a market opportunity because exploiting it would involve modifying an existing asset or product offering that harbors a heavy load of technical debt. The debt causes decision-makers to assess that the probability of success is so depressed by technical debt that the opportunity seems terrifying, and they therefore reject the opportunity. Typically, terrifying opportunities would be exploitable if the debt-bearing assets didn’t exist at all, because then we would be starting fresh. But given that terrifying opportunities require modifying existing assets that bear heavy loads of technical debt, commitment requires faith that the technical debt can be addressed successfully.

The sense of risk isn’t a reflection on the capabilities of the technical organization. Rather, it’s a result of the challenges involved in working with assets that bear high levels of technical debt. Given past performance of the technical organization relative to these debt-bearing assets, success can seem unlikely.

Computing the cost of a terrifying opportunity requires estimating the cost of not exploiting the opportunity, a difficult task in the best of circumstances. But whatever that cost is, it’s a form of MICs that we rarely recognize.

Building expertise in estimating MICs in all their forms is advantageous to any organization that seeks to make its technical debt more manageable. By making MICs visible, we can bring about better recognition of the cost of carrying technical debt, thereby providing an appropriate motivator for retiring technical debt.

References

[Brown 2010] Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, Raghvinder Sangwan, Carolyn Seaman, Kevin Sullivan, and Nico Zazworka. “Managing Technical Debt in Software-Reliant Systems,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research 2010, New York: ACM, 2010, 47-51.

Available: here; Retrieved: July 30, 2018

Cited in:

[Levy 2009] David A. Levy, Tools of Critical Thinking: Metathoughts for Psychology (second edition). Long Grove, Illinois: Waveland Press, Inc., 2009.

Order from Amazon

Cited in:

[Whitehead 1948] Alfred North Whitehead. Science and the Modern World. New York: Pelican Mentor (MacMillan), 1948 [1925].

Order from Amazon

Cited in:

Related posts

  • The Principal Principle: Focus on MICs
    The Principal Principle is that a focus on the metaphorical principal of a technical debt can be your undoing. Focus on the metaphorical interest charges. Drive them to Zero and keep them there.
  • Where the misunderstandings about MICs come from
    Misunderstandings about the metaphorical interest charges on technical debt are costly. They prevent us from exploiting the properties of technical debt that reduce carrying costs and retirement costs. And the misunderstandings arise from the fact that the technical debt metaphor is only a metaphor—technical debt and financial debt are different.
  • MICs can change when other debts are retired
    The metaphorical interest charges (MICs) and metaphorical principal (MPrin) of a particular class of technical debt can change as a result of retiring other seemingly unrelated classes of technical debt. In most cases, engineering expertise is required to determine technical debt retirement strategies that can exploit this property of technical debt.
  • MICs can sometimes be deferred or advanced without penalty
    Rescheduling interest payments on financial debt is possible only by prearrangement or in bankruptcy, but MICs on technical debt can often be rescheduled by rescheduling work that might incur them. This is useful when we plan to retire assets bearing technical debt, because their technical debt vanishes.
  • MICs can differ for different instances of the same kind of technical debt
    For financial debts, the interest charges associated with a unit of debt are (usually) the same for every unit of debt incurred under the same loan agreement. But for technical debt, the MICs associated with a given instance of a class of technical debt might differ substantially from any other instance of the same class of technical debt, even if those instances of technical debt were incurred at the same time as a result of a single decision or sequence of events.
  • How financial interest charges differ from interest charges on technical debt
    The common understanding of interest on financial debts doesn’t correspond to the reality of technology-based systems, which are the targets of the technical debt metaphor. Formulating sound technical debt policy depends on understanding the nature of the difference between interest on financial debt and the metaphorical interest charges associated with technical debt.
Show Buttons
Hide Buttons