Retiring technical debt can be a wicked problem

Last updated on December 11th, 2018 at 07:08 am

The theory of wicked problems originated with Horst Rittel in the mid-1960s. He was trying to address “that class of problems which are ill-formulated, where the information is confusing, where there are many decision makers and clients with conflicting values, and where the ramifications in the whole system are confusing.” [Churchman 1967] The term wicked isn’t intended as a moral judgment. It suggests the mischievous streak in these problems, many of which have the property that proposed solutions often lead to conditions even more problematic than the situation they were intended to resolve. Is it just me, or are you also thinking, “Ah, technical debt”? In this post, I suggest that retiring technical debt can be a wicked problem. I’ll show how wickedness explains many of the difficulties we associate with retiring forms of technical debt that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.


Prototypes of President Trump’s “border wall.”
Prototypes of President Trump’s “border wall.” Building such a wall is undoubtedly an example of a wicked problem. Building prototypes in short segments of the wall to see what they would look like is a tame problem. But these are just prototypes of short segments of the wall. They aren’t prototypes of the project. They don’t demonstrate the process by which private property will be taken for the wall, or how construction access roads will be built, or how wildlife will be affected (and respond), or how the government of the adjacent nation (in this case, Mexico) will respond, or how the wall will be repaired when drug gangs destroy sections of it in isolated regions, or even if the wall would actually work as promised. Prototyping works well for tame problems. It helps us project how the finished project will perform, and how difficult it will be to complete the project. But for wicked problems, prototyping is of limited value and as Rittel observes, prototyping can make the problem worse.
Photo by Mani Albrecht, U.S. Customs and Border Protection Office of Public Affairs, Visual Communications Division.

Horst Rittel was a design theorist at the University of California at Berkeley. His interest in wicked problems came about because of the need for designers to deal with the interactions between architecture and politics. In today’s technology-dependent enterprises, analogous problems arise when we try to retire technical debt, because multiple sets of quasi-independent stakeholders are affected by major technical debt retirement projects.

In the years since he first originated the wicked problem concept, people have extended it in ways that have led some to regard the concept as inflated and less than useful. But since extension and imitation rarely occur unless the extended concept has value, I take this phenomenon to mean that Rittel’s version of the concept is worthwhile. The focus of this post, then, is Rittel’s version of wicked problems, as applied to the problem of designing a complex technical debt retirement project.

The wicked problem concept has propagated mostly in the realm of public policy and social planning. Certainly wicked problems abound there: poverty, crime control, and climate change are examples. I know of no attempt to explore the wickedness of retiring technical debt in large enterprises, but have a look below and see what you think.

Rittel defines a problem as the discrepancy between the current state of affairs and the “state as it ought to be.” For the purposes of technical debt retirement planning, the state as it ought to be might at times be a bit ambitious. So I take the objective of a technical debt retirement project to be an attempt to resolve the discrepancy between the current state of affairs and some other state that’s more desirable. For the present purpose, then, the problem is designing a technical debt retirement project that converts the current state of affairs to a more desirable state that might still contain technical debt in some form, but it leaves us in a better position.

Actually, there is a subset of wicked problems—super-wicked problems—that I think might include some technical debt retirement problems. I address them in the post “Retiring technical debt can be a super wicked problem.”

For now, though, let’s examine the properties of wicked problems, and see how well they match up with the problem of designing technical debt retirement projects.

Attributes of wicked problems

Rittel’s summary of the attributes of wicked problems [Rittel 1973] convinced me that major technical debt retirement projects present wicked problems. Here are those attributes. In what follows, I use Rittel’s term tame problem to refer to a problem that isn’t wicked.

1. [No Definitive Formulation] Wicked problems have no definitive formulation

For any given tame problem, it’s possible to state it in such a way that it provides the problem-solver all information necessary to solve it. That’s what is meant by definitive formulation. For wicked problems, on the other hand, one’s understanding of the problem depends on the solution one is considering. Each candidate solution might—potentially—require its own understanding of the problem.

When designing a technical debt retirement project, we must fully grasp the impact of the effort on all activities in the enterprise. Each proposed project plan has its own schedule and risk profile, affecting enterprise activities in its own way. In principle, each candidate approach to the effort affects a different portfolio of enterprise assets in its own unique order. Because it isn’t practical to examine all possible candidate project plans, choosing a project plan by seeking an optimal set of effects isn’t practical. By the time you’re ready to execute the project plan you like best, the elapsed time required for such an analysis could render obsolete the data you collected first.

2. [No Stopping Rule] Wicked problems have no stopping rule

For any given tame problem, solutions have “stopping rules”—signatures that indicate clearly that they are indeed solutions. For example, in a chess problem to be solved in N moves, we know how to count to N and the position of checkmate is well defined.

Wicked problems have no stopping rule.

When planning a major technical debt retirement project, we must determine a task breakdown, a sequence for performing the tasks (which might be only partially ordered), a resource array including both human and non-human resources, a risk plan including risk mitigations and risk responses, a revenue stream interruption schedule, and so on. For each such plan, we can estimate the direct and indirect costs to the enterprise, and project the effects of the plan on market share for every affected product or service. Every plan has these attributes; when we compute them for a given candidate plan, the result doesn’t reveal that we’ve found “the solution.” We will have found only an estimate for that given solution. What we learn by doing this doesn’t reveal whether or not a “better” solution exists. There is no indicator contained in any given candidate solution that tells us we can “stop” solving the problem. Most often, we just stop when we run out of time for finding solutions. In some cases, we stop when we find just one solution.

3. [Solutions Are Good/Bad] Solutions to wicked problems are not true-or-false, but good-or-bad

The criteria for finding solutions to tame problems are unambiguous. For example, if a candidate function satisfies a differential equation, it’s a solution to the equation. The volume of concrete required to pave a section of roadway is a single number, determined by computing the area of roadway and multiplying by the thickness of the roadbed, and subtracting the volume of any reinforcing steel.

The solutions to wicked problems have no such clarity. When evaluating a candidate project plan for retiring a technical debt, we can estimate its cost, the time required, interruptions in revenue streams, and the timing of resource requirements. But determining how “good” that is might be difficult. Much depends on what other demands there might be for those resources or funds, and the political power of the people making those demands. No single number measures that.

4. [No Test of Solutions] There is no immediate and no ultimate test of a solution to a wicked problem

To test a candidate solution to a tame problem, the problem solving team determines whether the solution meets the requirements set in the tame problem statement. The consequences of implementing the solution are all evident to the problem solving team, and they’re well equipped to judge the success of the solution.

Not so with wicked problems. Any candidate solution to a wicked problem will generate waves of consequences. As these waves propagate, some of the problem’s stakeholders might find the solution unsatisfactory, and they’ll report this, possibly through politically powerful people or organizations, to the problem solving team. Because the consequences can be so diverse, the team can’t anticipate all of them, and in some cases, the team might have difficulty understanding how the troubles that plague some stakeholders were actually related to the implemented solution. Some undesirable consequences can be far more severe than any intended benefits. In other cases, the undesirable consequences might not be discovered until long after the solution has been deployed and accepted as operational.

When designing a technical debt retirement project, it’s necessary to determine everything that must be changed, what resources must be assembled to do the work, and what processes might be interrupted, when and for how long. Only rarely, if ever, can all of that be determined with certainty in advance. For that reason, determining that the design of the project is “correct” isn’t possible, except perhaps in the probabilistic sense. We never really know in advance that we’ve found a solution. Most of the time, after execution begins, we must make adjustments along the way, in real time.

5. [No Trial-and-error] Every solution to a wicked problem is a “one-shot operation”; because there is no opportunity to learn by trial-and-error, every attempt counts significantly

When solving tame problems, we can try candidate solutions without incurring significant penalties for the solution-finding effort. That is, trying a solution might require some effort (and therefore incur a cost), but it doesn’t otherwise affect the ability to find other solutions. Wicked problems are different. Every attempt to “try” a solution leaves traces that can potentially make further solution attempts more difficult, costly, or risky than they would have been if we hadn’t tried that solution. These traces of past solution attempts might also impose constraints on future solutions such that the wicked problem is effectively transformed into a new wicked problem. This property makes trial-and-error approaches undesirable and possibly infeasible. Indices of such undesirability are the half-lives of the traces of attempts to address the problem. A long half-life may mean that the problem solver has only one shot at addressing the problem.

When designing a technical debt retirement project, we sometimes try to “pilot” a potential approach to determine difficulty, costs, feasibility, political issues, or risk profiles. Even when we can revert the asset to its former state after a pilot is completed or suspended, the consequences for stakeholders and for stakeholder operations might not be reversible. When we next try another “pilot,” or perhaps a fully committed retirement project, these stakeholders might be significantly less willing to cooperate. Every attempted solution can leave political or financial traces like these, making future attempts riskier and more challenging.

6. [Solutions Are Not Describable] Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan

In devising solutions to tame problems, one common approach entails gathering the full set of possibilities, and screening them according to a set of favorability criteria. Reducing the field of possibilities is a useful strategy for finding optimal solutions—or even acceptable solutions—to tame problems.

Wicked problems defy such strategies. Gathering the full set of possible solutions to a given wicked problem can in itself be a wicked problem. The set of possible solutions cannot be parameterized; no finite set of attributes fully covers the solution space. We can never be certain that the set of candidates we have is a complete set.

Candidate designs for technical debt retirement projects present this same quality. We’re confronted with a dizzying array of choices, including the order in which we retire different kinds of technical debts, the order in which we address the debts borne by different assets, the possibilities of “refinancing” portions of the debt to intermediate forms of technical debt, what kinds of refactoring to perform and when, and much more. Because options like these are neither denumerable nor parameterizable, it’s difficult to know whether a given set of candidate project designs includes all possible project designs.

7. [Essential Uniqueness] Every wicked problem is essentially unique

Among tame problems, we can define classes or categories that have the property that all problems in a given class can be solved using the same method. For example, all the elements of the class of second order linear differential equations can all be solved using similar methods.

There are no classes of wicked problems that have such a property. There are no classes of wicked problems for which we can define a solution strategy that is effective for every member of the class. Even though we can define classes of wicked problems whose members are in some sense similar, that similarity doesn’t enable us to find a unified solution strategy that works for every member of the class.

So it is with designing technical debt retirement projects. Certainly, the collection of all technical debt retirement projects is a class; but the problem of designing a given retirement project is essentially unique. What “works” for one project in one enterprise in one fiscal year probably won’t work for another project in another enterprise in another fiscal year, or even another project in that same enterprise in that same fiscal year. Elements of the solution for one project might be useful for another project, but even then, we might need to adapt them to the conditions of that next project.

This essential uniqueness property of technical debt retirement projects collides with a common pattern decision-makers use when chartering major efforts. That pattern is reliance on consultants, employees, or contractors who “have demonstrated success and experience with this kind of work.” Because each technical debt retirement project is essentially unique, relying on a history of demonstrated success is a much less viable strategy than it would be with tame problems. Decision-makers would do well to keep this in mind when they seek approaches, leaders, and staff for major technical debt retirement efforts: no major technical debt retirement project is like any other.

8. [Problems as Symptoms] Every wicked problem can be considered to be a symptom of another problem

With tame problems or wicked, we typically begin the search for solutions by inquiring as to the cause of the current condition. When we find the cause or causes, and remove them, we usually find a new problem underlying them. Thus, for wicked problems, what we regarded initially as the problem is thereby converted into a symptom of newly recognized underlying problem. By repeating this process, we escalate the “level” of the problem we’re addressing. Higher-level problems do tend to be more difficult to resolve, but addressing symptoms, though easier, isn’t a path to ultimate resolution.

Rittel also observes that incremental approaches to resolving wicked problems can be self-defeating. The difficulty arises from the traces left behind by incrementalism, as described in the discussion of the unworkability of trial-and-error strategies. Rittel provides the example of the increase in difficulty of changing processes after we automate them.

To regard the wicked problem of designing a technical debt retirement project as a symptom of a higher-level wicked problem, we must be willing to regard as problems the very things that make the technical debt retirement project design effort a wicked problem. That is, the processes that lead to formation of technical debt, or that enhance its persistence, are themselves wicked problems. For example, one might inquire about how to change the enterprise culture so as to reduce the incidence of technical debt contagion. To undertake major technical debt retirement efforts without first determining what can be done to limit technical debt formation or persistence, due to contagion or due to other processes, might be unwise.

9. [No Controlled Experiments] The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution

When addressing tame problems, problem-solving teams can often perform controlled experiments. The general framework of these experiments is as follows. They form a hypothesis H as to the cause of the problem, conjecturing a solution. Then assuming that H is correct, and given a set of conditions C, they deduce a set of consequences E that must follow if H is correct. If any elements of E don’t occur, then H is falsified. The process repeats until an H’ is found that cannot be falsified. H’ is then used to develop a solution. This is, essentially, the scientific method.

With wicked problems, the method fails in numerous ways. Foremost among these failure modes is the absence of the ability to control C. That is, interventions that might be required to set C to be a desired C0 tend to be impossible. Moreover, even if one can establish C0, the experiments one performs to determine whether E is observed tend to leave the kinds of traces discussed in Proposition 5 [No Trial-and-error]. Finally, the nature of the elements of E is usually such that determining their presence or absence is largely subjective.

When planning enterprise-scale technical debt retirement projects, as with many projects of similar scale, we believe that we can benefit from running a pilot of our proposed plan, to determine its fitness. However, because we cannot control the conditions in which we execute the pilot, we cannot be confident that our interpretation of the results of the pilot will apply to the actual project. Moreover, a small-scale pilot cannot generate some of the effects we most want to observe because they occur only at full scale. These effects include staff shortages, resource contention, and revenue interruption incidents.

10. [100% Accountability] The planner (or designer) has no right to be wrong

In solving tame problems, solvers are free to experiment with proposed solutions. They make conjectures about what might work, and gather the results of trials to determine how to improve their conjectured solutions. There is no social or legal penalty for failed conjectures.

In solving wicked problems, experiments don’t exist. Any trial solution is a real solution, with real effects on stakeholders and later, on the problem solvers themselves. Problem solvers are held accountable for the undesirable consequences of each solution, whether it’s a trial or not.

In planning a technical debt retirement project, any attempt to gather data about how the approach would affect the enterprise could potentially have real, lasting, deleterious effects. The costs associated with these consequences are charged to the project, if not officially and financially, then politically. The politics of failure can lead to serious consequences for the problem solvers. Any approach that the team deploys, on any scale no matter how small, can potentially create financial problems for the enterprise, and political problems for anyone associated with the technical debt retirement project.

Last words

The fit between wicked problems and technical debt retirement project design looks pretty good to me. But the research on a subset of wicked problems—super wicked problems—is also intriguing. I’ll look at that in my next post. After that, we’ll be ready to examine which approaches to retiring technical debt take these matters into account.


[Churchman 1967] C. West Churchman. “Wicked problems,” Management Science 14:4, 1967, B-141–B-142

Available: here; Retrieved: October 16, 2018

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

Other posts in this thread

The Tragedy of the Commons is a distraction

Last updated on July 24th, 2018 at 08:19 pm

Many believe that technical debt arises, in part, because of a phenomenon known as the Tragedy of the Commons [Hardin 1968], which is an allegory that purports to demonstrate that the user communities associated with shared resources inevitably degrade those resources until they’re depleted. The allegory supposedly supports the thesis that only monocratic control of an asset can provide the strict regulation that prevents its inevitable degradation as a result of shared use. Advocates of this approach to limiting the degradation arising from the expansion of technical debt hold that assigning sole ownership of resources, resource by resource, is the only effective method of controlling technical debt.

A map of the Boston Common and Public Garden, circa 1890. This is the kind of “common” referred to in the tragedy of the commons.
A map of the Boston Common and Public Garden, circa 1890. By that time it was basically a park. But as late as 1830 it was still being used as a cow pasture. They didn’t have refrigeration at the home scale then, except by ice blocks, and the best way to get fresh dairy products was to have a cow. In the very early days, 1633-1640, anyone could graze on the Common, but as wealthy people acquired more animals, the common became overgrazed, and a 70-cow limit was imposed. That limit stood until 1830. It’s an example of a method for managing a shared resource. This map is from an atlas of Boston published by G.W. Bromley & Co.,, courtesy Wikimedia Commons
The resources in question here are the assets that tend to accumulate, or are accumulating, or have accumulated, technical debt. Adherents of the theory would impose order by dividing each technological asset into one or more sectors, sometimes called development silos, with only one organizational unit designated as the “owner,” empowered to develop, maintain, or extend that sector [Bossavit 2013] [Morris 2012]. Irreconcilable disagreements about the direction or purpose of a particular sector of the asset presumably would be resolved by branching.

Ironically, such an approach would — and demonstrably does — produce significant technical debt in the form of duplication of artifacts and services. Moreover, it elevates costs relative to a truly shared asset, by reducing sharing, and increasing the need for testing. We can regard such an approach as dysfunctional conflict avoidance [Brenner 2016b].

Although at one time the Tragedy of the Commons was regarded as a universally valid concept in political economics, subsequent research has demonstrated that the principle it describes is not generally applicable. Hardin first described the Tragedy of the Commons in 1968, in the form of an allegory [Hardin 1968]. In his words:

Picture a pasture open to all. It is to be expected that each herdsman will try to keep as many cattle as possible on the commons. Such an arrangement may work reasonably satisfactorily for centuries because tribal wars, poaching, and disease keep the numbers of both man and beast well below the carrying capacity of the land. Finally, however, comes the day of reckoning, that is, the day when the long-desired goal of social stability becomes a reality. At this point, the inherent logic of the commons remorselessly generates tragedy.

As a rational being, each herdsman seeks to maximize his gain. Explicitly or implicitly, more or less consciously, he asks, “What is the utility to me of adding one more animal to my herd?”

Hardin then explains that each herdsman is compelled by the logic of the situation to exploit the shared resource to the maximum. Each herdsman puts his own interests ahead of the welfare of the resource.

And so it goes, supposedly, with technical debt. Each user of the shared asset expends resources on development, maintenance, and enhancement only to the extent that the expenditure is justified by immediate need. Retiring any legacy technical debt, or any technical debt accumulated in the course of meeting those immediate needs, is regarded as low priority. Because resources for debt retirement are rarely if ever sufficient to meet the need, technical debt grows inexorably. Eventually, the shared asset becomes unmaintainable and must be abandoned.

However, careful research shows that Hardin’s Commons allegory is not applicable to every situation involving shared resources. That same research casts doubt on the validity of the assertion that development silos are necessary in any approach to technical debt management.

Certainly there are many examples of shared resources degrading along the lines outlined by Hardin, such as the collapse of the Northwest Atlantic cod fishery [Frank 2005], but many counterexamples exist. Research by the late political economist Elinor Ostrom uncovered numerous examples of complex social schemes for maintaining common resources efficiently and sustainably [Ostrom 2009] [Ostrom 1990]. Ostrom studied and reported on systems that successfully managed shared resources over long terms — in some cases, centuries. For this work, she received the Nobel Prize in Economics in 2009.

As Ostrom’s research demonstrated, the problem with Hardin’s allegory is that it applies only to shared resources that are open to use by all without regulation. The misapplication of the Tragedy of the Commons is clearly described in a World Bank Discussion Paper by Bromley and Cernea [Bromley 1989]:

For some time now, Hardin’s allegory of the “tragedy” has had remarkable currency among researchers and development practitioners. Not only has it become the dominant paradigm within which social scientists assess natural resource issues, but it appears explicitly and implicitly in the formulation of many programs and projects and in other beliefs and prejudices derived from it. Unfortunately, its power as a metaphor is not matched by its capacity for aiding our understanding of resource management regimes. By confusing an open access regime (a free-for-all) with a common property regime (in which group size and behavioral rules are specified) the metaphor denies the very possibility for resource users to act together and institute checks and balances, rules and sanctions, for their own interaction within a given environment.

Hardin himself later published an extension of the allegory that clarified the role of regulation [Hardin 1998], as had been observed much earlier by Lloyd [Lloyd 1833].

The real tragedy for technology managers would be their failure to learn from the past errors of social scientists and political economists, and to then repeat, in the context of technical debt management, this now well-understood confusion about the domain of applicability of Hardin’s allegory.

We can apply Ostrom’s result to the problem of managing technical debt if we identify the technical asset as the shared resource, and identify as the community exploiting the resource the stakeholders who employ, develop, maintain, cyber-defend, or extend that technical asset. Ostrom’s results tell us that sustainable exploitation is possible only if the community devises rules, customs, and sanctions that manage the technical debt. Kim and Wood [Kim 2011] provide an analysis that explains how regulation can avert depletion scenarios. Technology managers can apply these lessons to the problem of managing technical debt.

The Tragedy of the Commons is a distraction because technical debt isn’t an inevitable result of sharing assets when the organization adheres to a Principle of Sustainability. That principle is that sustainability is possible only if the community sharing the asset devises customs, rules, and sanctions that effectively control the level of technical debt. You just can’t have a free-for-all unregulated regime, as most organizations now do. Management and practitioners must collaborate to devise the customs, rules, and sanctions for managing the asset. And regular updating is probably necessary. Leadership in devising those customs, rules, and sanctions is a job for the policymaker.


[Bossavit 2013] Laurent Bossavit (@Morendil), “Zero Code Ownership will lead to a tragedy-of-the-commons situation, where everybody bemoans how ‘technical debt’ makes their job suck.”, a tweet published April 20, 2013.

Available: here; Retrieved December 29, 2016.

Cited in:

[Brenner 2016b] Richard Brenner. “Some Causes of Scope Creep,” Point Lookout 2:36, September 4, 2002.

Available here; Retrieved December 30, 2016.

Cited in:

[Bromley 1989] Daniel W. Bromley and Michael M. Cernea. “The Management of Common Property Natural Resources: Some Conceptual and Operational Fallacies.” World Bank Discussion Paper WDP-57. 1989.

Available here; Retrieved December 29, 2016.

Cited in:

[Churchman 1967] C. West Churchman. “Wicked problems,” Management Science 14:4, 1967, B-141–B-142

Available: here; Retrieved: October 16, 2018

Cited in:

[Frank 2005] Frank, Kenneth T., Brian Petrie, Jae S. Choi, William C. Leggett. "Trophic Cascades in a Formerly Cod-Dominated Ecosystem." Science. 308 (5728): 1621–1623. June 10, 2005.

Available here; Retrieved: March 10, 2017.

Cited in:

[Hardin 1968] Garrett Hardin. “The Tragedy of the Commons,” Science, 162, 1243-1248 1968.

Available: here; Retrieved December 29, 2016.

Cited in:

[Hardin 1998] Garrett Hardin. “Extensions of ‘The Tragedy of the Commons’,” Science, May 1, 1998: Vol. 280, Issue 5364, 682-683.

Available: here; Retrieved: July 30, 2017

Cited in:

[Kim 2011] Daniel H. Kim and Virginia Anderson. Systems Archetype Basics: From Story to Structure, Waltham, Massachusetts: Pegasus Communications, Inc., 2011

Available: here; Retrieved: July 4, 2017 Order from Amazon

Cited in:

[Lloyd 1833] Lloyd, W. F. Two Lectures on the Checks to Population, 1833.

Available: here; Retrieved: July 30, 2017

Cited in:

[Morris 2012] Ben Morris. “How to manage down the payments on your technical debt,” Ben Morris Software Architecture blog, September 3, 2012.

Available here; Retrieved December 30, 2016. This blog entry contains an assertion that controlling formation of new technical debt requires only “diligence, ownership and governance.”

Cited in:

[Ostrom 1990] Ostrom, Elinor. Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge: Cambridge University Press, 1990.

Cited in:

[Ostrom 2009] Ostrom, Elinor. “Beyond the tragedy of commons,” Stockholm whiteboard seminars.

Video, 8:26 min. Apr 3, 2009. here; Retrieved December 29, 2016.

Cited in:

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

Cited in:

Other posts in this thread