Month: November 2018

Last updated on December 1st, 2018 at 09:04 am

Several properties of the problem of designing technical debt retirement projects tend to make those design problems more likely to be wicked problems—that is, more likely to satisfy all ten of the criteria of Rittel and Webber [Rittel 1973]. I call these properties indicators of wickedness.

Although we usually have some notion of the degree of wickedness of a given design effort for a technical debt retirement project, actually executing the debt retirement project can reveal unanticipated issues and complexity. Some of what’s revealed can cause us to adjust our estimate of the degree of wickedness of the design effort. If we know in advance what kinds of revelations are most likely to cause such adjustments, we can reduce the incidence of unanticipated revelations.

As I noted in my post, “Degrees of wickedness,” we can regard all problems as lying on a Tame/Wicked spectrum, with wicked problems lying at the extreme Wicked end of the spectrum, and the tamest of the tame lying at the opposite end. As for the ten criteria of wickedness developed by Rittel and Webber, I proposed that they could be satisfied in degrees, with the most wicked problems satisfying all ten criteria absolutely.

As a quick review, here are the attributes of wicked problems as Rittel and Webber see them [Rittel 1973], rephrased for brevity:

1. There is no clear problem statement
2. There’s no way to tell when you’ve “solved” it
3. Solutions aren’t right/wrong, but good/bad
4. There’s no ultimate test of a solution
5. You can’t learn by trial-and-error
6. There’s no way to describe the set of possible solutions
7. Every problem is unique
8. Every problem can be seen as a symptom of another problem
9. How you explain the problem determines what solutions you investigate
10. The planner (or designer) is accountable for the consequences of trying a solution

Below is a sample of conditions or situations that tend to increase the wickedness of the problem of designing a technical debt retirement project. I have no data to support these conjectured effects. But the principles I used to generate them are three:

• If a phenomenon expands the set of stakeholders in a debt retirement project, it tends to enhance the wickedness of the design problem.
• If a phenomenon increases the number or heterogeneity of the assets or processes that must be considered, it tends to enhance the wickedness of the design problem.
• If a phenomenon creates a need for a rollback of work performed as part of the debt retirement project, and that rollback creates a need to re-design the debt retirement project, it tends thereby to enhance the wickedness of the design problem.

In what follows, I use the term “DRP” to indicate the Debt Retirement Project itself, as distinguished from the effort to design the DRP, which I refer to as “DDRP.” The problem whose wickedness we’re considering is not the DRP itself, but the DDRP. Also, let DBA (for debt-bearing assets) be the set of assets undergoing modification in the context of the DRP, and let IA (for interacting assets)  be the set of assets, excluding the DBA assets, that interact directly or indirectly with assets in the DBA.

With all this in mind, I offer the following nine examples of indicators of wickedness of the DDRP.

1.   A previous attempt to retire this debt was abandoned

Perhaps the most significant indicator of the wickedness of the DDRP is either the failure of a previous attempt to execute a DRP with similar objectives, or the failure of a previous attempt to execute a DDRP for a DRP with those objectives. There are two reasons why such failures are significant indicators of wickedness.

First, it’s reasonable to assume that these previous attempts weren’t founded on any recognition of the wickedness of the DRP or the DDRP. Few such efforts are.  (A Google search for the two phrases “technical debt” and “wicked problem” yields less than 1000 results) (update 12 Nov 2018: 1160 results) Consider first the DDRP. If it is a wicked problem, proceeding as if it were not would very likely fail. If the designers of the previous DDRP did assume that it was a wicked problem, investigating their approach could prove invaluable, and save much time and effort. An analogous argument applies for the DRP itself.

Second, if the previous attempt to execute a DRP with similar objectives has left traces of itself in the DBA, and if those traces must be taken into account while executing the DDRP, they might complicate the DRP, and they might be incompletely addressed in the DDRP. To the extent that these conditions prevail, Criterion 5 is satisfied, and the DDRP exhibits wickedness.

2.   Some revenue streams need to be interrupted

If the work of the DRP entails temporary interruption of revenue streams, executing the DRP can have significant and long lasting effects on the organization. In estimating the cost of the DRP, it’s clearly necessary to account for the financial impact of any revenue shifted into the future, and any revenue irretrievably lost as well. And in some cases, market share might also suffer. All of these factors tend to increase the wickedness of the DDRP.

When these effects are expected, political opposition to the DRP can develop. Senior management can prevent this opposition from halting the DDRP inappropriately by requiring that the business case for the DRP include these financial factors and demonstrate clearly the need to proceed despite them. Involving potential political opponents of the DRP in business case development can be an effective means of ensuring the strength of the business case.

The ability to model all these financial effects is an important organizational asset that can be developed and maintained, for deployment across multiple DDRPs. The organization can monitor DRPs, gathering actual experience data for comparison to the effects projected in the respective business cases of the DRPs. Those comparisons are useful for enhancing the modeling capability.

3.   Some assets not directly touched need to be re-tested

The DDRP is more likely to be a wicked problem if, as a result of the changes executed in the DRP, any of the assets in IA need to be re-tested after or during DRP execution. The need to re-test any assets in IA typically arises when there’s some risk that the DRP’s changes in the DBA could somehow affect the performance of the assets in IA, and when the consequences of such a risk event are severe.

This scenario enhances of the wickedness of the DDRP for at least five possible reasons.

• Baseline testing of IA is necessary to enable the DRP team to recognize the effects of the DRP on IA behavior. But this baseline testing can reveal pre-existing and unaddressed faults. Because leaving those faults in place can seriously complicate interpretation of anomalies that appear in IA assets after DRP work has begun, the DDRP team might insist that the owners of the IA assets in question address some of these faults. With regard to these issues, political differences between the DDRP team and the owners of IA assets are possible.
• The additional testing of IA assets tends to expand dramatically the set of stakeholders affected by the DRP, to include the owners, users, and maintainers of the IA assets.
• The additional testing of IA assets can increase the need to interrupt revenue streams temporarily, and increase the number, duration, and frequency of such interruptions.
• The additional testing of IA assets can require expertise and staffing beyond the DRP project team, which can disrupt other elements of the organization as the people needed are temporarily assigned to IA testing.
• The additional testing of IA assets can reveal unanticipated consequences of the DRP alterations, which can trigger re-planning or re-design of the DRP during its execution. That re-planning or re-design, in turn, can trigger alterations in the DDRP.

The need to re-test assets not directly touched in the DRP is more likely when the DRP alters the external behavior of any of the DBA assets. The goal of many DRPs is improvement of the internals of assets without altering their external behavior, except possibly for performance improvements. This goal is desirable because it limits the need for re-testing and re-certification of IA assets. However, some kinds of technical debt appear in the externals of the DBA assets, including their architecture, behavior, appearance, or interfaces. Compared to DRPs that do not alter the externals of DBA assets, retiring the kinds of technical debt that alter the externals of DBA assets is inherently more difficult and more risky because it requires more extensive re-testing and re-certification of both DBA and IA assets.

4.   Multiple sites are directly touched

When the DRP entails modification of technological assets of geographically dispersed organizations, one consequence is a tendency to increase the wickedness of the DDRP. This comes about because of factors including the following:

• Sites might be dispersed not only geographically, but they might also be separated by language boundaries, legal jurisdictions, cultural divides, time zones, and much more. The required work can vary from site to site for technical reasons and because of variations arising from these non-technical factors.
• The multiple sites might have different landlords, with different lease agreements governing the organization’s occupancy of the property. This is just one of many factors that increase the numbers of stakeholders and exacerbate their heterogeneity. And the leases might constrain the kind of work that can be performed according to the day of the week or time of day.
• If local vendors provide services such as communications or Internet connections to some of the sites, and if the work of the DRP involves these technologies and the local vendors, the task of coordinating all the different players can be complex and riddled with unanticipated obstacles.

For example, if the work involves networking hardware and software, work that we might prefer to perform at night or on a weekend might need to be carried out during business hours for some of the sites. For a global enterprise, there might not be a time of day when no sites are conducting regular business.

As a second example, consider a network upgrade for the retail branch offices of a global bank. If that upgrade requires trenching for new cable connections, the project design must take into account local regulations governing the trenches, including how they must be permitted, dug, re-filled, covered, and marked while still open. These regulations vary with national and sometimes local jurisdiction. The complexity causes most organizations to rely on local vendors, but even then, the vendor selection process must include reliable vendor assessment and evaluation. Scheduling becomes a complex and risky endeavor.

For these reasons, a DDRP that involves technological assets housed at multiple sites geographically dispersed has an elevated probability of exhibiting the properties of a wicked problem.

5.   Government agencies and/or industrial standards organizations must re-certify assets

Another driver of stakeholder expansion is the need for re-certification of assets after they’ve been modified in the course of executing the DRP. The certification agencies can range from local and municipal regulators to national regulators and pan-industrial standards organizations. The sheer number of possibilities is itself a contributor to increased wickedness, but the nature of the operating style of these organizations merits special notice.

For the most part, these agencies operate without competitors, whether they are government elements or private organizations. Perhaps for this reason, “customer service” might not be their strength, and gaining timely cooperation from them might be a challenging undertaking. Even though re-certification might be a small part of the DRP, it can easily become a blocking obstacle. Researching these requirements and their associated lead times, and maintaining a current knowledge base about them, can be a non-trivial task of the DDRP.

6.   Non-technical stakeholders must change their behavior

Generally, people don’t like to change how they do their jobs. There are exceptions, of course, if they recognize a benefit that arrives in some immediate and visible way. But unless there is a recognizable benefit, requiring people to change their work patterns as part of a DRP is likely to increase the wickedness of the DDRP. And the difficulty is more problematic if the people affected are technically unsophisticated, because they’re less likely to appreciate the value of managing technical debt, and less likely to accept explanations of that value when those explanations are offered.

DDRP wickedness increases in this case because, in addition to retiring the technical debt, it must address the tasks of motivating and training the affected population. That requires preparing materials, scheduling and accounting for the time spent in training, and monitoring training effectiveness. The business case must also address these issues, but in addition, it must provide evidence required to defuse any political opposition that might otherwise develop.

7.   Discovery of major unanticipated complexity triggers re-design

Unanticipated complexity happens in almost every project of almost any kind. But for DDRPs, unanticipated complexity significant enough to trigger significant adjustment or re-design of the DRP during execution of the DRP is another matter. Such a discovery can mean that the DBA assets or their connections to the IA assets have changed since the plan was developed. Or it can mean that the design team had an incomplete or incorrect understanding of the problem they were trying to solve. These events can occur for a number of reasons.

Technical causes are perhaps more easily imagined, so I’ll focus on non-technical causes, which can actually be more serious. For example, suppose that a political alliance enabled the VP of Sales and the VP of Engineering to reach a deal that enabled the DRP team to work on some DBA assets critical to the Sales function, by taking them off line for defined periods. If that political alliance weakens, or if the deal between the two VPs collapses for some other reason, the scheduled downtime of those assets might vanish or be shortened dramatically. This pattern is more likely to arise in situations in which the DDRP team is not a party to such agreements. The DDRP team must be a party to any agreements regarding access to assets by the DRP team.

As a second example, consider what happens when the enterprise undertakes an acquisition that wasn’t revealed to the DDRP team during their design effort. Because chances are good that the DDRP would have a significant amount of re-work to do in such cases, the DDRP team must be kept informed of any organizational changes that could affect the DRP, for the active life of the DRP.

Re-designing the DDRP can take time. Elements of the DDRP that have short shelf lives must be revisited during re-design. And the need to re-design can also indicate gaps in the DDRP team’s understanding of the problem.

All of these conditions tend to move the DDRP in the direction of increased wickedness.

8.   Weekend or middle-of-the-night work periods are required

The need to perform critical operations on weekends or in nighttime hours suggests three things. First, the work is risky in the sense that undetected faults that go into production can lead to costly operational errors. Second, the organization lacks a simulated operating environment that emulates the actual operating environment faithfully enough to enable detection of errors before deployment. Third, and finally, the organization lacks a rapid rollback mechanism that can restore the original state of an asset if the new modified state proves problematic when deployed.

These last two factors—the lack of a simulated operating environment and the lack of a rapid rollback mechanism—should be corrected if multiple DRPs are anticipated. Cost is usually the blocking issue. However, that cost must be compared against the cost of retarding all future DRPs, and the cost of any operational failures arising from deploying faulty systems.

Continued refusal to provide a simulated operating environment with rapid rollback increases the wickedness of this and any future DDRPs.

9.   Rollback(s) of attempted changes triggers re-design

A rollback is an incident in which, in the course of executing the DRP, it has become necessary to revert some (or all) of the work that has already been performed. Minor rollbacks do happen. But a major rollback, or a rollback the necessity of which is discovered long after completion of the work in question, could be an indication of a deep misunderstanding of the consequences of the work involved. Because that misunderstanding could have consequences not yet recognized, such a rollback could suggest that the wickedness if the DDRP had been underestimated.

Let DBA_f (faulty DBA) be the set of assets in the DBA that formerly contained some of the debt being retired, and which had been altered as part of the DRP, and whose alterations contained or led to exposure of some kind of fault(s) that forced a rollback after they were deployed. Let DBA_fw (wicked-faulty DBA) represent the subset of DBA_f for which that rollback did trigger a re-design of the DDRP. Then wickedness of the DDRP is correlated with the size of DBA_fw and the extent of the DDRP re-design that the rollback triggered.

For example, let E_fw be a member of DBA_fw. And suppose that E_fw is a modular element of a system that monitors the click-through behavior of users of a Web site. It records data for later analysis, and because of the fault it does so incorrectly. When the errors are discovered, the module is withdrawn and replaced by the original, unaltered, debt-bearing form. Because E_fw contaminated the original database, data rollback is impossible. The error was discovered, but there is no way to re-capture the data that has been lost. That’s why the DDRP (and after that, possibly the DRP) must be re-designed. This scenario is an example of Criterion 5. If there are political consequences for the loss of data, it could be an example of Criterion 10.

This example suggests how the frequency of incidents that trigger re-design of the DDRP can be an indicator of the wickedness of the DDRP.

Example of a non-indicator: the I-35 SR-30 interchange near Ames, Iowa

Just outside Ames, Iowa, is an interchange between Interstate 35 (a four-lane, divided, limited-access roadway) and U.S. Route 30 (also four-lane, divided, but not limited-access). The interchange is a conventional cloverleaf design. The “leaves” are rather tight, though, and consequently, there have been numerous rollovers and crashes at this interchange. We can regard these tight cloverleaf ramps as technical debt in the highway system, and the rollovers and crashes as metaphorical interest charges on that debt.

In 2016, construction began on a new “flyover” exit ramp from northbound Interstate 35 onto westbound U.S. Route 30. The objective was to reduce the number of accidents at the interchange by replacing the current tight-curvature cloverleaf ramp with a flyover exit ramp with a longer radius of curvature. We can regard this project as a Debt Retirement Project (DRP). The project that planned that DRP was an effort to Design a Debt Retirement Project (DDRP).

Completion of the DRP was scheduled for November 2018. When completed, the new ramp will replace the northeast leaf of the cloverleaf. Like most civil engineering projects, this project does have some elements of wickedness, but they were dealt with effectively. Nevertheless, a construction error is delaying completion [Magel 2018] [Iowa DOT 2018]. The error involves the height and position of the bolt anchors where steel bridge beams will connect to the concrete piers of the new flyover ramp. Six piers have been constructed to support the flyover. Those piers are being corrected by jackhammering the concrete tops, leaving the steel reinforcement in place. Then the beam anchors are positioned correctly, and concrete re-poured. The length of the delay in completion hasn’t yet been announced.

This effort, which constitutes a rollback and re-deployment, is a significant project in itself. It requires scheduling the work to be performed, but it also requires scheduling highway lane closures and lane shifts, working around high-volume traffic periods, and possibly pouring concrete in winter conditions. And after the piers are corrected, the bridge beam placement and bridge roadbed work must proceed on a new schedule.

Consequently, the construction error triggered a redesign of the flyover project’s DRP. But it probably did not trigger a significant re-design of the DDRP. The construction error is therefore unlikely to be an indicator of significant additional wickedness for the DDRP.

Last words

You can become better managers of the risk of unanticipated wickedness. If your organization is embarking upon a long-term program of technical debt retirement, you’ll be executing many DDRPs and DRPs. Gathering data about incidents of unanticipated wickedness in DDRPs can be a useful practice, if you use that data when you design new technical debt retirement projects.

References

[Iowa DOT 2016] “Construction drawing for the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation, February 2, 2016.

Available: here; Retrieved: October 31, 2018

• Nine indicators of wickedness

[Iowa DOT 2018] “Work Continues on the Northbound I-35 Flyover Ramp at U.S. 30 Near Ames,” Iowa Department of Transportation News Release, June 27, 2018.

Available: here; Retrieved: October 31, 2018

• Nine indicators of wickedness

[Magel 2018] Todd Magel. “Uh-oh! Construction crews must redo $23 million project after big mistake,” KCCI News, July 11, 2018.

Available: here; Retrieved: October 31, 2018

• Nine indicators of wickedness

[Rittel 1973] Horst W. J. Rittel and Melvin M. Webber. “Dilemmas in a General Theory of Planning”, Policy Sciences 4, 1973, 155-169.

Available: here; Retrieved: October 16, 2018

• Glossary and Terminology
• Retiring technical debt can be a wicked problem
• Degrees of wickedness
• Nine indicators of wickedness
• Retiring technical debt from irreplaceable assets
• Legacy technical debt retirement decisions