Tag: automation

Posted on

Automation-assisted technical debt retirement

Last updated on July 16th, 2021 at 04:23 pm

Collapse of the I-35W bridge in Minneapolis, Minnesota
The I-35W Bridge collapse, day 4, Minneapolis, Minnesota, August 5, 2007. The proximate cause of the collapse was underweight gusset plate design, which made the bridge vulnerable to the increased static load due to concrete road surfacing additions over the years, and to the weight of construction equipment and supplies during a repair project that was then underway [NTSB 2008].

When we conduct maintenance or technical debt retirement projects involving assets that must remain operational during project execution, we risk stressing the asset in ways that extend beyond its safe operation envelope. The National Transportation Safety Board found that this occurred in the case of the I-35W bridge collapse. These effects are more difficult to imagine in software systems, but they can occur when we shift load from the systems undergoing modification to other systems. Those load-bearing systems can then become overloaded. Or these effects can occur when we shift load not from one asset to another, but from one time window to another on the same asset. The result can be high loads in some time windows.

Photo by Kevin Rofidal, United States Coast Guard, courtesy Wikimedia Commons.

When we transform assets to retire some of the technical debt they carry, service disruptions are sometimes necessary. To minimize service disruptions while technical debt retirement efforts are underway, it’s advantageous to automate some procedures. Automation-assisted technical debt retirement provides two important benefits: reduced disruption of operations and reduced incidence of errors.

The meaning of automation

I’m using the concept of automation a bit loosely here. I don’t mean to imply that these procedures are autonomous. What I mean is that engineers have available tools for performing many operations with a minimum of thought. For example, in this sense of automated, an engineer can issue a command such as, “Test Module Alpha Using Test Suite Delta.” That command executes a predefined set of tests. Following execution, the appropriate engineers receive the results. The tool also archives those results appropriately. If the results are anomalous, engineers can then take appropriate action.

Benefits of automation-assisted technical debt retirement

The more obvious benefit of automated procedures is speed. For example, an asset removed from service for testing can be returned to service more quickly if the testing is automated. And if trouble erupts during operations of a newly transformed asset, engineers can swap the untransformed asset back into place quickly. So-called roll-out and roll-back tools are just a few of the many elements of a set of practices collectively known as continuous delivery [Humble 2010].

The second benefit of this kind of automation is error avoidance. For example, inconsistent or incomplete testing can fail to find errors and defects, and that leads to rework and further disruptions. Performing tests incorrectly, finding “defects” that aren’t there, is another way to generate trouble. Automated procedures are much less prone to error if we maintain, test, and certify them periodically. For example, consider subjecting a module to a particular test suite. With automation assistance, engineers needn’t remember (or take time to look up) how to prepare the asset for tests. They  needn’t remember (or take time to look up) how to run the tests, or what the members of the test suite are. Long advocated as an essential element of sound engineering practice, test automation can avoid some of these problems. But it’s far short of a panacea [Bach 1999].

Other automation opportunities

In some situations, we can automate debt retirement itself. When we can retire instances of the technical debt in question by performing an automated transformation on an asset, the transformation is faster and more reliable.

A most important practice associated with automation-assisted technical debt retirement is automation-assisted regression testing. Investments in thorough and focused regression testing have potentially shockingly high returns in the debt retirement context. They can be just as valuable during development and routine maintenance.

To perform a regression test on an asset that has undergone a change is to examine its behavior under a specified set of conditions. Such investigations can determine whether those changes caused the asset to misbehave. So a regression test determines whether the asset has regressed as a result of the change. Automated or automation-assisted regression tests help the project team detect problems in assets that they’ve transformed. And that’s much better than having the business units that depend on those assets encounter problems during operations [Ge 2014].

Many of these same regression tests can also be useful during enhancement and ongoing maintenance of the asset. Often, investing in automated regression tests in advance of the debt retirement project can enhance development and maintenance performance relative to those assets. Later, when the debt retirement project begins, the previously obtained results of regression tests will already be available.

Last words

For some debt retirement projects, specially created automated regression tests might be beneficial. Assign engineers to continual automation tool development for debt retirement projects. That’s probably the best way to support these needs.

These automation capabilities are unlikely to be available commercially, because they’re so specialized to the asset being tested. Because general applicability is unnecessary, building them in-house is both practical and economical.  If people with the necessary skills are unavailable, acquire them. We can justify these investments economically if we take into account the savings from reduced service disruptions during technical debt retirement projects.

References

[Bach 1999] James Bach. “Test Automation Snake Oil!” (1999).

Available: here; Retrieved: January 2, 2019

Cited in:

[Ge 2014] Xi Ge and Emerson Murphy-Hill. “Manual Refactoring Changes with Automated Refactoring Validation,” Proceedings of the 36th International Conference on Software Engineering. ACM, 2014.

Available: here; Retrieved: January 1, 2019

Cited in:

[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.

Cited in:

[NTSB 2008] National Transportation Safety Board. “Board Meeting Executive Summary: Collapse of I-35W Highway Bridge, Minneapolis, Minnesota, August 1, 2007,”, November 13, 2008.

Available: here; Retrieved: January 3, 2019.

Cited in:

Other posts in this thread

Posted on

Technical Debt: So What?

Last updated on July 8th, 2021 at 11:44 am

This post is for readers skeptical that technical debt is much of a problem. Some believe that technical debt is just the latest buzzword engineers use to justify budget and schedule overruns.

I have no knowledge of your specific situation, but technical debt is a real thing, and it probably affects your organization. Skepticism, though usually healthy, is unwise when it comes to technical debt. Here’s the short version:

If you produce or use technology in your organization, you’re probably carrying technical debt. It’s costing you real money, it’s slowing you down, and unless you address it, it will increase. Eventually it will take you out of the game.

A knight’s armor
A knight’s armor. To this day, in the United Kingdom, a law entitled ‘Statute Forbidding Bearing of Armour (1313)’ remains in effect. As bodies of legislation grow and evolve, they develop inconsistencies, and outmoded laws accumulate. They are the legislative equivalent of technical debt.
Technical debt makes systems more difficult to maintain, less cybersecure, more difficult to enhance, and more expensive to operate. This makes systems less effective in achieving organizational objectives. There is some disagreement about the definition of technical debt. But there’s broad agreement that the problem is growing rapidly [Fowler 2003]. If current trends continue or accelerate, someday soon many of our technology-based assets will become unmaintainable and cyber-indefensible. The people, enterprises, and governments that depend on those assets will be unable to adapt rapidly enough to changing markets, changing technologies, changing cyber-threats, and changing customer needs. If we are ever to gain effective control of technical debt, we must change organizational technology management policy.

Technical debt afflicts companies large and small

Technical debt afflicts organizations of all sizes. The massive problems—the ones that sometimes make the news—tend to belong to big corporations. For example, Google’s code base of “hundreds of millions” of lines of code once contained dependencies among its modules that were ungoverned (and ungovernable) [Morgenthaler 2012]. The sheer number of dependencies and the frequency of changes so slowed the development process that it affected Google’s operations. They dealt with this form of technical debt with three strategies: exploit automation, make it easy to do the right thing, and make it hard to do the wrong thing.

But small companies are also affected. Consider the fictitious company Alpha Properties LLC. Alpha manages condominium associations with fewer than 100 units. They provide excellent value to small clients by exploiting automation to keep their own operating expenses low. Many of their automation assets are implemented as Microsoft Excel macros. When Microsoft released Excel 2013, Alpha’s macros would have been affected. So they elected to incur technical debt by remaining in Excel 2010. However, mainstream support for Excel 2010 ended in October 2015, with extended support scheduled to end in October 2020. Alpha realizes that they must retire this debt well before that, but finding the resources to do it has been a challenge.

Technical debt exacts a high price

For non-engineers, especially policymakers, what exactly is the technical debt problem? It afflicts complex technological assets. That means almost anything humans can construct—highways, bridges, computers, satellites, software—anything. And that includes assets that have no physical manifestation, such as software, surgical procedures, and legislation. All these assets have associated bodies of knowledge, both of which evolve. When they do evolve, technical debt can arise. It can reside in the asset, in its associated body of knowledge, in the assets we use to interact with the asset, or all three.

We’re dealing with the consequences of technical debt when we’re aware of structures within or around an existing asset that can be improved, but we’ve deferred those improvements. Later, we find that making a change to an existing asset is so complicated and delicate that only a few experts can make the change. When they do, the cost of the effort is difficult to predict with useful precision, and there’s a significant probability of their failing multiple times before they finally succeed—if they ever do succeed. When we include all cost sources, costs can be high enough to rival or exceed the initial development cost of the asset, even when the changes in question seem relatively small.

Last words

Briefly, the technical debt problem is that as technological assets evolve, they can become increasingly difficult to maintain, defend, enhance, or extend. The difficulty can become so great that many owners of technological assets choose to begin anew rather than continue to operate the assets they have.

References

[Bach 1999] James Bach. “Test Automation Snake Oil!” (1999).

Available: here; Retrieved: January 2, 2019

Cited in:

[Fowler 2003] Martin Fowler. “TechnicalDebt,” blog entry at MartinFowler.com, 1 October 2003.

Retrieved January 2, 2016, available at here; .

Cited in:

[Ge 2014] Xi Ge and Emerson Murphy-Hill. “Manual Refactoring Changes with Automated Refactoring Validation,” Proceedings of the 36th International Conference on Software Engineering. ACM, 2014.

Available: here; Retrieved: January 1, 2019

Cited in:

[Humble 2010] Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation, Pearson Education, 2010.

Cited in:

[Morgenthaler 2012] J. David Morgenthaler, Misha Gridnev, Raluca Sauciuc, and Sanjay Bhansali. “Searching for Build Debt: Experiences Managing Technical Debt at Google,” Proceedings of the Third International Workshop on Managing Technical Debt (MTD 2012), Piscataway, NJ: IEEE Press, 2012, 1-6.

Available: here; Retrieved: November 11, 2017

Cited in:

[NTSB 2008] National Transportation Safety Board. “Board Meeting Executive Summary: Collapse of I-35W Highway Bridge, Minneapolis, Minnesota, August 1, 2007,”, November 13, 2008.

Available: here; Retrieved: January 3, 2019.

Cited in:
Show Buttons
Hide Buttons