Posted on

How outsourcing leads to increasing technical debt

Last updated on July 8th, 2021 at 01:37 pm

Most of the nontechnical precursors of technical debt that afflict in-house work also afflict outsourced work. For example, the planning fallacy affects internal planners. But it also afflicts the bidders for contracts that cover outsourced work. As described in “Unrealistic optimism: the planning fallacy and the n-person prisoner’s dilemma,” Boehm, et al., call this the “Conspiracy of Optimism.” [Boehm 2016] But outsourcing engineering work can introduce pathways for incurring technical debt that are specific to outsourcing.

The risks of incurring technical debt from outsourcing

Green fields
Green fields. Greenfield outsourcing, also known as startup outsourcing, is the outsourcing of activity that the enterprise has never performed in-house. Greenfield outsourcing is especially tricky with respect to technical debt formation. The risk arises because much of the expertise necessary to perform the work in question is probably not resident within the enterprise. That void in enterprise expertise makes for difficulties in managing technical debt in the outsourced artifacts.
Outsourcing is inherently more likely to incur technical debt than is equivalent work performed in-house. When most enterprises contract for development of systems or software, the criteria for acceptance rarely include specifications for maintainability or extensibility. This happens, in part, because such qualitative specifications are difficult to define quantitatively. That’s why the condition of deliverables relative to maintainability and extensibility is so variable. Outsourced deliverables can best be described as bearing an unknown level of technical debt.

The root cause of the problem is likely a lack of a universally accepted metrics for quantifying technical debt [Li 2015]. That’s why it’s difficult to specify in the vendor contract an acceptability threshold for technical debt. And since the consequences of technical debt in deliverables potentially remain hidden during the lifetime of the outsource contract, years might pass before the issue becomes evident. That inevitably complicates the task of understanding the root cause of the problem.

Six ways outsourcing can contribute to technical debt

In what follows, I use the term vendor to denote the organization performing the outsourced work. I use the term enterprise to denote the organization that has outsourced a portion of its engineering work. The retained organization is the portion of the enterprise directly relevant to the outsourced work, and which has remained in-house.

Among the mechanisms that lead to incurring technical debt in the outsourcing context are the six mechanisms sketched below. Gauging the size of these effects by auditing deliverables or by auditing the internal processes of the vendor could be helpful in managing levels of technical debt arising from outsourcing.

This list isn’t exhaustive. Quite possibly other phenomena also contribute to technical debt formation in the context of outsourcing.

Progressive erosion of retained organization capabilities

Over time, after outsourcing work on a particular asset, the enterprise can expect erosion of some engineering expertise. The expertise most vulnerable is that needed to manage, evaluate, understand, or, if need be, to re-insource (or backsource) the outsourced work [Willcocks 2004][Beardsell 2010]. Typically, staff who formerly performed the outsourced work do move on to other work, voluntarily or not, either within the enterprise or elsewhere. Indeed, cost savings from terminations and employee buyouts often accompany outsourcing decisions. That reallocation is part of the economic justification of outsourcing decisions.

But even if the enterprise continues to employ the people who formerly performed the outsourced work, the deleterious effects remain. Those employees, filling new roles, likely become less familiar with the current work and therefore less able to perform it. And since they’re now carrying out other assignments, their availability is limited. In the public sector, the organizations that perform the outsourced work exacerbate this phenomenon by recruiting staff from their former agencies [Kusnet 2007]. In manufacturing, Kinkel, et al., suggest that, paraphrasing, outsourcing disturbs the formation of internal competence [Kinkel 2016].

Thus, outsourcing engineering efforts can erode the ability of the enterprise to perform the outsourced work internally. Likewise, it erodes enterprise capability to monitor or evaluate the work when vendors perform it. Consequently, the enterprise is less able to monitor, evaluate, or retire any technical debt that accumulates in the outsourced work products.

Stovepiping among vendors

Most multi-vendor efforts use a separation-of-concerns approach [Laplante 2007] to dividing the work. That is, each vendor is empowered to use any approach it can, consistent with its own contract and statement of work. In some cases, two or more vendors might have overlapping needs that cause them each to produce similar capabilities as elements of their respective deliverables. Sharing their results is of course possible. But unless both of their contracts anticipate the need for sharing, sharing is unlikely. Failure to share those results that could have been shared can lead to incurring unrecognized technical debt.

Stovepiping within vendors

Technical debt formation is possible even if there is only one vendor. Different teams or individuals working for that vendor might unwittingly create elements with overlapping capabilities. That duplication could lead to technical debt, or it could constitute technical debt in itself. For example, two teams working for the same vendor might have similar needs, and might develop duplicative tools independently. As a second example, consider a version of stovepiping combined with temporal displacement. Suppose that one team is unaware that a previous effort for the same customer had already developed a capability that it now needs. That team then might re-create that already-existing capability.

Stovepiping within vendors is less likely when the vendor operates under a single vendor technical lead, and the enterprise interacts with that single lead through a single in-house technical lead. When either side of the relationship manages communications through multiple contacts, stovepiping is more likely. New technical debt is more likely to form.

Loss of continuity in the outsourced engineering staff

Unless the agreement between the vendor and the enterprise specifically addresses staff volatility, an additional risk arises. Staff turnover or reassignment within the vendor organization can lead to technical debt. This can happen in just the same way that turnover in-house can lead to technical debt. But the risk is most significant at interfaces between one version of the product or service and the next. With outsourcing, however, the vendor has little internalized motivation to control this phenomenon. Moreover, the enterprise likely has less control and less awareness of staff assignments within the vendor organization than it does within the enterprise. Thus, loss of continuity in the outsourced engineering staff is both more likely and more likely to lead to technical debt.

Reduced coordination of engineering approaches and business objectives

Lack of coordination between engineering approaches and business planning can cause technical debt formation. This happens because engineers create and deploy artifacts that they must revisit later. The need for rework arises after engineers learn of business plans they didn’t know about at the time they produced those artifacts. See “Failure to communicate long-term business strategy.” This scenario is more likely, and possibly irresolvable, when the enterprise withholds its long-term plans from the outsourcing vendor to protect its strategy.

Hiring and retention problems

In some instances, the enterprise has never before performed work like the outsourced work [Delen 2007]. This kind of outsourcing is startup outsourcing or greenfield outsourcing. In these cases, typically, the enterprise must reassign existing employees or hire new employees to interface with the outsource vendor. These roles are analogous to remote supervisors, except that the teams they supervise aren’t enterprise employees. Hiring and retaining people for these roles can be difficult. Startup challenges arise both within the enterprise and within the vendor organization. Recruitment and especially retention problems in these roles can decrease the likelihood of controlling technical debt, and increase the likelihood of incurring technical debt.

Last words

A policy that would address these risks is one that would facilitate accomplishing three objectives:

  • Retain organizational capability sufficient to assess the effect on technical debt of any outsourced engineering work
  • Represent the effect on technical debt faithfully in any financial models used in making the outsourcing decision
  • Include in financial models the effects of technical debt, the cost of carrying technical debt, and the effects on controlling technical debt when deciding whether to extend outsourcing contracts with vendors.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Zengyang Li, Paris Avgeriou, and Peng Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread

Posted on

Separating responsibility for maintenance and acquisition

Last updated on July 8th, 2021 at 01:28 pm

Separating responsibility for maintenance and acquisition or development of technical assets can lead to uncontrolled growth of technical debt. The problem arises when we measure without regard for technical debt the performance of the business acquisition function or the performance of the development organization. In that circumstance, technical debt is likely to expand unchecked. To limit such expansion, policymakers must devise performance measures that hold these organizations accountable for technical debt resulting from their actions.

Software systems

Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010
Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010 [NOAA 2013]. The storms were so severe that the floodwaters overtopped the gauge’s measurable range. Moreover, the National Weather Service (NWS) lacked a database of measurable ranges for flood gauges. Quoting the NWS report: “A lesson learned here was that maximum recordable river levels should be known by NWS staff, not only so staff aren’t surprised when this type of issue arises, but also to notify USGS personnel so that they can install a temporary gage and remove or elevate threatened equipment.” Technical debt, if ever I’ve seen it.
For systems consisting solely of software, separating responsibility for maintenance and acquisition or system development is risky. It enables the acquiring organization to act with little regard for the consequences of its decisions vis-à-vis maintenance matters [Boehm 2016]. This is unfortunate—it increases the rate of accumulation of new technical debt. And it increases the lifetime of legacy technical debt. This happens more frequently when the acquiring organization doesn’t suffer the MICs associated with the technical debt.

For example, a focus on performance of the organization that’s responsible for acquisition biases them in favor of attending to the direct and immediate costs of the acquisition. They are likely to have little or no regard for ongoing maintenance issues. The maintenance organization must then deal with whatever the acquired system contains (or lacks).

An analogous mechanism operates for organizations that develop, market, and maintain products or services. When there are software elements in their respective infrastructures, separation of the development function from the maintenance function enables the development function to act independently of the maintenance consequences of its decisions.

Systems that include hardware

But the separation-of-responsibilities mechanism that leads to uncontrolled technical debt isn’t restricted to software. Any technological asset that has ongoing maintenance needs (and most of them do) can potentially present this problem.

For example, in the United States, and many other countries, two streams of resources support publicly-owned infrastructure [Blair 2017]. The funding stream covers construction, operations and maintenance, and repairs. Its usual sources are taxes, tolls, licenses, other user fees, sale of ad space, and so on. The financing stream covers up-front construction costs, to bridge the period from conception through construction, until the funding stream begins delivering resources. The financing stream usually comes from bond sales.

Although legislatures, or agencies they establish, control both streams of resources, the effects of the streams differ fundamentally. The financing stream is dominant during construction and the early stages of the asset’s lifecycle. The funding stream is dominant after that—when maintenance and operations are most important. Legislators and agencies are generally reluctant to supply funding because of the impact on taxpayers and users. Legislators and agencies find financing much more palatable. For this reason, among others, funds for U.S. infrastructure maintenance are generally insufficient, and technical debt gradually accumulates.

So it is with technological assets in organizations. For accounting purposes, capital expenses are treated differently from operational expenses. The result is that operational expenses can have a more significant impact on current financial results than capital expenses do. This leads organizations to underfund operations and maintenance, which contributes to technical debt accumulation.

Last words

Control of new technical debt accumulation and enhancement of technical debt retirement rates is possible only if we can somehow hold accountable the acquisition or development organizations for the MICs that result from their actions. Securitization of the debt incurred, as I’ll address in a forthcoming post, is one possible means of imposing this accountability. But reserves are also required, because some of the debt incurred might not be known at the time the asset is acquired or created.

Separating responsibility for maintenance and acquisition or system development is actually a form of stovepiping. See “Stovepiping can lead to technical debt” for more on stovepiping.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Zengyang Li, Paris Avgeriou, and Peng Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread

Posted on

Stovepiping can lead to technical debt

Last updated on July 8th, 2021 at 01:18 pm

Stovepiping can lead to technical debt. Actual stovepipes are the tubes that vent exhaust from stoves. These tubes serve as a metaphor for the flow of information in “stovepiped” organizations. In stovepiped organizations, information can flow predominantly (or only) up or down along the parallel chains of command. But information can flow only rarely (or never) from a point in one chain of command across to some other chain of command [Waters 2010]. The stovepipe metaphor is imperfect, in the sense that in actual stovepipes, smoke and fumes rarely flow downwards. By contrast, in organizations, some information does flow down the chains of command. But the metaphor does clarify the problem of limited flow of information. Transferring whatever the organization learns in one metaphorical stovepipe into other metaphorical stovepipes is difficult.

Two forms of stovepiping

The stovepipes in a wood-burning stove in a farm museum
A wood-burning stove in a farm museum in Lower Bavaria (German: Niederbayern). Lower Bavaria is one of the seven administrative regions of Bavaria, Germany. The stovepipe, which is the black tube running upwards from the stove, channels smoke and fumes out of the kitchen into the chimney.
Stovepiping can occur in both organizational structures and in engineered systems. These two forms of stovepiping are intimately related, and both can lead to uncontrolled formation of new technical debt, or increased persistence of existing technical debt.

In organizational structures, stovepiping occurs when elements of different organizational units with similar capabilities act relatively independently. An example is the dispersal of some elements of the IT function out into IT’s customers. When independent organizations have similar technical needs, they’re at risk of generating new technical debt. The debt they generate results from independently implementing technological capabilities that duplicate each other.

Stovepiping occurs in engineering, for example, when the organization manages and maintains independently two distinct technological assets [McGovern 2003]. In that situation, distinct engineering efforts working on those assets might happen to solve the same problem, possibly in two different ways. Then each party might be either ignorant or possibly disparaging, of the other’s efforts.

How stovepiping relates to technical debt

In whichever way duplication of technological capability comes about, it can increase levels of technical debt. Alternatively, it can increase persistence of existing technical debt. These effects happen because the organization might need to execute future maintenance or enhancement efforts multiple times—once for each instance of the technical artifact. That exposes the organization to additional cost, additional load on its staff, and additional risk of creating defects and incurring liability. Compare this situation to one in which all units that need a particular asset share it. Duplication is expensive.

The problem is actually even more worrisome. First, suppose there exists a defect in one version of a technological artifact. The people who are aware of the defect might not realize that another version of the artifact exists. If that second version also has an analogous defect, its defect might go unrecognized for some time, with all the usual attendant negative consequences. Second, suppose there is a necessary extension of the artifact’s capabilities. The maintainers of one version might recognize the need for the extension and implement it. Meanwhile, the maintainers of other versions might not recognize the need for the extension. They might not take action until something bad happens or a possibly urgent need arises. It’s easy to conjure other unfavorable—and costly—scenarios.

Stovepiping in technological systems

In engineering more generally, stovepiping can occur internally in systems, even though only one business unit is involved, and even though the stovepiped artifacts serve purposes invisible to the world outside the system. This can occur whenever there is weak communication between the teams designing or maintaining the portions of the system that host the similar artifacts. For readers familiar with the Apollo XIII incident, the incompatibility of the two carbon dioxide scrubbers in the command module and the lunar excursion module serves as an example of the risks of technical stovepiping.

When distinct business units or functions operate their own engineering or IT organizations, there’s an elevated probability of duplicating technological assets. The same effect can occur when they depend on a shared engineering function but require similar work. This happens when the organizational structure or the timing of the work encourages separate engineering efforts. Engineering or IT functions operated separately under the control of distinct business units or functions can clearly produce duplicated capabilities. However, duplication can also occur when the engineering function is shared across distinct business units or functions. This happens when the actual people and teams performing the work differ for different efforts. And it can happen too when communication is weak between those teams, whether or not the efforts are conducted contemporaneously.

Last words

Because identifying these forms of technical debt after they appear is notoriously difficult, preventing their formation is preferable. Prevention is possible if the enterprise establishes mechanisms that facilitate consultation and sharing among elements of different, separately operated technology development or maintenance functions. In other words, the organization must “break” the stovepipes—no mean feat, politically speaking.

Another challenge, of course, is providing resources for such sharing mechanisms, because preventing technical debt is rarely recognized as a value generator. If it were so recognized, the resources would likely appear. Changes in cost accounting might make such recognition more likely.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Zengyang Li, Paris Avgeriou, and Peng Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.

Order from Amazon

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Waters 2010] Donald Waters. Global Logistics: New Directions In Supply Chain Management, 6th Edition, London: Kogan Page Limited, 2010.

Order from Amazon

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread