Posted on

How outsourcing leads to increasing technical debt

Most of the non-technical precursors of technical debt that afflict in-house work also afflict outsourced work. For example, the planning fallacy affects internal planners, but it also afflicts the bidders for contracts offered by the enterprise in the context of outsourcing. As described in “Unrealistic optimism: the planning fallacy and the n-person prisoner’s dilemma,” Boehm, et al., [Boehm 2016] call this the “Conspiracy of Optimism.” But outsourcing engineering work can introduce pathways for incurring technical debt that are specific to outsourcing.

Green fields
Green fields. Greenfield outsourcing, also known as startup outsourcing, is the outsourcing of activity that has never been performed within the enterprise. It’s especially tricky with respect to technical debt formation, because much of the expertise necessary to perform the work being outsourced is probably not resident within the enterprise. That void in enterprise expertise makes for difficulties in managing technical debt in the outsourced artifacts.

The risks of incurring technical debt associated with outsourcing are inherently elevated, even setting aside those factors that also afflict in-house activity. When most enterprises contract for development of systems or software, the criteria for acceptance rarely include specifications for maintainability or extensibility. This happens, in part, because such qualitative specifications are so difficult to define quantitatively. That’s why the condition of deliverables relative to maintainability and extensibility is so variable. Outsourced deliverables can best be described as bearing an unknown level of technical debt.

The root cause of the problem is likely a lack of a universally accepted metrics for quantifying technical debt [Li 2015]. That’s why it’s difficult to specify in the vendor contract an acceptability threshold for technical debt. And since the consequences of the presence of technical debt in deliverables potentially do not become clear during the lifetime of the contract under which the debt was incurred, years may pass before the issue becomes evident, which complicates the task of understanding the root cause of the problem.

In what follows, I use the term vendor to denote the organization to which work has been outsourced, and the term enterprise to denote the organization that has outsourced a portion of its engineering work. The retained organization is the portion of the enterprise directly relevant to the outsourced work, but which was not itself outsourced. Among the mechanisms that lead to incurring technical debt in the outsourcing context are the six mechanisms sketched below. Gauging the size of these effects by auditing deliverables or by auditing the internal processes of the vendor could be helpful in managing levels of technical debt arising from outsourcing.

This list isn’t intended to be exhaustive. Quite possibly other phenomena also contribute to technical debt formation in the context of outsourcing.

Progressive erosion of retained organization capabilities

Over time, the enterprise can expect erosion of the engineering expertise needed to manage, evaluate, understand, or, if need be, to re-insource (or backsource) the work that has been outsourced [Willcocks 2004][Beardsell 2010]. Typically, staff who formerly performed the outsourced work do move on to other work, voluntarily or not, either within the enterprise or elsewhere. Indeed, cost savings from terminations and employee buyouts often accompany — and economically justify — outsourcing decisions. But even if the enterprise continues to employ the people who formerly performed the work that is now outsourced, those employees, filling new roles, can become less familiar with the current work and therefore less able to perform it. And since they are now engaged in other assignments, their availability is limited. In the public sector, the organizations that perform the outsourced work exacerbate this phenomenon by recruiting from the agencies they serve [Kusnet 2007]. In manufacturing, Kinkel et al. suggest that, paraphrasing, outsourcing disturbs the formation of internal competence [Kinkel 2016].

In short, outsourcing engineering efforts can erode the ability of the enterprise to perform internally the work that is outsourced, or to monitor or evaluate it when performed by the vendor. Consequently, the enterprise is less able to monitor, evaluate, or retire any technical debt that accumulates in the outsourced work products. A policy that would address this risk is one that would (a) require retained organizational capability sufficient to assess the effect on technical debt of any outsourced engineering work; (b) require attention to technical debt issues in any financial models used in making the initial outsourcing decision; (c) require financial models to include the effects of technical debt and controlling technical debt when deciding whether to extend outsourcing contracts with vendors.

Stovepiping among vendors

Most multi-vendor efforts use a separation-of-concerns approach [Laplante 2007] to dividing the work. That is, each vendor is empowered to use any approach it can, consistent with its own contract and statement of work. In some cases, two or more vendors might have overlapping needs that cause them each to produce similar capabilities as elements of their respective deliverables. Sharing of their results is of course possible, but unless both of their contracts anticipate the need for sharing, sharing is unlikely. Failure to share those results that could have been shared can lead to incurring unrecognized technical debt.

Stovepiping within vendors

With regard to the efforts of a single outsource vendor, it’s possible that different teams or individuals working for that vendor might unwittingly create elements with overlapping capabilities. That duplication could lead to technical debt, or it could constitute technical debt in itself. For example, two teams working for the same vendor might have similar testing needs, and might develop testing tools independently. As a second example, in a version of stovepiping combined with temporal displacement, suppose that one team is unaware that a previous effort for the same customer had already developed a capability that it now needs. That team then might re-create that already-existing capability.

Stovepiping within vendors is less likely when the vendor operates under a single vendor technical lead, and the enterprise interacts with that single lead through a single in-house technical lead. When either side of the relationship is managed through multiple contacts, stovepiping is more likely, and new technical debt is more likely to form.

Loss of continuity in the outsourced engineering staff

Unless specified in the agreement between the vendor and the enterprise, staff turnover or reassignment within the vendor organization, between one version of the product or service and the next, can lead to technical debt in just the same way that turnover in-house can lead to technical debt. With outsourcing, however, the vendor has little internalized motivation to control this phenomenon, and the enterprise likely has less control—and less awareness—of staff assignments within the vendor organization than it does within the enterprise. Thus, loss of continuity in the outsourced engineering staff is both more likely and more likely to lead to technical debt.

Reduced coordination of engineering approaches and business objectives

Lack of coordination between engineering approaches and business planning can cause engineers to create and deploy artifacts that must be revisited later, when they learn of business plans that were unknown to the engineers at the time they produced those artifacts. See “Failure to communicate long-term business strategy.” This scenario is more likely, and possibly irresolvable, when the enterprise withholds its long-term plans from the outsourcing vendor to protect its strategy.

Hiring and retention problems

In some instances, commonly called startup outsourcing or greenfield outsourcing, the work being outsourced has never been performed within the enterprise [Delen 2007]. In these cases, typically, the enterprise must then reassign existing employees or hire new employees to interface with the outsource vendor. These roles are analogous to remote supervisors, except that the teams they supervise are not employees of the enterprise. Hiring and retaining people for these roles can be difficult, because of startup challenges both within the enterprise and within the vendor organization. Recruitment and especially retention problems in these roles can decrease the likelihood of controlling technical debt, and increase the likelihood of incurring technical debt.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Z. Li, P. Avgeriou, and P. Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread

Posted on

Separating responsibility for maintenance and acquisition

Last updated on July 24th, 2018 at 08:49 pm

Separating responsibility for maintenance and acquisition of technical assets can lead to uncontrolled growth of technical debt. When the performance of the business acquisition function or the performance of the development organization is measured without regard for the technical debt that arises as a consequence of their actions, technical debt is likely to expand unchecked. To limit such expansion, policymakers must devise performance measures that hold these organizations accountable for the technical debt arising from their actions.

Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010
Road damage in Warwick, Rhode Island, resulting from historic storms in March 2010 [NOAA 2013]. The storms were so severe that at least one river flood gauge “flat-lined” — the floodwaters overtopped the gauge’s measurable range. Moreover, the National Weather Service (NWS) lacked a database of measurable ranges for flood gauges. Quoting the NWS report: “A lesson learned here was that maximum recordable river levels should be known by NWS staff, not only so staff aren’t surprised when this type of issue arises, but also to notify USGS personnel so that they can install a temporary gage and remove or elevate threatened equipment.” Technical debt, if ever I’ve seen it.
For systems consisting solely of software, separation of responsibility for system maintenance and system development or acquisition enables the acquiring organization to act with little regard for the consequences of its decisions vis-à-vis maintenance matters [Boehm 2016]. This is an unfortunate state of affairs that increases the rate of accumulation of new technical debt, and increases the lifetime of legacy technical debt, because the MICs associated with the technical debt aren’t borne by the acquiring organization.

For example, a focus on performance of the organization that’s responsible for acquisition biases them in favor of attending to the direct and immediate costs of the acquisition, with little or no regard for ongoing maintenance issues. The maintenance organization is then left to deal with whatever the acquired system contains (or lacks).

An analogous mechanism operates for organizations that develop, market, and maintain products or services with software elements in their respective infrastructures. In that case, separation of the development function from the maintenance function enables the development function to act independently of the consequences of its decisions for maintenance matters.

But the separation-of-responsibilities mechanism that leads to uncontrolled technical debt isn’t restricted to software. Any technological asset that has ongoing maintenance needs (and most of them do) can potentially present this problem.

For example, in the United States, and many other countries, two streams of resources support publicly-owned infrastructure [Blair 2017]. The funding stream covers construction, operations and maintenance, and repairs. Its usual sources are taxes, tolls, licenses, other user fees, sale of ad space, and so on. The financing stream covers up-front construction costs, to bridge the period from conception through construction, until the funding stream begins delivering resources. The financing stream usually comes from bond sales.

Although both streams are controlled by legislatures, or by agencies they establish, the effects of the two streams differ fundamentally. The financing stream is dominant in the early stages of the asset’s lifecycle — during construction. The funding stream is dominant after that — when maintenance and operations are most important. Legislators and agencies are generally reluctant to supply funding because of the impact on taxpayers and users. Legislators and agencies find financing much more palatable. For this reason, among others, U.S. infrastructure maintenance is generally under-resourced, and technical debt gradually accumulates.

So it is with technological assets in organizations. For accounting purposes, capital expenses are treated differently from operational expenses, and the result is that operational expenses can have a more significant impact on current financial results than capital expenses do. This leads organizations to underfund operations and maintenance, which contributes to the accumulation of technical debt.

Control of new technical debt accumulation and enhancement of technical debt retirement rates is possible only if the acquisition or development organizations can somehow be held accountable for the MICs that result from their actions. Securitization of the debt incurred, as I’ll address in a forthcoming post, is one possible means of imposing this accountability. But reserves are also required, because some of the debt incurred might not be known at the time the asset is acquired or created.

Separation of responsibility for system maintenance and system acquisition or system development is actually a form of stovepiping. See “Stovepiping can lead to technical debt” for more on stovepiping.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Z. Li, P. Avgeriou, and P. Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread

Posted on

Stovepiping can lead to technical debt

Last updated on February 1st, 2018 at 07:28 am

Stovepiping can lead to technical debt. Actual stovepipes — the tubes that vent exhaust from stoves — serve as a metaphor for the flow of information in “stovepipe” organizations. In “stovepipe” organizations, information flows predominantly (or only) up or down along the parallel chains of command, and rarely (or never) from a point in one chain of command across to some other chain of command [Waters 2010]. The stovepipe metaphor is imperfect, in the sense that in actual stovepipes, smoke and fumes rarely flow downwards. By contrast, in organizations, some information does flow down the chains of command. But the metaphor does clarify the problem that whatever the organization learns in one metaphorical stovepipe isn’t easily transferred to other metaphorical stovepipes.

A wood-burning stove in a farm museum
A wood-burning stove in a farm museum in Lower Bavaria (German: Niederbayern), which is one of the seven administrative regions of Bavaria, Germany. The stovepipe, which is the black tube running upwards from the stove, channels smoke and fumes out of the kitchen into the chimney, and then, presumably, out of the farmhouse.

Stovepiping can occur in both organizational structures and in engineered systems. These two forms of stovepiping are intimately related, and both can lead to uncontrolled formation of new technical debt, or increased persistence of existing technical debt.

In organizational structures, stovepiping occurs when similar capabilities are implemented in elements of two or more different organizational units that act relatively independently of each other. An example is the dispersal of some elements of the IT function out into IT’s customers. When independent organizations have similar technical needs, they’re at risk of independently implementing technological capabilities that duplicate each other, at least in some respects.

In engineering, stovepiping occurs, for example, when two or more technological assets are managed and maintained independently [McGovern 2003]. In that situation, distinct engineering efforts working on those assets might happen to solve the same problem, possibly in two different ways, with each party either ignorant, or possibly disparaging, of the other’s efforts.

In whichever way duplication of technological capability comes about, it can lead to increasing levels of technical debt, or to increased persistence of technical debt. This happens because the organization might be required to execute future maintenance or enhancement multiple times — once for each instance of the technical artifact. That exposes the organization to additional cost, additional load on its staff, and additional risk of creating defects and incurring liability, compared to a situation in which technical assets are shared among all who need them.

The problem is actually even more worrisome. First, in the case of a defect found in one version of a technological artifact, it’s possible that the people who are aware of the defect might not realize that another version of the artifact exists. If that other version also has an analogous defect, its defect might go unrecognized for some time, with all the usual attendant negative consequences. Second, in the case of a necessary extension of the artifact’s capabilities, the maintainers of one version might recognize the need for an extension, and implement it. Meanwhile the maintainers of other versions might not recognize the need for extension. They might not take action until something bad happens or a possibly urgent need arises. It’s easy to conjure other unfavorable — and costly — scenarios.

In engineering more generally, stovepiping can occur internally in systems, even though only one business unit is involved, and even though the stovepiped artifacts serve purposes invisible to the world outside the system. This can occur whenever communication is weak between the teams designing or maintaining the portions of the system that host the similar artifacts. For those familiar with the Apollo XIII incident, the incompatibility of the two carbon dioxide scrubbers in the command module and the lunar excursion module serves as an example of the risks of technical stovepiping.

When distinct business units or functions operate their own engineering or IT functions, or when they depend on a shared engineering function but require similar work, there is an elevated probability of duplication of technological assets or capabilities. This happens when the organizational structure, or the timing of the work, encourages separation of the engineering efforts. Engineering or IT functions operated separately under the control of distinct business units or functions can clearly produce duplicated capabilities. However, duplication can also occur when the engineering function is shared across distinct business units or functions, but the actual people and teams performing the work differ for different efforts, and when communication is weak between those teams. This can happen whether or not the efforts are conducted contemporaneously.

Because identifying these forms of technical debt after they’re created is notoriously difficult, preventing their formation is preferable to trying to detect them post-implementation. Prevention is possible if the enterprise establishes mechanisms that facilitate consultation and sharing among elements of different, separately operated technology development or maintenance functions. In other words, the organization must “break” the stovepipes — no mean feat, politically speaking.

Another challenge, of course, is providing resources for such sharing mechanisms, because preventing technical debt is rarely recognized as a value generator. If it were so recognized, the resources would likely appear. Changes in cost accounting might make such recognition more likely.

References

[Beardsell 2010] Julie Beardsell. “IT Backsourcing: is it the solution to innovation?”, SMC Working Paper Series, Issue: 02/2010, Swiss Management Center University, 2010.

Available: here; Retrieved: February 15, 2018

Cited in:

[Blair 2017] Hunter Blair. “No free bridge: Why public–private partnerships or other ‘innovative’ financing of infrastructure will not save taxpayers money,” Economic Policy Institute blog, March 21, 2017.

Available: here; Retrieved: January 29, 2018

Cited in:

[Boehm 2016] Barry Boehm, Celia Chen, Kamonphop Srisopha, Reem Alfayez, and Lin Shiy. “Avoiding Non-Technical Sources of Software Maintenance Technical Debt,” USC Course notes, Fall 2016.

Available: here; Retrieved: July 25, 2017

Cited in:

[Delen 2007] Guus Delen. “Decision and Control Factors for IT-sourcing,” in Handbook of Network and System Administration, Jan Bergstra and Mark Burgess, eds., Boston: Elsevier, 929-946, 2007.

Order from Amazon

Cited in:

[Kinkel 2016] Steffen Kinkel, Angela Jäger, Djerdj Horvath, and Bernhard Rieder. “The effects of in-house manufacturing and outsourcing on companies’ profits and productivity,” 23rd International Annual EurOMA Conference, At Trondheim, Volume: 23, June 2016.

Cited in:

[Kusnet 2007] David Kusnet. “Highway Robbery II,” report of the National Association of State Highway and Transportation Unions (NASHTU).

Cited in:

[Laplante 2007] Phillip A. Laplante. What Every Engineer Should Know About Software Engineering. New York: CRC Press, 2007.

Order from Amazon

Cited in:

[Li 2015] Z. Li, P. Avgeriou, and P. Liang. “A systematic mapping study on technical debt and its management,” Journal of Systems and Software 101, 193-220, 2015.

Cited in:

[McGovern 2003] James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, and Elias K. Jo. A Practical Guide to Enterprise Architecture, Upper Saddle River, New Jersey: Prentice Hall PTR, 2003.

Order from Amazon

Cited in:

[NOAA 2013] NOAA/National Weather Service. “The March, 2010 Floods in Southern New England,” WFO Taunton Storm Series Report #2013-01, January 2013.

Available: here; Retrieved: January 30, 2018

Cited in:

[Waters 2010] Donald Waters. Global Logistics: New Directions In Supply Chain Management, 6th Edition, London: Kogan Page Limited, 2010.

Order from Amazon

Cited in:

[Willcocks 2004] L. Willcocks, J. Hindle, D. Feeny, and M. Lacity. “IT and Business Process Outsourcing: The Knowledge Potential,” Information Systems Management 21:3, 7-15, 2004.

Cited in:

Other posts in this thread