The Tech Debt Spiral
In the world of health insurance, a provider enters a death spiral when healthier people opt out of coverage, forcing the sick to pay ever-increasing premiums. In the world of software development, an organization enters a spiral of tech debt when its code and infrastructure become so difficult to maintain that releasing any incremental improvement becomes increasingly difficult within the short timeframes that many companies use to drive their sprints.
There are really only two ways the needle can go: every change you make is either increasing or decreasing your technical debt. You are either improving your code and infrastructure by making it easier to understand, test, maintain, and iterate on, or… you’re not. To quote the eternal wisdom of Chris Farley in Tommy Boy, “there ain’t no third direction.”
In the real world, we are always tacking like a sailboat, left and right, in order to steer towards our desired goal. A good sailor knows how to navigate across these winds, and so too must a good developer know how and when to make some messes and when to pick up the pieces.
Kinds of Technical Debt
You can boil technical debt down to a few different facets, but the most meaningful distinctions for me come down to these: intentional vs. unintentional and acknowledged vs. unacknowledged. Imagine building a prototype for a feature that needs the ability to select a country from a list. The company is currently doing business only in the People’s Republic of Techistan, so you can hard-code that option and move on to building other parts of your feature. This is taking on technical debt intentionally.
When technical debt is intentionally generated, it should be acknowledged (e.g. by creating a ticket in the backlog) so that the product- and business-people have visibility into the hidden costs involved in delivering a feature. Yes, this is simple accounting, but communication is harder than coding.
The more insidious type of technical debt comes from lack of experience, when developers are unaware of the longer term implications of their particular solutions. Unintentional technical debt can never be easily acknowledged or backlogged because the authors were not intentionally introducing it. No code is perfect, but tech debt tends to flourish more with inexperienced developers and in organizations that lack sufficient guidance and standards. This kind of technical debt may show up as poorly structured code, improper data models, inaccurate database indexes, overly complicated infrastructure, or it might even be the choice of a language that is ill-suited to a particular problem (e.g. think Python for web development, Ruby for concurrency, or… Node (kidding… I think)).
Signs that Tech Debt is Taking Over
There are signs that indicate when tech debt is becoming a problem:
Features are taking longer and longer to deliver
Technical debt can be a big reason why your teams are underperforming. The friction of dealing with bad code and messy workarounds taxes everyone’s performance and motivation. “I want a job writing half-assed band-aid fixes rather than coming up with elegant solutions to real problems” said no developer ever.
Excuses are increasingly made to get something out the door
You know you’re not doing things right — maybe you took a few shortcuts with respect to coding conventions, test coverage, or meaningful code reviews, but hey, it was all for a good cause!! Discussions with management start sounding like conversations with a drug addict: “yeah yeah yeah, but I need my feature! I’ll pay you back, I promise!”
Residual debt is not addressed
There should be tickets in each sprint that specifically address the fixing of tech debt. You might even be lucky enough to get an entire sprint devoted to this. But when you’re in a tech debt death spiral, new features are constantly introduced, which in turn, generate more bugs. In the end all your time is spent chasing your tail; ironically, you are never allowed the time to fix the problems that got you into this mess in the first place.
You can tell a lot about the health of an organization’s software and its longterm prognosis by looking at their sprints. Are there any forward-looking tickets in there that are specifically geared towards paying back tech debt? Or are the tickets a hodgepodge of pushing features and the accompanying bug-fixes? Addressing tech debt is part of good software hygiene: without it, the corpus of code becomes bloated and prone to something akin to cardiac arrest.
Note to self: a good question to ask a prospective employer during an interview is “what percentage of your tickets each sprint is devoted to paying back tech debt?”
Infrastructure becoming overly complicated
Robert C. Martin’s book on Clean Code may be the most important book about technical debt I have ever read, even though it never uses that term. Uncle Bob wisely reminds us that “comments do not make up for bad code.” This sentiment can be extrapolated onto infrastructure. If your organization relies on a complex web of interconnected services which requires the continual upkeep of detailed flow charts for anyone to understand what’s going on, then maybe the solution isn’t to improve the diagram. Maybe the solution is to simplify the infrastructure.
Microservices deserve special mention here: more often than not, I have seen them as a symptom of poor software hygiene and unhandled technical debt than I have seen them as a viable solution to problems of efficiency and scale. Why? Because rather than addressing the fundamental problems that made one app/repo/service untenable, parasitical development practices instead target a new host. Without proper software hygiene and a habit of addressing technical debt, chances are that the new app/repo/service will become just as untenable as the last.
Tangential note: this is why the idea of colonizing Mars is absurd. Until we can demonstrate a fitness for sustainable living, adding more resources is a temporary solution at best.
High developer turnover, especially the departure of senior developers
There are many complex reasons why employees may choose to leave a company, but if an organization is experiencing higher-than-normal rates of burnout, it may indicate toxic levels of technical debt. Those with more experience (and more job offers) are probably more likely to recognize that their future is limited in an organization that is unable or unwilling to clean up its messes. Look around: where are the Jedi masters?
Tech debt is not specifically acknowledged
It is no use to wave your hands in the air and say “yeah, it’s a mess”: there must be tickets in the backlog (or in the sprint) targeting specific instances of technical debt.
How to Escape?
Most software companies struggle with managing tech debt, just like most of us citizens struggle with financial debt. Managing technical debt is a lot like maintaining your own body through regular exercise, healthy diet, and proper hygiene: your organization must either start healthy or get healthy. There are precious few examples of companies who conscientiously began their journey with a culture committed to the proper management of technical debt. The only ones that come to mind are Basecamp and Pivotal. Most companies need to “get in shape” and sometimes that is more difficult than staying healthy to begin with.
Perhaps our notions of good software hygiene are something like our notions of being “healthy” 100 years ago or so, when smoking was good, exercise was ridiculous, and we weren’t drowning in dietary problems caused by processed foods. Software is still relatively young and it changes so quickly that we have not yet learned about how to keep it healthy. I don’t think there are any quick solutions to complex long-term problems like this, but here’s my stab at it:
- Get experienced people involved; inexperienced developers may not even recognize there’s a problem
- Acknowledge tech debt when it occurs (e.g. by creating a backlog ticket)
- Devote a percentage of time each period to addressing tech debt
- Require meaningful code reviews by experienced developers (keep the code clean)
- Increase your developer skills to help reduce unintentional debt
Happy coding!
