Musings on Tech Debt

My friend, mentor, and former boss, Robert Harris, recently posted about a novel approach he takes to making tech debt more tangible; he makes people fill out a credit slip whenever they take on tech debt. It’s worth reading his post (and giving him a follow).
As always, I have some thoughts on this topic…
I’ve always loved the way Robert N. Harris manages to make abstract and intangible concepts more concrete for the teams he leads and the stakeholders who are his peers. The “debt receipts” are one of countless examples.
I particularly like the psychology here. At one point I tried a “cash” budgeting system where my disposable income was limited to the physical currency that was in my wallet. Every transaction was palpable in a way that tapping a credit card was not. It made me more mindful because I saw the immediate cost at transaction time rather than some vague future statement that was “future Mike’s” problem.
I also like the presence of debt slips on the wall. This makes real an aspect of tech debt that we too-often overlook; we see the effect of tech debt that’s “written off.” A spike, a POC, or MVP is often riddled with tech debt but this is a time when the highest value we can deliver is not a perfectly engineered app but rather the answer to a question.
These are often questions like: “Is this approach feasible” or “is there sufficient market fit?” When the answer is no, we have little invested so we are free to pivot without succumbing to sunk-cost fallacy. As for the debt post-pivot, we simply take down and throw away the debt notes. Taking more of an agile (lower-case a) approach to building software allows us to make payments only when we have a high degree of confidence in an ROI.
All that said, I don’t believe “all code becomes tech debt on a long enough time scale…“ for several reasons.
As a concrete example, it’s not even 9:00 AM (I’m still working on my first cup of coffee), yet I have already used several pieces of software that have been working flawlessly for 40-50 years. We must ask why, then, does so much of the software we write have the shelf-life of milk?
I keep thinking about something I heard Dave Thomas say last year. “We don’t create things, we change them… development is a nested set of change cycles.” Harris’ more honest definition of technical debt is: “Any decision that helps you make progress in the short term, but will cause problems in the long term…“ I believe this gets to the heart of what Dave is talking about. If tech debt makes it harder to change things in the future, we are working against change-driven engineering.
The vast majority of the time we write software that must change. Weak modularity is the norm and requires writing code to compose modules into systems (that must change as systems change). We also excessively couple to frameworks that, so far, exhibit zero interest in longevity (c.f. most web frameworks). This latter source of tech debt is insidious as it isn’t immediately obvious that we’re opting into tech debt until the framework maintainers decide to release a new version and suddenly we’re either shifting time into keeping the old code working or we’re letting it decay while npm security warnings proliferate in our build output.
This is akin to purchasing a modular home. Sure the design is a little cookie-cutter and basic but it’s cheap, fast, and adequate for most needs. The problem is, we’re planting our modular home (an asset we own) in a trailer park on land we don’t own. The landowner is free to raise our rent, change the neighborhood covenant, or even sell the land out from under us. The high speed and low cost is an illusion.
I’m not saying don’t use frameworks, but I wish the hidden costs were more visible. Whether it’s frameworks, APIs, operating systems, or hardware; the status quo of software development means we will continue to be blindsided by unexpected and enormous balloon payments on debts that we (or someone else) created.
But what about the exception? Why am I using software that was written before PCs existed? The answer is simple; because it still works and is still valuable. Specifically, I’m referring to many of the apps that reside in the various /bin
directories in my linux environment. These apps conform to various standards (C, POSIX, etc.) which provides portability across both operating systems and hardware. They also typically only do a single thing which means I can conjure novel functionality from the ether by dynamically composing these single-purpose applications into powerful composites. This is not how we build software today and the wisdom has largely been forgotten as we chase novelty-bias and the latest shiny framework.
“Composability is to software as compound interest is to finance.”
Although I lament the fact that these ideas have largely fallen out of the mainstream, forward-looking individuals who are privileged to engage in long-term thinking are quietly working on a future where tech-debt can become far less pervasive.
Strategic tech debt will always have value, but we are drowning in unnecessary tech debt. Tech debt is not just the problem of deliberate decisions, or the decisions of those we rely on in our software ecosystem; it’s baked into the dominant model of software development. We also haven’t even begun to feel the effects of LLMs generating code without crucial context. AI won’t save us, it will just further shorten the lifespan of software.
Alternatives exist, and I’m actively working with others to make alternative practices available to the mainstream. Webassembly is poised to truly deliver on the promise of write-once-run-anywhere. As geopolitical tensions continue to ratchet up and compute needs evolve, we are probably nearing the end of the TSMC hegemony and entering an era of silicon sovereignty. As nation states onshore semiconductor supply chains, we will see a fracture in our current processor monoculture. LLVM is our bridge. The standards surrounding Linked Data provide a path to declarative data integration at massive scale, rather than our current imperative approach where we write the same code over and over again every time we need to integrate more data–then change that code every time the data source changes. The oft-overlooked aspects of REST are paving the way toward resource-oriented computing which allows software systems to realize the evolvability and economics of the web combined with the composability of the *nix environment.
I believe there is a growing hunger for long-term business-value capture in software and current practices aren’t getting us there. Unnecessary tech debt and accidental complexity need not be a foregone conclusion.