Technical debt is an ever-present issue in software creation that becomes bigger as a project scales. Although you may not be able to see technical debt, you are paying it all the time if you are not looking for it! Is it ever acceptable, and how should you work to reduce it as much as possible?
Where does technical debt come from?
We should start with what technical debt is not in order to determine where it actually comes from; otherwise, we may mistakenly identify things that are not technical debt as technical debt, eliminating our ability to control it.
Technical debt is not lazy programming, and these two concepts are not synonyms in any way. Huge, unsophisticated modules and overly verbose executables that run inefficiently are problems that can be virtually eliminated by inspiring or changing up your team. In most cases, technical debt comes from a conscious decision and choice that is made by a fully competent team - do we have time to research and test a more efficient solution, or do we have to go with the less efficient but more easily implemented obvious programming choice?
Can we get rid of all technical debt?
Hypothetically, yes. Functionally, no.
If a great group of programmers had an unlimited amount of time to work on a project, they could probably eliminate all of the technical debt in that project. However, there is no company with this unlimited reserve of time. At some point, you are going to have to prioritize the functionality of the code over its artistic merit and even its efficiency.
The secret to technical debt is its management - determining the amount that can be eliminated and properly holding the rest to only an incremental effect on the final product.
Technical Debt and Agile Development
Everyone is looking for more agile development because it is the hottest new term on the market, not because it is necessarily the best process for a project. Agile development creates a more efficient product at the end of the coding, but the process itself is anything but agile.
Agile development is all about breaking down a project into component parts. Each of those parts is then optimized, of course, bringing the entire project to a higher level of efficiency. However, as you employ DSDM, FDD, or XP, you are actually extending your timeline to completion. Some projects simply do not have the time for this intermediate process, not to mention the money that it takes for the extra man hours. When the deadline and the total budget numbers are creeping up, sometimes a project must find its end in a less efficient final effort.
The Amount of Debt that is Acceptable
As with any debt/credit transaction, simply having debt is not a bad thing. Many people have become rich holding debt over commercial properties. The trick in the real estate industry is to maintain an income that pays for the debt, and this is the same mindset that can be applied to technical debt.
Have your programmers identify the areas of the project that weigh upon the rest of the project the least. You may have a relatively large amount of technical debt in a single module or function, but that module or function may not be used throughout the rest of the program very much. If the technical debt from one of these modules turns into a bonafide error, that error is at least manageable and non-critical.
The one critique of this "snowball programming" methodology is that this effort leads to more efficiency in the least important parts of the project. The team can fix this by identifying mission critical functions and modules as early in the process as possible, and looking for ways to employ these functions throughout the programming. This may require an extra layer of strategizing before actually sitting down to code, but it is well worth it at the start of the project, when you have the most time available to the team.
Estimate the percentage of bulk that can be allowed into mission critical functions and less important functions. Give this benchmark to your programmers from the beginning of the process, so they will have a quantifiable number to work towards. As time winds down, they will naturally include this productivity benchmark alongside their other key performance metrics instead of trying to include efficiency as a last minute addition to the process.
Stop worrying about the technical debt boogeyman in the shadows of your coding. If you set a precedent before the project begins, you will have a benchmark that will help you predict the efficiency of your programming efforts. You may not be able to get rid of technical debt completely, but you can control it while maintaining your schedule and budget!