The Never-ending Struggle on CodeQuality due to the growth of teams and codebases

reading-time:: 6m

As your embedded software codebase grows, the risk of introducing errors and bugs increases exponentially. Doubling the number of developers will roughly double the risk of mistakes. We need new “tricks” to strive for the ever-demanded high quality.
This is a constant battle. As embracing tools, processes, or disciplines helps only once.

Partly this is due to the mathematical relationship between the number of developers, the lines of code, and the likelihood of introducing errors. Some even estimate the risk of mistakes grows even faster; for simplicity, we use a factor of two and ignore the overhead in communication and added complexity, as we will lower that.

Also, the ever-increasing demand for non-primary features and new technology make (modern) embedded systems a lot more complex than some time ago. See (e.g.) Why Modern Embedded Software Systems nowadays embrace webserver-technology. Also, that “external” code increases the risk of errors.

Lowering the risks

There is no single, ultimate solution. No trick –as I like to call them– will remove all errors. We need to think in terms of “risks of mistakes” and constantly lower that risk. When we can decrease the risk of errors by 60% (or at least 50%) by introducing a new “trick” it will compensate for the hike due to doubling.

There is no single solution to this challenge. Instead, teams must constantly adapt and employ new techniques to ensure that quality is stable (improve) when teams or code size doubles.

Remember: code quality is related to functional quality, but not the same – high-quality code can contain (functional) bugs, and we can have unreadable, bug-free code. However, unreadable code will probably lead to bugs in the next release.
Therefore, a professional SW-Engineer should also lower the risk of mistakes in upcoming versions.

classical approaches

Over the years, various techniques have been developed to help manage software quality and reduce the risk of introducing errors. Some of these tricks have been around for a long, such as the Fagan inspection, code reviews, and testing. A drawback of many of them is that they do not scale when the code base grows.
For example: When I adopted Fagan, we planned for 100 hours for just a few hundred lines. That works for a small codebase, but a Fagan inspection of the Linux kernel or the FreeBSD code (15-40Mloc) would take forever!

The agile manifesto in 2001 led to some new tricks and enforced others. For example, Agile implies short cycles, which results in frequent execution of (the same) test(s). And so, automated tests became the norm. But as new features are added every sprint, the code is updated almost daily.
For many people, that is a motivation to write clean code –as it is called nowadays.

new techniques

Slowly, some of those tricks are replaced by other more time-efficient and/or better ones.
For example, static code-analysis tools replaced Fagan, and to some extent (code) reviews. Often the older tricks are still used, now focusing on other aspects – like readability, where we like the human touch.

Other technologies facilitate clean, readable & testable code also. For example, OO brought the option to replace local variables in huge functions – which didn’t allow us to split those functions– with instance variables in a local (sub) class. And split those functions into nice, short ones.

discipline

Present-day tricks often focus on discipline. Test & Behavior Driven Development are examples of discipline-based improvements.

Therefore, it is not easy to start with them. Using the tools is easy; adding process is a bit harder. But mastering the art, gaining speed, and getting clean code is hard –as old habits should be removed too.

A constant battle

Older and recent innovations, such as TDD and BDD, have proven effective. But they are not foolproof. As long as teams and code bases grow, each team must constantly explore new approaches to meet the demands for a (very) low bug rate and a high code quality.

Looking ahead, we can expect to see continued growth in contemporary practices. But we need fundamental new tricks also.

Beter tricks

Some old tricks constantly come back with new names. Is CI/CD really different from the old nightly build?
Even though they are conceptually the same, IMHO, they come with innovations too. Flexible pipelines, more computer power, and new technology enable this. A big screen with “the build” status is a great trick. Especially when it shows all revisions in priority order. It constantly reminds the team of what is important; a red tile on a dev branch can be ignored for a while. An orange one on an integration branch needs attention.

Any not-green tile on the top line should result in immediate action – drop everything. Even when that is just the line-count script! When we agreed that functions should be shorter as -let’s say– 7 lines without an explicit waiver, we should stick to it.

Here, the trick is not to change those seven lines. No, a perfect clean-coders team should wonder how that mistake did propagate to the top-priority pipeline. It should have been found long before; apparently, there is a breach in the WoW. That one should be fixed now!
Will we ever have such a highly disciplined team? Probably not, but it is nice to have ambition and some room to improve…

Upcoming tricks

Some technology innovations are already lurking to be incorporated. They are available but hardly used. And can help to lower the risk of errors.

For example, are you already using mutation testing? Discovered in 1971 already, and the first tool became available in the 80j. The idea is simple: test the coverage of your test set by making small changes to the code. Then, at least one test should fail; or you need more, better tests.
This really works but needs a lot of computer time. As we run all tests a zillion times. Slowly, this becomes feasible.

Another drawback: it doesn’t add anything when you don’t practice UnitTesting (or better: TDD) already.
This applies to many more tricks: there is a kind of natural order!

How to Maintain Quality?

This blog shows a few tricks to lower the risk of undetected mistakes. And argue that you constantly need more “tricks” as your team and/or codebase will grow.
Others, like Uncle Bob, already showed that this is happing in the general IT world. My experience is that it also applies to Modern Embedded Software Systems. I hope to write an article on that soon.

Ask for it!

Lastly, let me reveal the most simple “trick” to win this struggle – one that lasts. Just ask for it:

What do you bring to help to raise code quality constantly?

When I build a team, I need clever engineers. People that can solve problems. I regularly ask the above question when mature designers apply. I’m not that interested in the answer; the line of thought gives me more info – like, is (s)he driven by code quality.
Most programmers love a puzzle: tell them the problem, and they deal with it.

More important: When the teams grow, it has more people. More people to find new tricks. I consider that meta-trick better and more effective than finding the answer yourself.

Try it. It is fun!–albert

Comments

comments powered by Disqus