Category Archives: Quality

Regression – Lets stop it!

I hate it.

You change something and you can’t tell if your change broke something in the system.

If you’re lucky, you did not break anything. Or nobody noticed it.
Next to that, on the same scale of luck, the potential bugs are found in the manual testing phase.

But often there is no time to do all the regression testing by hand. It will take days and days, and the change you made looked so insignificant. It should go live. What could possibly go wrong?.

Then it happens. You’re live, your changes work, but the inevitable happens. Regression!

Of course, this has to be fixed. We can’t let our customers have a product with new features while the features of the previous version(s) are broken.

And so the patching process begins.

I call it patching, because often you are not done with one patch. While you were working hard to get the first patch live, other regression bugs are found and need to be patched asap as well! And so you end up with a few patches. You could be done with with a few patch releases. But it could easily extend ten-fold.

This process is very stressful for the customer and the development team. As the team is working to get these patches out soon, the customer is unhappy with his ‘broken system’. Even worse, once a few bugs are found, more testing is done on the live system to make sure everything still works, and more regression bugs poor in, adding up to the stress. To the development team it begins to look like…

From the customer’s point of view, it looks like the team working on the product is not in control. It is as if the team does not seem to know what they are doing. To them their product, which seemed rock solid at start, is degrading to a house of cards.

You can debate about high and low impact issues, and the matter of urgency to fix these issues. The perception of the customer is likely to be the same, regardless.

This is how I see it:

It is us developers who are responsible for letting regression happen.

Not testers.
Not project managers.
Not stakeholders.
Not the customer.

It is us and us alone.

We write the code, we change the code, we are in control of the code (at least we should be!).

Even if you happened to be depending on a third-party system, it is your job to keep an eye out on that system. Verify that it behaves as you would expect it to. Why? Your system depends on the behaviour of another system, trusting that this behaviour does not change is not enough. You have to be *sure*.

Its all about attitude
Do you always deal with regression bugs after each release?

Stop accepting it, it is not normal.

Rather, start thinking about how you can prevent this. Don’t look how other people could prevent this. Think of what you could do right now. There are many ways to reduce the amount of regression bugs. For instance: add tests before changing any code. Fixate the behavior with black box tests. When you refactor, keep running your tests so you know you did not break existing behaviour. Add new tests for new features you introduce. Create a test suite that you can trust. Make integration tests. Is it hard to write tests? Make it easier. Don’t back away from the code, it is your code and you should be in control.

Besides the code, improve your own skills. Start reading about how to deal with legacy code. Attend a legacy code retreat to hone your skills. Practice, practice, practice! Get better.

Reap what you sow in your daily work.

But isn’t the whole team responsible?
Ah, of course! But does that mean that you, as a developer can now do less? Would it be okay in a team to not test, because you have testers? (“its their job right?”).

In a team we all have our strengths and weaknesses.

We understand code, and we can change code. No other role in your team is responsible for understanding the code then you. Being in a team does not make you less responsible.

Again, it is all about attitude. Stand for your craft, deliver high quality work and make sure the system is in check. It should be you who controls the system.

Attitude, again
There are developers out there who really think they know everything of the system. And to be honest, I once had a time where I always knew what changes had impact and what I could do. And even though I was right about the impact on changes…

…I was at least wrong as many times as I was right.

But sometimes it is not just being over-confident. Sometimes it is being ignorant, or even arrogant.

Please, don’t be like this guy…

Just because it is hard, doesn’t mean you shouldn’t do it
Regression is a pain. It can be dealt with.

It is not easy.

You will not completely eliminate regression bugs. But with the correct mindset, tools and safety-net(s), you will greatly reduce the amount of regression bugs.

It is necessary. Just do it. For the love of our (your!) craft, do it, for everyone who depends on us:

The customer.
The stakeholders.
The project managers.
Yes, even the testers.

Experiencing a Code Kata – Become a better developer while having fun!

Recently I have been experimenting with a Code Kata, and in this post I’d like to share my experiences with it.

Code Kata?

Code Kata’s have been around for a while, but it really came into my attention while reading Chapter 6 from the book The Clean Coder by Robert C Martin. This chapter makes an anology that at your work you’re a performer like a musician and outside work you (should be, like a musician) practicing. (Of course, you will learn while at work, but that is not the point).

But what is a Kata?

I have played the piano for around 6 years (followed lessons) and played it much less after that (in fact, I don’t really play at all anymore). In those six years I had to practice etudes as well as more famous pieces. I did not understand why I had to practice Etudes, until much later.

So what has an Etude to do with Kata’s? Lets look at the description of an Etude (at wikipedia):

“an instrumental musical composition, most commonly of considerable difficulty, usually designed to provide practice material for perfecting a particular technical skill”

Without going into detail of a Kata itself, it is used to practice and perfect a set of techniques. Repetition and practice until you’re able to perfectly perform a Kata (or an Etude if you will) will help you further when you need to improvise or apply it in different forms. A lot of Etudes, techniques, are used in real pieces. A lot of Code Kata’s are actually dealing with real world problems.

Its not all about the solution!

So if I practice enough code kata’s I will become good at any problem I might face when writing software?

Not quite.

Code Kata’s are flexible; meaning you must set yourself a goal you want to achieve by doing a kata. When doing an etude you don’t have a lot of options. Your main goal is getting better with your fingers to play a series of notes or transitions. With a Code Kata you could practice your typing. Or perhaps practice all short-cuts of your IDE. Or heck, learn a new IDE while doing one. Perhaps you want to learn a new language. Perhaps you want to get better at TDD. Or you simply want to get to the solution and find the most efficient way to do so.

Atleast, I found doing a code kata much more fun than doing an etude ūüôā

Bowling Game Kata

The Bowling Game Kata is a Kata that challanges you to write a class (Game) that simulates a bowling ball game. You can roll balls and give the amount of pins knocked down. At the end you can call the score() method and you should get the correct score. It takes all rules into account, gutter, spares, strikes and the tenth frame where you can have 3 rolls instead of 2. The perfect game has 300 points. The worst game 0.

My initial thought: How hard can this be? I mean come on, I’ve dealt with harder things than a bowling game scoring system. Since I did not do any Code Kata before I set my goal to find the solution to this kata while doing TDD. I also did not want to look too much ahead in Uncle Bob’s (very nice) presentation (with solution). So I stopped when the game interface was given (roll() and score()) and I went ahead. Again, how hard could it be?

I have tried this Bowling Game Kata three times, and for each attempt I have written my experiences. All in all it was a very good experience and I recommend to try it out yourself. I believe if you want to get better, you need to practice. And only if you tried this multiple times, only then you know how it is.

So instead of talk the talk, let me walk the talk…

First attempt – Deception

Goal: Get it working, while doing TDD.

I set up a simple project and started with the GameTest. The first two tests where easy to do (0 pins, and all ones). But as soon as I got into Spares my first thought was to create a Frame class. Because a Frame represents a ‘turn’ where you can only roll 2 balls. The Frame class was born, along with its unit test. And I thought it felt good. I even added more ‘cheating’ detection. So you cannot roll 3 balls in one frame, or you cannot say you rolled 2 pins and then 9 (making 11 in one frame). I felt great and got unit tests working like nothing could stop me. Until the ‘perfect game’ test came around and my model just fell apart.

There was no way I could make it fit, without bending my entire model/solution.

So there I was, totally excited and thinking “just one more test and I’m done”.. and I got this.

Eventually I fixed it, I made my Frame class more flexible so I could set the maximum rolls and added flags. I then could create a TenthFrame class and set its flags so it would score differently. I also had created tests for the TenthFrame and even used an Abstract test class so I did not have duplicate code. Even so, I felt like this was wrong. I was bending my design just to work for one exception in the rules.

When I got all my unit tests passing, even the perfect score game, I just felt a great deception. My design sucked. Also, it took me almost 4 to 6 hours to get it working. Way too long for a code kata right?

Lessons learned
– TDD cycles where not strict enough; so…
– TDD cycles where slow, I had to switch mouse/keyboard to rerun the unit test(s)
– I made design decisions too early and later got ‘stuck’ and had to bend the design to make it work completely
– Finding the solution the first time takes time
– I made much more stuff than I had to (over-engineering?)

– Time taken: roughly 6 hours.
– Amount of unit tests: 33

Second attempt – No need to bend the universe

Goal: Get it working, while doing TDD. Take lessons learned from first attempt. Aka: Tighter TDD cycles, get faster at TDD cycles, etc.

I started this kata late in the evening. I spent a fraction of the time compared to the first attempt: ~ 45 minutes(!). The later half hour mainly refactoring and keeping green bars. The actual solution was there within an hour. I did not have to bend my design, I could keep everything within the Game class!

Something little, but practical I learned about the IDE I used (Eclipse) is to short-key the ‘rerun last test’, so my TDD cycles where shorter.

I also noticed I understood the scoring of the bowling game much better. I don’t play this game very much, and when I do, the computer does all the scoring for me. So I guess the first attempt at the Kata took also longer because I had to understand the scoring rules.

Design wise I found that I still use “frames”, but not as a separate class. I do not have any cheating detection (so I can roll 12 pins and it won’t complain), but the scoring will utterly fail in that respect. Building these checks in would not be a problem though, because the spare/strike detection is now so easy.

Lessons learned
– Faster TDD cycle by using shortcut for re-running tests
– TDD cycles where stricter, but could be even more tightened
– Commenting out tests when more than one breaks really helps you get focussed on getting one thing to work (so leave one test breaking), instead of fixing all tests at once.
– No cheating detection, no over-engineering
– Of the time consumed, I spent more time refactoring relatively to the first attempt, than thinking / finding the solution.
– I could refactor a lot of code, and make it much more cleaner. And safely due the tests.

– Time taken: roughly 45 minutes (!!)
– Amount of unit tests: 7

Third attempt – The only way to go fast is to go well

Goal: Tighten the TDD cycle. Get it done in 30 minutes or less.
This time I started fresh on a Sunday morning. Since I know the solution and I knew the design choices I made earlier (I know what works, and what does not work) things went very quick. I finished it within 30 minutes. I had the same amount of unit tests and the greatest thing was that the last test (perfect game) worked immediately. I did not had to change anything to make it work.

Once I had the last test working, I checked the code a bit and called it done. This time I also wanted to check my solution against the original presentation Uncle Bob made, to see if I missed anything or not. I figured that some of my tests where faulty:

– If you roll only ones, you can only roll 20 times and not 21 times. (i had a weird if statement to fix this up, but now it seemed that this was flawed).
– The perfect game was in my case 21 strikes, while you can only roll 12 times in that case. When I changed my test, it still worked.

It struck me that I was approaching this technically (21 rolls is maximum), and not from a functional point of view (ie 20 is max when only ones).

I also found that my TDD cycles where still to wide. I could do run the cycles close to each line of code, but I tend to write 2 or 3 lines before re-running my tests. Especially the first tests where suffering from this, later tests went better.

Lessons learned
– TDD cycles can be shortened
– Functional point of view caught errors in my tests
– Shortest kata ever (under 30 minutes)
– Latest test worked, design was good. Design was even better after fixing the test for ‘only ones’, so I could remove weird if statements.

– Time taken: < 30 minutes
– Amount of unit tests: 5


So, after 3 attempts, do these Code Kata’s work for me? It surely learned me a few lessons. In short:
– The first time you do a Kata, it is slow. And you’re focussed on the solution. This could throw you off, but you have to persist…
– Later attempts are going much faster, and your focus shifts to other techniques. Mine was mainly speeding up my TDD cycle.
– I have learned a few things to speed up my TDD cycle, which I can apply in real world stuff as well. Which is good!
– Over-engineering will bite you, one way or another. Your design will be toast.
– I have learned that even in my last example I was approaching the solution too technical. As a Developer I still did not approach it entirely functionally. This meant that even though I thought I was done, I wasn’t. I do plan to use Acceptance tests (using JBehave) for this. That will be covered in a next blog.
– Above all, doing a Code Kata is fun!

I would advice to other developers to do Code Kata’s and get better at what they are doing. There are tons of areas where you can improve. In short, yes I do believe in them and I think you should give them a go, if you haven’t already!

Find more about Code Kata’s:

If you really want light-weight warm up exercises, you might want to go to:

My experience with (un)certainty about estimates in relation to technical debt

Not too long ago, Martin Fowler pointed out a nice blog post by Jay Fields. Jay Fields refers to a nice talk he had about accidental complexity and essential complexity and how this has impact on your estimates. He found that not all developers consider the accidental complexity and therefor have lower estimates.

I found this a very interesting thought. It got me thinking how I estimate and how far I’m off. I found that, especially with larger solutions, I’m most of the time under estimating. Even with more complex things, and adding some ‘unforseen complexity percentage’. I’m still under estimating most of the time. However, I’ve also had better experience on other projects. Especially the latest project I’m working on the estimates of fixes and rework are not as much off. How is this possible?

I find myself labeling this phenomenom as “lack of overview”. If you read the definition of Accidental Complexity, it is described as “…accidental complexity is caused by the approach chosen to solve the problem.”. I believe this ‘approach chosen to solve the problem’ is the design of the code. This is different comparing to Essential Complexity, which I belief is much like Cyclomatic Complexity.

I made mistakes with my estimates, even when I knew the code well. Often it was due a dependency that ‘got in the way’, or worse, the lack of dependencies. All functionality was in one class! Adding similar behaviour required me to duplicate code. I consider this a bad practice, so I had to extract code from the other class. I was untangling the code. Whenever I had to untangle that code (ie, seperate concerns), I had a hard time doing so: Because untangling a tangled (tightly coupled) piece of code forced me to untangle other pieces of code as well. I had to stop somewhere. Like someone once said to me: The devil is in the details. (this is one of the reasons I encourage my co-developers to talk to interfaces, and not implementations).

But why are estimates off anyway? Is it because (lack) of experience with the code? Even with code I’ve worked with for years I still made bad estimates. And I could not find a way to get them better. The newer project went way better for me to estimate. I already knew why it was going better:

My mental model of the code matched better to the actual code. Was it because I worked on it lately and knew how it worked exactly in detail? No, not at all! The Technical Debt is much lower on this project. One of the principles that played a huge rule was the Single Responsibility Principle (PDF). When I had to make a change, it was often in one place. When I had to add code, I could easily move code out of the class and seperate responsibilities. The code was less tangled, tightly coupled.

This phenomenom of untangling code, seperating concerns and having a hard time maintaining code is clearly a sign of repaying serious interests of technical debt. And I clearly see that as a result of a ‘choosen approach to solve the problem’.

Therefor I believe the technical debt is linked to essential and accidential complexity and even more (what about readability?). Accidential Complexity is something that is very hard to grasp. I think this ‘uncertainty’ needs to be clarified and be added to each initial estimate in order to get a ‘more realistic’ estimate.

I would recommend to estimate code while looking at the code itself, rather than use just your mental model of the code.
Finally, repaying the interest of the technical debt should be prioritized in order to be able to maintain the system and to prevent to get a mad customer getting ever less features using ever increasing time to make them.

An example of refactoring

As I have promised in my previous post, I would post an example of small refactorings in order to greatly improve the readability and understandability of code.

I own a little project called Dune II – The Maker, and I started writing it a little over 10 years ago. In those years I have learned a lot. I did not have much time in those days to apply my new knowledge to the project. You could say the software was rotting. In order to make it better I need to refactor a lot and I encounter the best examples to improve code without pointing fingers :). In any case I have experienced you have to make mistakes in order to get better. I hope you will learn from the mistakes I made.

So here is a little example I have just checked in the dune2themaker repository, I’ll give you the before (revision 411) and after (revision 412). Of course, I have taken smaller steps to get to the end result. First the original piece of code:

Revision 411 (before)
[sourcecode language=”cpp”]
void cGame::think_winlose() {
bool bSucces = false;
bool bFailed = true;

// determine if player is still alive
for (int i = 0; i < MAX_STRUCTURES; i++)
if (structure[i])
if (structure[i]->getOwner() == 0) {
bFailed = false; // no, we are not failing just yet

// determine if any unit is found
if (bFailed) {
// check if any unit is ours, if not, we have a problem (airborn does not count)
for (int i = 0; i < MAX_UNITS; i++)
if (unit[i].isValid())
if (unit[i].iPlayer == 0) {
bFailed = false;

// win by money quota
if (iWinQuota > 0) {
if (player[0].credits >= iWinQuota) {
// won!
bSucces = true;
} else {
// determine if any player (except sandworm) is dead
bool bAllDead = true;
for (int i = 0; i < MAX_STRUCTURES; i++)
if (structure[i])
if (structure[i]->getOwner() > 0 && structure[i]->getOwner()
!= AI_WORM) {
bAllDead = false;

if (bAllDead) {
// check units now
for (int i = 0; i < MAX_UNITS; i++)
if (unit[i].isValid())
if (unit[i].iPlayer > 0 && unit[i].iPlayer != AI_WORM)
if (units[unit[i].iType].airborn == false) {
bAllDead = false;


if (bAllDead)
bSucces = true;


// On succes…
if (bSucces) {
// <snip>


if (bFailed) {
// <snip>


The intention of the think_winlose() function is to determine if the player has won or lost, and if so it transitions the game state. These transitions have been snipped.

So when does a player win or lose? It depends if there is a ‘win quota’, or not. The win quota is a number, whenever it is above zero it means the player has to collect at least that many of credits (spice) in order to win. If the win quota is not set, the default win rule : destroy everything of the enemy, will be used. (do you notice I need this much text for just a simple rule? Which I could have prevented If I had code that said this in the first place? At the bottom of this post you can see what I mean :))

Lets take a look at the code and point out what could be done better:

  • There are two booleans bSuccess and bFailed. Which is confusing and ambigious. What is succesfull? What did fail? Why aren’t they one boolean?
  • There are comments all over the place, meaning we could refactor these pieces to code so comments are not needed. (Comments are seen as clutter and should be removed)
  • The code formatting could be done better. If statements should start with { and end with }, even with one line.

And there are more things you will probably find yourself. What I’ll do is point out a few things that could be improved. If you just want to see the final result, just take a look below.

Lets start with the booleans bSuccess and bFailed. Why are there two booleans and whey are they called so vaguely? A little bit of searching in the code and we find out that bSuccess actually means “Mission is accomplished” (player has won), and bFailed means the player has no units and structures (which implicates the player has lost the game). They are not the same boolean, because a player could be alive and not have yet won the game of course. Now we know they are not actually the same boolean, but their naming was vague. A simple “rename variable” made things easier to understand!

[sourcecode language=”cpp”]
void cGame::think_winlose() {
bool bMissionAccomplished = false;
bool isPlayerAlive= true;
(when posting this I realize the two booleans are named differently, consistency is also important to improve readability, so either both should start with “is” or both with a “b”, I prefer the first though)

Right after the booleans a few for loops are used just to find out if there is anything alive for the player. A little bit below we see such for loops again, but for the AI. This is duplicate code and should be removed. Extracting them into a method and make them return a boolean value is easy to do:

[sourcecode language=”cpp”]
bool cGame::playerHasAnyStructures(int iPlayerId) {
for (int i = 0; i < MAX_STRUCTURES; i++) {
if (structure[i]) {
if (structure[i]->getOwner() == iPlayerId) {
return true;
return false;

(Again, while posting this I realize this could be even improved a bit more, the iPlayerId should be called ownerId (or the getOwner should be a getPlayerId), so it is obvious we match two of the same kind. Now it could confuse us: is an owner the same as the playerId? Since I know it is, why isn’t it called that way?… :))

Since we extract these for loops we can now set the isPlayerAlive boolean immidiately instead of setting a variable within the loop as it was done in the original example above. Reducing 24 lines into one!:

[sourcecode language=”cpp”]
bool isPlayerAlive = playerHasAnyStructures(HUMAN) || playerHasAnyGroundUnits(HUMAN);

The final result of revision 412 is shown below. It will clearly show the major improvement regarding readability and understandability. Any other developer who comes to this code can see what it does and it is almost a no-brainer.

Result revision 412
[sourcecode language=”cpp”]
void cGame::think_winlose() {
bool bMissionAccomplished = false;
bool isPlayerAlive = playerHasAnyStructures(HUMAN) || playerHasAnyGroundUnits(HUMAN);

if (isWinQuotaSet()) {
bMissionAccomplished = playerHasMetQuota(HUMAN);
} else {
bool isAnyAIPlayerAlive = false;
for (int i = (HUMAN + 1); i < AI_WORM; i++ ) {
if (playerHasAnyStructures(i) || playerHasAnyGroundUnits(i)) {
isAnyAIPlayerAlive = true;

bMissionAccomplished = !isAnyAIPlayerAlive;

if (bMissionAccomplished) {
// <snip>

} else if (!isPlayerAlive) {
// <snip>


The tremendous power of tiny refactorings

More and more I am being intrigued by the power of a small code refactorings. The positive impact it has on the readability, the maintainability and understandability of your code is great. It keeps code clean(er) and since the changes you make are really small (I’ll demonstrate how small),¬†the chance they will break things is small. Of course, with unit tests (you are writing them right?) making sure you did not break anything: a¬†small refactoring is a low-risk high-benefit practice.

In my experience, small refactorings are undervalued. In fact, I undervalued them much myself since not too long ago. They are disregarded as refactorings that don’t help at all, because it is obvious what the code does. However, the flaw in this rationale, as I see it, is that the intended audience is not only¬†you but also the other developer you work with. Also, you know what code does right now. But would you understand it as quickly if you did not look at it for a week and came back? Would another developer understand the code right away?

When working on code, you’re constantly trying to ‘translate’ the code in your mind in order to know what it is doing. Doing this it leads you to where the bugs are or the areas where you need to make changes, etcetera. This process of ‘translating’ code in your mind comes at a price. Literally the energy you need to burn in your brain to grasp the meaning of a piece of code:¬†brainpower; The easier we understand code, the less brainpower we need. The less energy we burn by understanding what is going on, the more energy we have left to create new things, or fix that bug.

I’ve created a little example. The code below represents an implementation of a mail service. The mail service allows you to send an email using a method that uses 4 parameters: to, from, the subject and the message. When all parameters are filled, the email needs to be sent. That is the only requirement for now. Of course, later we might want to validate if the given email adress of from and to are valid. But for the sake of the argument, lets keep it simple. The following code is ‘mind-boggling’, compared to its simple intention:

[sourcecode language=”java”]
public class MailServiceImpl implements MailService {

public void sendMail(String from, String to, String subject, String message) {
if (from != null && !"".equals(from) &&
to != null && !"".equals(to) &&
subject != null && !"".equals(subject) &&
message != null && !"".equals(subject)) {
// send the email


Basically what this says it that any field may not be null or an empty string. It took 4 lines for just to show. Even though you recognize the pattern of a ‘null or empty check’, it costs you time and energy to make that translation happen. So here is a first suggestion to make it¬†read easier:

[sourcecode language=”java”]
public class MailServiceImpl implements MailService {

public void sendMail(String from, String to, String subject, String message) {
if (parametersAreNotNullOrEmpty(from, to, subject, message)) {
// send the email

private boolean parametersAreNotNullOrEmpty(String from, String to, String subject,
String message) {
return from != null && !"".equals(from) &&
to != null && !"".equals(to) &&
subject != null && !"".equals(subject) &&
message != null && !"".equals(subject);


When another developer is reading the sendMail method, he will now know that when the parametersAreNotNullOrEmpty the mail will be sent. It does not need any translation, the method name just says what it does! Simple! By doing this, you greatly reduce the needed brainpower to understand what is going on. The refactoring method used is called Extract method.

Reading code is sometimes easy for your brain to handle. Sometimes your brain seems to explode because of the complex statements and context you need to be aware of. It is strongly tied with the Cyclomatic Complexity, the Coupling between Objects (CBO) and the lack of Cohesion in your code. If you are using any tools to measure your code, like Sonar for example, look for these metrics to find code that needs attention. But it is always better to refactor while you have made the translation in your brain, if you see things can be written simpler to reduce the needed brainpower, by all means do so. Not giving software the appropriate attention might let your code rot. Small refactorings help you prevent that.

I hope you have seen a bit of the power of small refactorings. I will get back to them in my future posts as I will post more concrete examples and how I would/have dealt with them. To me, small refactorings need to be part of your system and are introduced when you do TDD. All too often when the code works, it is not looked at again. Making these small refactorings can make a big difference and take relatively no time.

A Software Quality Model (Part II) – Translating customer language into metrics, scoring quality

In my previous post I have explained the context of my thesis, and the various software quality models that are evaluated.

For my thesis I have extended the Software Quality Model of Willmer. Although it is not an exact implementation of the model, it is inspired by it. Also, the influence of the customer is processed into this model. The goal of this model is to translate customer desires into metrics, in order to calculate the total quality.

When talking about software quality, it is either very concrete (I want a red car!) or very abstract (it has to be reliable!).¬†Customers tend to tell their ‘experiences of quality’ in sentences. This is the first step of the model. Try to get a few (eight at most) of these sentences. They must be distinctive. (Don’t have four sentences preach about security…)

Translating these sentences into concrete, measurable ‘things’ for developers, is another story. But before doing that , ask your customer what the relative importance is of all these sentences you just have written down. Just imagine that there is this situation where you have to go live. But , there is an issue that needs to be tackled, all blocking, one for each aspect (sentence). If you could pick one, which would you pick? Would you tackle the first sentence, the second? etc?

Of course, your customer will would tell ¬†in his own domain language what the most important thing is. Try to map that (and confirm) with these sentences. Score them and try to get an ‘order of importance’. After you have done that, you have your first important goal reached: You know the relative importance of each quality sentence (aspect from now on).

So, what now? The next step is to map these aspects/sentences to Software Quality Attributes (also known as non-functional requirements). You either need a Software Engineer to do that, or even better, try to do it with your team. Before mapping these, try to make a selection of Quality Attributes first that are most relevant to you. Ie, try to use tree of Boehm or ISO9160 as reference. Within my theses I have used 9 Quality Attributes, some Quality Attributes are ‘sub-quality attributes’ of others. Example of Quality Attributes: Understandability, Reliability, Security, Availability, Complexity, etc.

The result of this mapping is that you get for each aspect several quality attributes. Not all quality attributes are applicable to an aspect. Try to figure out how much quality attributes are applicable to an aspect. Do this by asking teammembers and for each member that selects a quality attribute, score it. This way you can calculate relatively how your team thinks the mapping should be. This is important, because the eventual result of your measurement (see below) should be the product of the customer and the team that work on that product.

So, you have a few aspects, and each aspect has a few quality attributes. All that is left is to map metrics to quality attributes. Mapping this is fairly easy, there are quite a bunch of metrics out there. Each metric tries to measure (a piece of) a quality attribute. Some are easy, like complexity (quality attribute) can be measured by (although it is not limited to) the Cyclomatic Complexity metric by McCabe.

So basically you end up with this:

Aspect(n..8) –> Quality Attribute(n) –> Metrics(n)


  • the total quality of the system, is the combination of all aspects (all aspects relativity make 100%)
  • you should keep eight aspects (believe me, more will only make it harder to distinguish and make decisions)
  • you should attach quality attributes to each aspect, and determine their relative applicability to this aspect
  • you should attach metrics to quality attributes

So in the end, how do you score quality? Is it possible with this model? Yes, certainly it is.

Once you have found metrics, and attached them to quality attributes. You should formulate ‘scoring rules’. This means, you need to write down how you will interpret results of a metric and translate that onto a scale of 1 to 10. A scoring rule could be:

“Lines of Code (LOC) may be 1000 for a file max. Every 100 lines more substracts one point. (Giving 10 points with 1000 or lower and with 2000 lines a 1)”

This means, a LOC of 1000 will score a 10. A LOC of 1500 scores a 5, a LOC of 2000 or higher scores a 1.

Do this for all metrics, and eventually you will be able to calculate the total quality of the system.

In order to make this more concrete, here an example of such a calculation:

Total Quality Score = sum of scores of each aspect

Aspect score = (Sum of all relative scores of all applicable attributes) * relative importance

Attribute score = (Sum of all relative scores of all applicable metrics)

Example (For the sake of this example, attributes are scored already)

Aspect #1

Is for 30% important (relative to Aspect #2)


A -> for 40% applicable

B -> for 60% applicable

Aspect #2

Is for 70% important (relative to Aspect #1)


C -> for 70% applicable

B -> for 30% applicable


Attribute A scores 7 (out of 10)

Attribute B scores 5 (out of 10)

Attribute C scores 8 (out of 10)

Total quality calculation:

Aspect #1

A = 7 * 40% = 2,8

B = 5 * 60% = 3,0

Absolute score is 2,8 + 3,0 = 5,8

Aspect #2

C = 8 * 70% = 5,6

B = 5 * 30% = 1,5

Absolute score is 5,6 + 1,5 = 7,1

Total quality is:

Aspect #1 –> 5,8 * 30% (importance) = 1,74

Aspect #2 –> 7,1 * 70% (importance) = 4,97

Total quality score is 1,74 + 6,71 ==> 6,71

6,71 on a score of 1 to 10 (1 being bad, 10 being best).

A Software Quality Model (Part I)

This is the first post, of more to come, about my thesis and the software quality model I have developed.

First I will explain in what context my thesis is placed. This post will focus on decisions made to actually form a ‘new’ model (actually, derived from…).

For my thesis I have investigated how to improve quality of a software product. The first step was to choose one of the five perspectives on quality, given by Garvin (see here for more information). The product based perspective is chosen for scoping reasons. With that perspective in mind, three software quality models where evaluated:

The first, and one of the most known models, is the model of Boehm (literally translated ‘the tree of Boehm’):

The second model was from Willmer and the third from Bemelmans. Basically Willmer takes Boehms model and took it a bit further in order to translate customer desires into metrics. Bemelmans takes a whole different approach: He takes three main characteristics: “requirements for usage”, “requirements for maintenance” and “requirements for product information”. Which each have their own sub-characteristcs. (though it does not look like a tree like Boehm, as it only has one dimension).

One model actually caught the attention because it tends to ‘translate’ customer language into a developer language. The basic idea is that a software quality characteristic is mapped with a metric. Unfortunately the book I had did not tell how to translate this. Also, there was a need to translate customer language into software quality characteristics. This was something that was not being explained either.

So how does the translation look like? Imagine a matrix, where at the left several software quality characteristics are given (for every row). And at the top, every column is a metric. With some means you could assign metrics to characteristics and get a basic idea what is important.

Taking the ‘translation matrix’ of Willmer as a starting point, I have developed a model that actually does three things:

1st. It translates customer language into software quality characteristics. Not only that, but it also gives a % of relevance. (more on that later).
2nd. The actual translation from characteristics to metrics. This looks a lot like how Willmer does this, but it also has a % per metric per characteristic.
3rd. The  interpretation of metrics. This is where things get really concrete. Here the actual scoring is defined. How will the metric LOC be interpreted?

With these three, it is possible to actually ‘calculate’ the total quality of a software product, with the ‘product based perspective’ of Garvin in mind! (So, it does not tell you how to improve your processes. Nor does it tell you how customers experience the website).

So how do you use this model? Using the “Plan/Do/Check/Act” (Circle of Deming) as basis, the suggested plan is:

– First ‘define’ quality. ¬†(Plan)
– Meassure and analyse (Do)
– Evaluate (Check)
– Change definition / software (Act)

In my next posts I will explain briefly how the model looks like.

The need for quality of Web Applications

I’ve been working on my thesis (about meassuring Software Quality) and its reaching its final state. I will blog some posts about my thoughts and findings about Software Quality from here. This post was a bit old, I dusted it off and actualised it a bit. More to come!

Here it comes:

Today, it is unthinkable to not use an internet related service in your daily life. Think about staying in touch with family and friends using email (gmail, yahoo, hotmail, etc), social networking (twitter, facebook, hyves, etc) or more.

If you want to book a trip, you arrange this using the internet. When you want to buy a new laptop, you find the specs and reviews on the internet. You use a comparison site to find the cheapest (web)store. And if you want to buy it, you use the internet.

It is fair to say that the internet has become more and more important to our lives. Even more than we imagined 5 years ago. How would we deal when our mail provider is down? How would we react when facebook is down? What about twitter? How do we do our bank business when we cannot reach the online-banking site?

I think you got the point.

Companies developing and maintaining (ecommerce) websites or services are constantly pressured to deliver high quality software in the shortest possible amount of time. Having these factors, time and quality; one cannot exclude the other. They do influence each other though. When you want to deliver fast, you ought to make sacrifices in terms of designing and testing the product. This will eventually lead to a lower quality. The opposite is true as well, if you spend more time in the design and testing, you will likely deliver higher quality. Also, unlike with traditional software, it is not always the first to deliver that is the best.

In the land of the internet, it is not only the case to get users to your website, but also to keep them revisiting. If you deliver a fancy new feature, you might get the attention. But if you did not put enough effort to make it high quality, you might lose them to your competitor who delivers a similiar feature a few days later, but twice as good. Remember, your visitors can easily switch if you cannot deliver what they expect.

So what does a visitor expect from a website? Looking around on the net you can find some definitions of quality of good websites. That is the biggest question of all. How can you deliver fast, and yet have some sort of quality to rely on?

You could use a Software Quality Model for that, like the ISO-9126. I would say, you’d use a model like the ISO-9126 and create/use a Web Application Quality Standard.

So what is this standard? I haven’t found a definite model, although I have found a good resource here.
In my opinion, the following Software Quality Attributes would belong in such a model:
performance. Your visitors will leave if loading a page takes longer than 2 seconds. Directly influencing possible sales!
security. Atleast be aware of the top ten of OWASP. Better yet, make sure you are not vulnerable to these exploits.
maintainability. Developers should be able to easily extend code, or fix bugs without causing too much regression.
reliability. Should your website simply shut down once the load is too high? How much ‘page faults’ may appear? One of thousand requests? One of hundred?.

Each of these attributes can be split into several measurable units. For example, performance wise you could measure the amount of time it takes to get a page loaded in the browser. Maintainability has some known measures (like McCabe’s Cyclomatic Complexity).

Note: Recently I figured that the ISO-9126 is actually not a model, but a ‘standard’. It really does not tell you *how* to do things, but it does give guidelines. The Software Quality model I’ve made does tell¬† you how to do things.

Choosing a Software Quality Model

For my thesis (Software Quality) I am currently figuring out how to choose a proper Software Quality Model in order to lay down the foundation of a Software Quality Definition for the project I am working on.

For now I have choosen this approach, and I’d like to hear from others how they (would) have tackled this problem.

It is good to know that I have narrowed down what “Software Quality” is. I’ll be focussing on the product (Garvin, Gosby) quality and the targetted audience are Software Engineers.

At first I will determine what kind of product I am working on. In this case it is a website, selling things. Some characteristics come to mind, like “24/7” or “should display always the correct (price) information”.

With these characteristics, I’ll look up Software Quality Attributes that would fit in.

This gives me a table, with in the first column all the characteristics, and next to them a few Software Quality Attributes that would fall into that specific characteristic description.

Now that I have a list of Software Quality Attributes, I can look up what Quality Model has the most matching Software Quality Attributes in them as well. Models that focus on different attributes would not interest me anymore, because it would not apply to the kind of product I focus on.

I found a nice model comparison, with Software Quality Attributes, from NASA. For your convenience I show it here:

With this, it would be possible to choose a Software Quality Model and work from there.

I am half-way with this method, and so far I think this is the way to go. If you think this is a bit too easy, or I should really take other considerations. Please let me know.

Note: thanks to Wouter for his suggestion who got me into this direction.