When you raise a good bug, goodness follows:

• Someone can look at the bug and figure out what's wrong without having to waste time.
• Anyone with the right hw environment can reproduce it and therefore fix it.
• The problem will be gone soon, and you'll be able to use that part of the system from then onwards.

When I look at a bug and see:

Ubuntu doesn't install.

or

Unity doesn't work.

… I want to cry. Because these two descriptions are the worst bugs ever, they don't say what is not working, nor how to reproduce it, no indication of what kind of system Ubuntu was running on, or version of the OS,  they don't even say what the user was doing / trying to achieve when everything went south, or whether there are conditions that need to be met  for the bug to be reproduced. They are essentially useless bugs. 

When I do testing and I find a problem I know it is going to take me a while to raise it, because there are some things that need to be done to be thorough:

• Is the problem consistent and reproducible, i.e. does it happen every time I do the same thing?
• If it only happens sometimes, can I determine the factors that make it more likely to occur?
• Does it also happen in this other hardware, on a VM, on my laptop, on the other laptop? (This tells me whether developers will be able to reproduce it to fix it)
• Which logs do I need to add to demonstrate my system was in a problematic state?

And once I have put the bug in the database, I ask myself:

• Is this the only problem or is this area problematic and has more than one issue? And continue to test cases around the one found, to make sure the problem was isolated or to start the bug raising process again. This process is called exploratory testing, when I know which area of the system I need to explore and I  poke around the functionality to make sure it is solid and behaving as expected. Exploratory testing needs to be done in a scientific way to be fully useful, documenting the things that I tried and making sure there's no bugs left for other people to find in the vicinity. 

Making sure the area  is sound helps, because there is going to be someone looking at one bug, so they may as well fix other problems if there are more problems associated with the initial issue. 

I have tons of email to read and some projects I'd like to see in production soon:

• Bootspeed testing
• Upgrade testing to be added to the generic reporting
• Improve the number of tests run during smoke testing

I am not sure if you've seen the dashboard go live last year, but you are welcome to have a look in: http://reports.qa.ubuntu.com. 

You probably noticed, one of my new year's resolutions is to post more regularly. Happy 2013 everyone!

1. When you have no tests, you know you don't know anything about the quality of your product. Knowing that you don't know something is good.
2. Having bad tests passing gives the impression things are good. How good? Well, not very good but somewhat good. 30% good? 45% good? How good is good? No idea. 

When a test case fails and there is a bug to explain that failure, then you are happy. Whenever the failure is fixed, your test will pass again. But not all test cases are like that.

False negatives in your results are bad. When test cases fail and there is no real bug to be found, you are just wasting your time hunting for the bug that doesn't exist. 

But when a test never fails, even when a problem is there, it's worse. Because you may be experiencing false positives, but you will be convinced that everything is green because you are testing that condition, and the test passes.

Not any test is a good test, some tests are unreliable and just distract from the real problems. Why are people so attached to them? It's just a test case and it is wrong, just remove it! So, if you've spent more than a few hours hunting for a problem that wasn't really there, you may as well spend another one fixing your test for that not to happen in the future or getting rid of it. There is a downside to this advice, you may be having a bad day and the bug/problem is really there, take a couple of weeks to run and verify your findings.

If it takes you hours to figure out what the problem is, you'd have been better off investing a bit more time in your test case upfront than spending all that time every time the test fails.

In my opinion, a good test would be better than no test, but not any test is good enough to be considered better than no test.

A bad test case is a very expensive piece of software and a sink of engineering time, avoid it.

In the Ubuntu project things were a bit different 6 months ago. We had several projects solving the same kind of issues in slightly different ways, a fragmented testing landscape, there was a lot of duplicated code and testing projects that slowly died due to no-one having the time to maintain them when they became big enough. This is one of the problems with automated testing. It is easy to get excited and code a lot of stuff just for the fun of it, but it is not that easy to create automated tests in a way that they are cheap to maintain as well as useful. 

Ok, so let's say you are convinced that testing is something you need for your project/package and you want to create some tests for it. You do your python test scripts or C programs that report if they pass or fail. Now what? Well, you want to run them in the latest version of Ubuntu, you want to run them on i386, amd64, with this or that kernel, you want to run them in old versions to make sure you are backwards compatible, you want to run them in the current version so that you are compatible with the most modern stuff and quickly you get depressed because you realise that it is not easy to move from having some tests written in python to actually having a good set of results you can work with, provision all those systems and run your tests becomes a huge burden. 

UTAH is meant to help people plug their lightweight scripts/test cases and help with the provisioning (this is what we mean by end-to-end: provision, run tests, get results back). If you want to provision a VM or a spare machine at home (HW provisioning not fully implemented yet, working on it), or if you just want to run a bunch of scripts for a particular purpose on your own machine, any of this scenarios should be easily achieved by using the Ubuntu Test Automation Harness. We have made sure that VMs are provisioned and tests can run on them. We are trying to give the community a way to define what the test cases do and a way to run them easily on different configurations. 

The tool has been developed and is being Alpha tested. We hope to be able to start substituting current smoke testing jobs in Jenkins for UTAH smoke testing jobs soon. I will tell you all about the migration so that you can decide whether this is the right tool for you! If you want to have a look at the documentation, here it is, but bear in mind this is being reworked whilst we do the migration, to have it completely up to date by the end of it. 

We even have a mailing list, in case you want to join this adventure!

There have been initiatives over the years in terms of automating test cases, some people have come up with code that did exercise the system somewhat, and we have settled in using Jenkins as a CI tool. We don't use it extensively, i.e. we don't build everything with it, but several teams have started to use it to run some automatic tests before submitting and let it decide whether the code is ready or not. This is an excellent starting point, but not enough, in my opinion. We could set all that up with irrelevant test cases, and we would have a very sofisticated way of tricking ourselves into believing things are good, when they in fact are not.

I am a pure QA person, i.e. I don't develop code unless it is absolutely necessary to be able to do my job. In the past, I have done a lot of metrics with Excel/LibreOffice Calc, and I am not ashamed of it, it was necessary. As part of my job, I try to chose the right tools for the right tasks, and Jenkins, as good as it is as a scheduler/building tool, didn't really cut it for me in terms of reporting test results.

Instead of trying to get other people to do what we needed without fully understanding the problem, I decided to put some hours into it and come up with a reporting tool that actually reported the right thing. It's not the official dashboard for Ubuntu, but it is a project that helps me understand what I need and communicate it to others, in fact, the smoke testing part is just a different view for this other page. In my opinion the new dashboard is much clearer and actually shows aggregated results in a more meaningful way, we've even managed to populate bug information. The dashboard feeds from the Jenkins api and adds the results to a local database where manipulating them for our own interest becomes trivial.

I will be working in figuring out the best way to report bootspeed testing and Kernel SRU / LTS backport testing soon. In the meantime, I am fighting yet another battle, the battle of the test results in jenkins. Because the fact that things look green or red in Jenkins means nothing to me (by things I mean jobs, it is just that I dread to talk in "jobs" terms, because jobs do not mean anything to a QA engineer, they mean builds to developers, which I understand can be useful in that context). Yellow in Jenkins? Whatever. I hope I can convey this idea to as many people as possible in the near future. What we really need is to have countable test cases, that we can report on and triage. I don't really care if Jenkins thinks the job is yellow, I want to know which of my N test cases failed and why, so that someone can fix them!

This is an interesting journey of starting to do testing regularly and care about the results so that they get a lot of attention and bugs get fixed soon, but also looking into the future and where we want our testing to be in 2 or 3 years. Relying on manual testing and bits and pieces of automated testing is where we are. Full automated testing and important bits and pieces done manually when automation is not possible is where we want to be. The Ubuntu QA Team is working hard to get this right and we welcome as much help as we can get.

We have a Jenkins instance in place, where we publish the results of the testing we do on daily basis. We also have instructions on how to read those results and make sense of them. We have a list of bugs that are currently open, that we found with the automated testing and the list of bugs that were found by our testing and have been closed. This is where we are at the moment, improving every day our operations and reporting so that we can give a better service and we can get more out of our testing.

Any ideas, suggestions or help is more than welcome, just get in touch with us at ubuntu-qa@ubuntu.com.

It has been very productive, though. We decided plenty of things that needed to be decided and started the Precise Pangolin release with a lot of work to do in the QA area and only 6 months to be able to achieve it. 

To start with, at UDS it was announced that the Ubuntu packages needed to be more stable from now on, so developers are going to be asked, at least for the main components, to write test cases and demonstrate that their code works. This is a huge step towards achieving better levels of quality going forward, because it will improve the testability of the software, if the developers have to test the code they produce, they need to make sure they are able to. Of course, these things do not happen overnight, but it is only natural that we see gradually an increase in the quality of the product and on the stability of the development branches. 

As for the QA Team, we went to plenty of sessions where Quality was being discussed as a priority. Since Precise Pangolin is an LTS release, which means it will be around for 5 years, quality is a key aspect of it. I will explain in more detail our plans to take the testing of Ubuntu to the next level, for now I just wanted to touch base and say that I am back now, after two months of a very busy move and one month full of international flights, so hopefully, I will find time to start blogging more regularly. 

This week I am at UDS-P, it is almost Friday already, so we are finishing off the planning for the coming release and we are about to start implementing all the things we've agreed upon during the week.

I have learnt many things during my first UDS, the most important one being, even though we all talk about quality and testing, each individual you speak to understand a different thing by the most common terms. Let's take test case as an example, most of the people I have spoken to in the Ubuntu Community, when they say test case they mean a list of steps to do something. Coming from a strong QA background, I think that does not constitute a test case, yet we are overloading the term with such meanings. Other persons think that a use case is the same as a test case. This is not true, since use case is a collection of test cases, yet the term is again overloaded and used to mean different things.

That is why we held a session to talk about what is ultimately a test case. Not because I think I know better, in fact, I took a definition from the ISTQB standard, which is an industry standard used by many companies and QA professionals because I don't like to reinvent the wheel when there are wheels already out there being used in all sorts of situations. Having said this, we had some interesting filosofical discussion regarding if the conventional definition of test case is obsolete in the current agile environments or not. I had people in the session saying that it was, that in the XXI century we need to be doing something different, which is fair enough, but I do not agree because I don't really believe you can cut in test planning without a cost down the line. As always, a compromise needs to be found for the particular problem you are trying to solve and that's what we've tried. This is the test case template we came up with and presented: https://wiki.ubuntu.com/QATeam/AutomatedTesting/TestCase.

A test case is a set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

• How boring, pressing buttons all day.
• How cool, being able to test computer games all day and play non stop!

My main duty as a QA Engineer is to help ship a product that is better than if I weren't there doing my job. Products always ship with defects/bugs on them, they key point is not necessarily to fix them all, but to make sure as little as possible escape the testing and go out undiscovered. The ones that threaten security, data integrity or main functionality need to be fixed before releasing, the minor ones are not as critical. It is QA's responsibility as well helping put in place the right controls to avoid introducing those problems in the first place.

The earlier you find a bug (i.e. by writing properly a requirement or a design document, or discovering it before the code is submitted) the cheaper it is to fix.