Precise Quality

January 17, 2012 Gema Automation, Metrics

During the past three months we have been trying to improve the quality and stability of Precise Pangolin, Ubuntu next LTS release.

This is an interesting journey of starting to do testing regularly and care about the results so that they get a lot of attention and bugs get fixed soon, but also looking into the future and where we want our testing to be in 2 or 3 years. Relying on manual testing and bits and pieces of automated testing is where we are. Full automated testing and important bits and pieces done manually when automation is not possible is where we want to be. The Ubuntu QA Team is working hard to get this right and we welcome as much help as we can get.

We have a Jenkins instance in place, where we publish the results of the testing we do on daily basis. We also have instructions on how to read those results and make sense of them. We have a list of bugs that are currently open, that we found with the automated testing and the list of bugs that were found by our testing and have been closed. This is where we are at the moment, improving every day our operations and reporting so that we can give a better service and we can get more out of our testing.

Any ideas, suggestions or help is more than welcome, just get in touch with us at ubuntu-qa@ubuntu.com.

No comments

Crazy month

November 24, 2011 Gema QA, Testability

This last month I have been travelling a lot for work. I went to UDS in Orlando at the end of October, then back home to the UK, then a week later went to Lexington in Boston to do a Sprint (one week of everybody in the same room, working together) with the QA Team. All in all, it's been 4 weeks now since I have had a weekend off work, because I have been travelling on the weekends. 

It has been very productive, though. We decided plenty of things that needed to be decided and started the Precise Pangolin release with a lot of work to do in the QA area and only 6 months to be able to achieve it. 

To start with, at UDS it was announced that the Ubuntu packages needed to be more stable from now on, so developers are going to be asked, at least for the main components, to write test cases and demonstrate that their code works. This is a huge step towards achieving better levels of quality going forward, because it will improve the testability of the software, if the developers have to test the code they produce, they need to make sure they are able to. Of course, these things do not happen overnight, but it is only natural that we see gradually an increase in the quality of the product and on the stability of the development branches. 

As for the QA Team, we went to plenty of sessions where Quality was being discussed as a priority. Since Precise Pangolin is an LTS release, which means it will be around for 5 years, quality is a key aspect of it. I will explain in more detail our plans to take the testing of Ubuntu to the next level, for now I just wanted to touch base and say that I am back now, after two months of a very busy move and one month full of international flights, so hopefully, I will find time to start blogging more regularly. 

 

No comments

Quality, agreeing on the terminology

November 4, 2011 Gema Glossary

During my first three months at Canonical, and being part of the QA Ubuntu crew, I have been trying to figure out what needed to happen to improve the testing in Ubuntu. It is clear to me now, that we are starting from the basics and building from there. This is good in many ways, because it allows Ubuntu to get the bulk of the testing right from the beginning. So far we have relied a lot in the early adopters finding the problems and raising them, we want to move to better user experience from day one, so that testers that want to contribute to the quality of the product can focus on more complex problems, rather than finding the annoying simple ones.

This week I am at UDS-P, it is almost Friday already, so we are finishing off the planning for the coming release and we are about to start implementing all the things we've agreed upon during the week.

I have learnt many things during my first UDS, the most important one being, even though we all talk about quality and testing, each individual you speak to understand a different thing by the most common terms. Let's take test case as an example, most of the people I have spoken to in the Ubuntu Community, when they say test case they mean a list of steps to do something. Coming from a strong QA background, I think that does not constitute a test case, yet we are overloading the term with such meanings. Other persons think that a use case is the same as a test case. This is not true, since use case is a collection of test cases, yet the term is again overloaded and used to mean different things.

That is why we held a session to talk about what is ultimately a test case. Not because I think I know better, in fact, I took a definition from the ISTQB standard, which is an industry standard used by many companies and QA professionals because I don't like to reinvent the wheel when there are wheels already out there being used in all sorts of situations. Having said this, we had some interesting filosofical discussion regarding if the conventional definition of test case is obsolete in the current agile environments or not. I had people in the session saying that it was, that in the XXI century we need to be doing something different, which is fair enough, but I do not agree because I don't really believe you can cut in test planning without a cost down the line. As always, a compromise needs to be found for the particular problem you are trying to solve and that's what we've tried. This is the test case template we came up with and presented: https://wiki.ubuntu.com/QATeam/AutomatedTesting/TestCase.

 

A test case is a set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

 

 

No comments

What is testing?

August 30, 2011 Gema QA, Testability, Usability

I would like to give a brief introduction to testing as I see it in the first post. I know that when I say to people I do software testing, the first thing that comes to their mind is one of these two:

  • How boring, pressing buttons all day.
  • How cool, being able to test computer games all day and play non stop!

Neither of those views is completely accurate and the job is sometimes boring and sometimes very cool. You need to be able to do coding, to understand how software is created, designed, submitted and released, and most importantly, you need to be able to distinguish between correct behaviour and incorrect behaviour.

What makes software behaviour correct or incorrect is something open for discussion in my opinion, but I would say that when a feature works as expected by an average user, then its behaviour is correct. Formally, when a feature works as it says in the documentation (and the documentation answers all that was asked for in the requirements), then the behaviour is correct; my only problem with this is sometimes documents say weird things. So I think test/QA people should be involved in reviewing engineering documents (requirements, designs, specifications, API definitions) as early as possible, to help correct ambiguities and shape software towards testability and usability.

My main duty as a QA Engineer is to help ship a product that is better than if I weren't there doing my job. Products always ship with defects/bugs on them, they key point is not necessarily to fix them all, but to make sure as little as possible escape the testing and go out undiscovered. The ones that threaten security, data integrity or main functionality need to be fixed before releasing, the minor ones are not as critical. It is QA's responsibility as well helping put in place the right controls to avoid introducing those problems in the first place.

The earlier you find a bug (i.e. by writing properly a requirement or a design document, or discovering it before the code is submitted) the cheaper it is to fix.
No comments