Extras-testing/QA checklist

qgil (Talk | contribs)
(New page: == Community Quality Assurance == Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software develo...)
Newer edit →

Revision as of 11:43, 22 October 2009

Contents

Community Quality Assurance

Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software developed by community projects can match commercial software in terms of features, usability, reliability and fun. But we also face the risk of getting maemo.org Extras associated to beta quality software without the last mile of polishing made by geeks for geeks only.

This is why we have put a community QA process in place in order to help developers to get their software ready for end users. Free community certification for free software.

How it works in practice

Developers upload their software to extras-devel, the unstable repository. extras-devel is where anything can break and where no end users are awaited unless they know perfectly well what are they doing. One day the developer of an application thinks that it's ready for the masses and promotes it to extras-testing. There is a series of automatic tests filtering the jump from extras-devel to extras-testing already. If everything is ok, the application ends up in extras-testing, and from that point it will be subject to a human evaluation.

The extras-testing QA queue & you

The list of applications waiting to be evaluated can be found at http://maemo.org/packages/repository/qa/fremantle_extras-testing_free_armel/ . They are sorted by age: the first application is the oldest in the extras-testing queue and the last is the youngest. When a developer sends a new version of an application the old version disappears from the list and the new one is located at the end of the list.

There are three basic types of betatesters:

  • The ones that concentrate on the applications on top of the list. Do this unless you have a better agenda. It is important to give feedback to developers soon. Either they get their software approved to Extras or they get bad reports to work on a new version in extras-devel.
  • The ones that concentrate on the applications they regularly use. By following this path you become an expert in specific applications, able to evaluate quite fast new releases and get slowly involved in the project as a regular.
  • The ones that concentrate on a type of testing. Some prefer to check that the features work and there are no crashes. Others might prefer to look at performance or power management metrics. Testing is a complex activity and nobody is expected to be an expert in all fields.

Report soon, report often

Testing a specific version of an application throughout takes time, and most of us are busy. This is why you are encouraged to handle testing in chunks, testing a specific aspect of one application and reporting back to the page where the evaluations are made. If you report that the content of ExampleApp is correct (something that took you only 10 minutes to check), then someone else after you can concentrate on something else. 10 people can look at one thing, instead of 10 people having to go each one through the 10 things.

Below you have a modular checklist to help you evaluate apps soon and often.


There are several elements to be considered by developers before promoting software to extras-testing. The same criteria are useful for betatesters before approving applications to go to Extras.efore voting Up/Down. Please take your time looking at them. Don't evaluate lightly an application! If you are busy or in a hurry just let others do the job.

Blockers

If one of these criteria is not met, the application cannot go to Extras.

  • Reproducible crashes.
  • Visibly affects performance and responsiveness of the system.
  • Power management issues affecting battery life.
  • Unacceptable security risks.
  • Core features advertised through UI or product page don't work or are missing.
  • Lack of bug reporting database (http://bugs.maemo.org is the preferred option, otherwise it needs to be identified in the http://maemo.org/packages/ page).
  • Only for updates of apps already existing in Extras: missing page with screenshot in http://maemo.org/downloads.
  • Evident licensing or copyright violation.
  • (((Anything else to say about content? e.g. racist, pornographic...)))

Warnings

If one of these criteria is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.

  • Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
  • Application icon missing or not visible in the device.

Testers with good technical knowledge are invited to look deeper in the Maemo Quality considerations (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.

Retrieved from "http://wiki.maemo.org/Extras-testing/QA_checklist"