Extras-testing
(→Memory usage) |
(→Memory usage) |
||
| Line 100: | Line 100: | ||
=== Process tracing === | === Process tracing === | ||
Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces. | Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces. | ||
| + | |||
| + | === CPU usage === | ||
| + | * Terminal apps: top, htop | ||
| + | * Applet: load-applet from [http://maemo.org/downloads/product/Maemo5/load-applet/ Extras] is a handy graphical tool to monitor CPU usage from the Status area on the desktop. The blinking column on the left indicates CPU activity. | ||
=== Memory usage === | === Memory usage === | ||
Following memory use for leaks. | Following memory use for leaks. | ||
| - | load-applet from [http://maemo.org/downloads/product/Maemo5/load-applet/ Extras] is a handy | + | load-applet from [http://maemo.org/downloads/product/Maemo5/load-applet/ Extras] is a handy graphical tool to monitor memory usage from the Status area on the desktop. The blinking column on the right indicates memory use. |
=== Network traffic === | === Network traffic === | ||
Revision as of 15:12, 13 December 2009
PLEASE use it only for testing purposes. Be ready to file proper bug reports instead of posting complaints.
Potential problems: crashes, battery drain, poor system performance, full disk space & more - SERIOUSLY!
The maemo.org extras-testing repository hosts community applications and related packages that are waiting to be evaluated and promoted to Maemo 5 Extras. The packages hosted here are promoted from extras-devel and have gone through an automated quality check. Now they need real humans assessing that they are indeed ready to be released.
Contents |
Community Quality Assurance
Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software developed by community projects can match commercial software in terms of features, usability, reliability and fun. But we also face the risk of getting maemo.org Extras associated with beta quality software made by geeks for geeks only, without the last mile of polishing.
This is why we have put a community QA process in place in order to help developers to get their software ready for end users. Free community certification for free software.
You can help improve this QA process. Feel free to check the discussion page and the Talk discussion thread.
We run occasional testing marathons, where we get together for a day and work on getting through the extras-testing queue. Join in!
How it works in practice
Developers upload their software to Extras-devel, the unstable repository. extras-devel is where anything can break and where no end users are expected unless they know perfectly well what they are doing. When the developer of an application thinks that it's ready for the masses, he or she promotes it to extras-testing. There is a series of automatic tests to filter the jump from extras-devel to extras-testing already. If everything is ok, the application ends up in extras-testing, and from that point it will be subject to a human evaluation.
The extras-testing QA queue & you
The list of applications waiting to be evaluated in Extras-testing can be found at
http://maemo.org/packages/repository/qa/fremantle_extras-testing/
They are sorted by age: the first application is the oldest in the extras-testing queue and the last is the youngest. When a developer sends a new version of an application the old version disappears from the list and the new one is located at the end of the list.
There are three basic types of beta testers:
- The ones that concentrate on the applications on top of the list (the apps that have been longest in the queue). Do this unless you have a better agenda. It is important to give feedback to developers soon. Either they get their software approved to Extras or they get bad reports so work on a new version in extras-devel.
- The ones that concentrate on the applications they regularly use. By following this path you become an expert in specific applications, able to evaluate new releases quickly and get involved in the project as a regular.
- The ones that concentrate on a type of testing. Some prefer to check that the features work and there are no crashes. Others might prefer to look at performance or power management metrics. Testing is a complex activity and nobody is expected to be an expert in all fields.
Report soon, report often
Testing a specific version of an application thoroughly takes time, and most of us are busy. This is why you are encouraged to handle testing in chunks, testing a specific aspect of one application and reporting back to the page where the evaluations are made. If you report that the content of ExampleApp is correct (something that took you only ten minutes to check), then someone else after you can concentrate on something else. Ten people can look at one thing each, instead of ten people each having to go through the ten things.
Try to hint the blockers first
The QA Checklist contains several points that are blockers for an app to get promoted to Extras. If there is a flagrant and evident blocker (e.g. wrong use of the Maemo trademark) then there is no immediate need to go further on testing more complex and subtle aspects like power management or system reliability. Just file the blocker bug and raise the flag asking the developer to prepare a fixed version in Extras-devel.
One after another
If you are testing aspects like power management and system stability it is recommended that you install and try out applications one by one (this is a good idea for all users anyway). If you install many applications at once then it will be difficult to find out which one is causing trouble.
Below you have a modular checklist to help you evaluate apps soon and often.
How to activate Extras-testing
Users willing to give a hand testing and evaluating community applications must start activating the extras-testing repository in the Application Manager of their Maemo 5 compatible device.
Application manager --> Application catalogs --> New Catalog name: maemo.org extras-testing Web address: http://repository.maemo.org/extras-testing Distribution: fremantle Components: free non-free
Packages waiting to be evaluated
- Power users can install any of the user applications and start the evaluation.
- Developers might want to have a look at the full list of packages as well.
Quality Assurance criteria
There are several elements to be considered before voting Up/Down. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry, perhaps let others do the testing.
If you want to help evaluating one application follow the Extras-testing QA_Checklist
Thumbs Up
Don't vote up an application before having a good understanding of its quality based on your own testing and the opinion of other contributors involved.
Thumbs Down
Whenever you decide to vote an application down, please leave a comment describing the issues you found. The best feedback is a bug report from the bug-tracking system of the application because it allows the developers and testers to track and discuss the problems effectively. Voting thumbs down without any explanation doesn't help the developer in providing better software for you and the end users.
Promotion / Demotion NOT FINAL FIXME
An application is promoted to Extras when these criteria are fulfilled:
- At least 10 days of quarantine
- Karma 10 or more (1 vote up = 1 karma point / 1 vote down = -4 karma points)
- At least 3 members of the testers group have voted. (((what is the testers group needs to be defined, but they are meant to be senior and reliable contributors)))
An application is demoted to Extras-devel when blockers are found by at least 5 voters, or when the own developer decides to demote the version candidate.
Tools for testers
Testing tools help finding non-obvious flaws and getting useful data from visible problems.
All the tools mentioned below are available from the Maemo 5 SDK tools repository. Even though the repository is intended for SDK use, all tools in the repository can be run on-device as well.
To activate the tools repository go through the following steps:
Application manager --> Application catalogs --> New Catalog name: Maemo 5 SDK tools Web address: http://repository.maemo.org/ Distribution: fremantle/tools Components: free non-free
The tools in the tools repository are command line utilities, so they will not add applications to the application manager list of downloadable applications. This also means that the tools need to be installed from the X terminal as root. Use rootsh from extras to gain root access.
To get an idea of what kind of tools are/will be in the Maemo 5 SDK tools repository see the Maemo 4.1 SDK tools.
Several things can be observed from the application to give an idea of whether it is performing as it should. For all these areas tools exist to analyse the application.
Process tracing
Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces.
CPU usage
- Terminal apps: top, htop
- Applet: load-applet from Extras is a handy graphical tool to monitor CPU usage from the Status area on the desktop. The blinking column on the left indicates CPU activity.
Memory usage
Following memory use for leaks.
load-applet from Extras is a handy graphical tool to monitor memory usage from the Status area on the desktop. The blinking column on the right indicates memory use.
Network traffic
Network use can also be looked at to see whether the application is behaving as expected.
Power consumption
For widgets: Users should run as root the command "powertop" before and after adding the widget to the desktop. Symptoms of a power-hungry application are a large increase in the number of overall wakeups from the "hildon-home" process.
Testing marathons
Every now and again, we hold testing marathons, to help clear the queue of applications awaiting approval for extras-testing. You can find out more about the testing marathons here.