Extras-testing

(→Activating extras-testing)
(→Tools for testers)
Line 62: Line 62:
Testing tools help finding non-obvious flaws and getting useful data from visible problems.
Testing tools help finding non-obvious flaws and getting useful data from visible problems.
 +
 +
All the tools mentioned below are available from the Maemo 5 SDK tools repository. Even though the repository is intended for SDK use, all tools in the repository can be run on-device as well.
 +
 +
To activate the tools repository go through the following steps:
 +
 +
Application manager --> Application catalogs --> New
 +
Catalog name: Maemo 5 SDK tools
 +
Web address: http://repository.maemo.org/
 +
Distribution: fremantle/tools
 +
Components: free non-free
 +
 +
The tools in the tools repository are command line utilities, so they will not add applications to the application manager list of downloadable applications. This also means that the tools need to be installed from the X terminal as root. Use rootsh from extras to gain root access.
 +
 +
To get an idea of what kind of tools are/will be in the Maemo 5 SDK tools repository see the [http://maemo.org/development/tools/ Maemo 4.1 SDK tools].
 +
 +
Several things can be observed from the application to give an idea of whether it is performing as it should. For all these areas tools exist to analyse the application.
 +
 +
=== Process tracing ===
 +
Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces.
 +
 +
=== Memory usage ===
 +
Following memory use for leaks.
 +
 +
=== Network traffic ===
 +
Network use can also be looked at to see whether the application is behaving as expected.
 +
 +
=== Power consumption ===
=== Crash Reporter ===
=== Crash Reporter ===

Revision as of 19:15, 30 September 2009

The maemo.org extras-testing repository hosts community applications and related packages that are waiting to be evaluated and promoted to Maemo 5 Extras. The packages hosted here are promoted from extras-devel and have gone through an automated quality check. Now they need real humans assessing that they are indeed ready to be released.

Contents

Target: Maemo 5 testers

Power users willing to help Maemo 5 developers getting their apps ready are encouraged to activate this repository. Applications in extras-testing might break and cause trouble to your system! Regular users are enouraged to stay away.

Activating extras-testing

Users willing to give a hand testing and evaluating community applications must start activating the extras-testing repository in the Application Manager of their Maemo 5 compatible device. 

Application manager --> Application catalogs --> New
Catalog name: maemo.org extras-testing
Web address: http://repository.maemo.org/extras-testing
Distribution: fremantle
Components: free non-free

Packages waiting to be evaluated

Quality criteria

There are several elements to be considered before voting Up/Down. Please take your time looking at them. Don't evaluate lightly an application! If you are busy or in a hurry just let others do the job.

Blockers

If one of these criteria is not met, the application cannot go to Extras.

  • Reproducible crashes.
  • Visibly affects performance and responsiveness of the system.
  • Power management issues affecting battery life.
  • Unacceptable security risks.
  • Core features advertised through UI or product page don't work or are missing.
  • Lack of bug reporting database (http://bugs.maemo.org is the preferred option, otherwise it needs to be identified in the http://maemo.org/packages/ page).
  • Only for updates of apps already existing in Extras: missing page with screenshot in http://maemo.org/downloads.
  • Evident licensing or copyright violation.
  • (((Anything else to say about content? e.g. racist, pornographic...)))

Warnings

If one of these criteria is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.

  • Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
  • Application icon missing or not visible in the device.

Testers with good technical knowledge are invited to look deeper in the Maemo Quality considerations (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.

Voting Up/Down

Whenever you decide to vote an application down, please leave a comment describing what issues you found. This will help the developer understand why his application has been voted down.

Promotion / Demotion

An application is promoted to Extras when these criteria are fulfilled:

  • At least 10 days of quarantine
  • Karma 10 or more (1 vote up = 1 karma point / 1 vote down = -4 karma points)
  • At least 3 members of the testers group have voted. (((what is the testers group needs to be defined, but they are meant to be senior and reliable contributors)))

An application is demoted to Extras-devel when, for some reason, "extras-devel issues" still show up: missing dependencies, install/uninstall problems, "Other" section (what else? are they documented anywhere?).

Tools for testers

Testing tools help finding non-obvious flaws and getting useful data from visible problems.

All the tools mentioned below are available from the Maemo 5 SDK tools repository. Even though the repository is intended for SDK use, all tools in the repository can be run on-device as well.

To activate the tools repository go through the following steps: 

Application manager --> Application catalogs --> New
Catalog name: Maemo 5 SDK tools
Web address: http://repository.maemo.org/
Distribution: fremantle/tools
Components: free non-free

The tools in the tools repository are command line utilities, so they will not add applications to the application manager list of downloadable applications. This also means that the tools need to be installed from the X terminal as root. Use rootsh from extras to gain root access.

To get an idea of what kind of tools are/will be in the Maemo 5 SDK tools repository see the Maemo 4.1 SDK tools.

Several things can be observed from the application to give an idea of whether it is performing as it should. For all these areas tools exist to analyse the application.

Process tracing

Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces.

Memory usage

Following memory use for leaks.

Network traffic

Network use can also be looked at to see whether the application is behaving as expected.

Power consumption

Crash Reporter

Install this tool in your device (((once the Fremantle version is public))) in order to register data from crashes suffered by an application or its dependent components.

Promoting packages to extras-testing

Developers can tinker as much as they want in the [[extras-devel repository. Once they think their an application is ready to go public they need to promote it to extras-testing following these steps:

  • Go to the armel version of your application.
  • Check if you are listed as maintainer for the package.
  • Check if there is a 'Promote package' link, if there is: click it

If there are no errors shown your application will be promoted to extras-testing. This can take some time.

In case there are warning or errors on the details page of your package, please try to resolve these issues first. If the interface doesn't detect any problem, it will unlock the 'Promote package' link automatically.

You don't have to promote the i386 versions of your packages, they will be promoted automatically.

A promotion can fail because your application depends on an other 'user/*' category application. In this case you need to promote the other application first and wait until that package shows up in Extras-testing.

Promotion checks

The package interface will try to prevent promotion for packages with known issues. The following tests are in place:

  • Application is using one of the official 'user/*' sections.
  • Promoting person is maintainer for the package.
  • All specified dependencies can be found in the origin repository or neighbour repositories.
  • All specified version dependencies are met.
  • Application doesn't depend on another 'user/*' application which hasn't been promoted yet.
  • Check if all dependencies which are in origin and not in target can be promoted.
  • Check dependencies of dependencies. (Check complete tree)
  • Check if the application hasn't been promoted already

Extras-testing -> Extras specific tests

  • Application karma > X
  • TO BE ADDED: Age > X days
  • TO BE ADDED: X voters from tester group

Maemian/Minimae checks after builds

Jeremiah is implementing the following checks in minimae for this Sprint:

  • Control information is UTF-8
  • Control file contains an image (Not having an image is not an error, it's just that minimae should notice if there is an image or not, its size and type.)
  • XB- (or XS-) Fields in control file
  • Use of trademarked names in package (not in version string.) i.e. A package cannot be called MaemoCamera or MaemoMusic since that is use of a registered trademark.
  • Make sure the name of the binary package that gets built is the same as the package name. i.e. a package called foo should build a binary called foo, not bar.
  • Each package has to have a copyright file
  • Copyright file cannot be compressed, i.e. zipped
  • Remove helper files from copyright, check for properly assigned copyright
  • Make sure the Architecture field is correct according to debian specifications. I.e. all, any, or a specific arch - multiple arches not allowed.
Retrieved from "http://wiki.maemo.org/Extras-testing"