• Intro
• Downloads
• Community
• Development
• News
• TALK

Extras-testing

Revision as of 14:24, 19 August 2009 (view source) jeremiah (Talk | contribs) (→Maemian/Minimae checks after builds) ← Older edit		Latest revision as of 21:00, 12 April 2013 (view source) joerg_rw (Talk | contribs) (→Demotion: clearing ambiguities)
(72 intermediate revisions not shown)
Line 1:		Line 1:
-	The maemo.org extras-testing repository hosts community applications and related packages that are waiting to be evaluated and promoted to Maemo 5 [[Extras]]. The packages hosted here are promoted from [[extras-devel]] and have gone through an automated quality check. Now they need real humans assessing that they are indeed ready to be released.	+	{{Extras-testing}}
-	== Target: Maemo 5 testers ==	+	<br>
		+	The maemo.org extras-testing repository hosts community applications and related packages that are waiting to be evaluated and promoted to Maemo 5 [[Extras]]. The packages hosted here are promoted from [[extras-devel]] and have gone through an automated quality check. Now they need real humans assessing that they are indeed ready to be released. If this article is hard to understand but you want to test software, please see [[Help testing software]] -article.
-	Power users willing to help Maemo 5 developers getting their apps ready are encouraged to activate this repository.	+	== Community Quality Assurance ==
-	'''	+	Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software developed by community projects can match commercial software in terms of features, usability, reliability and fun. But we also face the risk of getting maemo.org Extras associated with beta quality software made by geeks for geeks only, without the last mile of polishing.
-	Applications in extras-testing might break and cause trouble to your system!''' Regular users are enouraged to stay away.	+
-	=== Activating extras-testing ===	+	This is why we have put a community QA process in place in order to help developers to get their software ready for end users. Free community certification for free software.
-	Users willing to give a hand testing and evaluating community applications must start activating the extras-testing repository in the Application Manager of their Maemo 5 compatible device.	+	You can help improve this QA process. Feel free to check the [[Talk:Extras-testing/QA_Checklist|discussion page]] and the [http://talk.maemo.org/showthread.php?t=33317 Talk discussion thread].
-	Application manager --> Application catalogs --> New	+	We run occasional [[Extras-testing/Testing marathon | testing marathons]], where we get together for a day and work on getting through the extras-testing queue. Join in!
		+
		+	== How it works in practice ==
		+	Developers upload their software to [[Extras-devel]], the unstable repository. extras-devel is where anything can break and where no end users are expected unless they know perfectly well what they are doing. When the developer of an application thinks that it's ready for the masses, he or she promotes it to extras-testing. There is a series of automatic tests to filter the jump from extras-devel to extras-testing already. If everything is ok, the application ends up in extras-testing, and from that point it will be subject to a human evaluation.
		+
		+	===Follow new packages on Twitter (@maemoextras)===
		+	If you want to be informed when new packages enter the Extras-Testing repository, you might want to follow [http://twitter.com/maemoextras maemoextras on Twitter].
		+
		+	== The extras-testing QA queue & you ==
		+	The list of applications waiting to be evaluated in Extras-testing can be found at
		+	http://maemo.org/packages/repository/qa/fremantle_extras-testing/
		+
		+	They are sorted by age: the first application is the oldest in the extras-testing queue and the last is the youngest. When a developer sends a new version of an application the old version disappears from the list and the new one is located at the end of the list.
		+
		+	There are three basic types of beta testers:
		+	* The ones that concentrate on the applications on top of the list (the apps that have been longest in the queue). Do this unless you have a better agenda. It is important to give feedback to developers soon. Either they get their software approved to Extras or they get bad reports so work on a new version in extras-devel.
		+	* The ones that concentrate on the applications they regularly use. By following this path you become an expert in specific applications, able to evaluate new releases quickly and get involved in the project as a regular.
		+	* The ones that concentrate on a type of testing. Some prefer to check that the features work and there are no crashes. Others might prefer to look at performance or power management metrics. Testing is a complex activity and nobody is expected to be an expert in all fields.
		+
		+	=== Report soon, report often ===
		+	Testing a specific version of an application thoroughly takes time, and most of us are busy. This is why you are encouraged to handle testing in chunks, testing a specific aspect of one application and reporting back to the page where the evaluations are made. If you report that the content of ExampleApp is correct (something that took you only ten minutes to check), then someone else after you can concentrate on something else. Ten people can look at one thing each, instead of ten people each having to go through the ten things.
		+
		+	=== Try to find the blockers first ===
		+	The [[Extras-testing/QA Checklist|QA Checklist]] contains several points that block an app from promotion to Extras. If there is a flagrant and evident blocker (e.g. wrong use of the Maemo trademark) then there is no immediate need to go further on testing more complex and subtle aspects like power management or system reliability. Just file the blocker bug and raise the flag asking the developer to prepare a fixed version in Extras-devel.
		+
		+	=== One after another ===
		+	If you are testing aspects like power management and system stability it is recommended that you install and try out applications one by one (this is a good idea for all users anyway). If you install many applications at once then it will be difficult to find out which one is causing trouble.
		+
		+	Below you have a modular checklist to help you evaluate apps soon and often.
		+
		+	== How to activate Extras-testing ==
		+
		+	Users willing to give a hand testing and evaluating community applications must start activating the extras-testing repository in the Application Manager of their Maemo 5 compatible device. Note, the word 'fremantle' contains only two letter 'e'. Note the trailing slash in the URL
		+
		+	Application manager, menu, Application catalogs, New
	Catalog name: maemo.org extras-testing		Catalog name: maemo.org extras-testing
-	Web address: http://repository.maemo.org/extras-testing	+	Web address: http://repository.maemo.org/extras-testing/
	Distribution: fremantle		Distribution: fremantle
	Components: free non-free		Components: free non-free
-	The Fremantle Extras-testing repository is being signed with a gpg key. You can add the key to apt like this (as root):	+	== Vote on Packages waiting to be evaluated ==
-	wget http://repository.maemo.org/extras-testing/dists/fremantle/fremantle-extras-testing-pubkey.asc	+	* Voting for fremantle extras-testing packages takes place at [http://maemo.org/packages/repository/qa/fremantle_extras-testing/ Fremantle Repository QA Queue] after completing your QA evaluation.
-	apt-key add fremantle-extras-testing-pubkey.asc	+	* Developers might want to have a look at the [http://maemo.org/packages/repository/qa/all/fremantle_extras-testing_free_armel/ full list of packages] as well.
-	=== Packages waiting to be evaluated ===	+	== Quality Assurance criteria ==
		+	There are several elements to be considered before voting Up/Down. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry, perhaps let others do the testing.
-	* Power users can install any of the [http://maemo.org/packages/repository/qa/fremantle_extras-testing_free_armel/ user applications] and start the evaluation.	+	If you want to help evaluating one application follow the [[Extras-testing/QA checklist|Extras-testing QA Checklist]]
-	* Developers might want to have a look to the [http://maemo.org/packages/repository/qa/all/fremantle_extras-testing_free_armel/ full list of packages] as well.	+
-	=== Quality criteria ===	+	=== Thumbs Up ===
-	There are several elements to be considered before voting Up/Down. Please take your time looking at them. Don't evaluate lightly an application! If you are busy or in a hurry just let others do the job.	+
-	====Blockers====	+	'''Don't vote up an application before having a good understanding of its quality''' based on your own testing and the opinion of other contributors involved.
-	If one of these criteria is not met, the application cannot go to Extras.	+	=== Thumbs Down ===
-	* Reproducible crashes.	+	Whenever you decide to vote an application '''down''', please leave a comment describing the issues you found. The best feedback is a bug report from the bug-tracking system of the application because it allows the developers and testers to track and discuss the problems effectively. Voting thumbs down without any explanation doesn't help the developer in providing better software for you and the end users. Remember, this is not about "like" nearly as much as it is about "doesn't break things".
-	* Visible affects performance and responsiveness of the system.	+
-	* Power management issues affecting battery life.	+
-	* Unacceptable security risks.	+
-	* Core features advertised through UI or product page don't work or are missing.	+
-	* Lack of bug reporting database (http://bugs.maemo.org is the preferred option, otherwise it needs to be identified in the http://maemo.org/packages/ page).	+
-	* Only for updates of apps already existing in Extras: missing page with screenshot in http://maemo.org/downloads.	+
-	* Evident licensing or copyright violation.	+
-	* (((Anything else to say about content? e.g. racist, pornographic...)))	+
-	====Warnings====	+	There is also a mailing [https://garage.maemo.org/mailman/listinfo/testingsquad-comments comments list] that contains all the traffic from the comments on QA pages - it is a good reference to see how testing communication works, what the common issues are, etc.
-	If one of these criteria is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.	+
-	* Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.	+	== Command line applications ==
-	* Application icon missing or not visible in the device.	+
-	Testers with good technical knowledge are invited to look deeper in the [http://maemo.org/maemo_release_documentation/maemo4.1.x/node16.html Maemo Quality considerations] (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.	+	[[/Command line applications | Command line apps]] are a special-case in apps and app testing. They have their own special icon, and should have detailed descriptions and documentation describing their function.
-	=== Voting Up/Down ===	+	== Promotion ==
-		+
-	Whenever you decide to vote an application '''down''', please leave a comment describing what issues you found. This will help the developer understand why his application got voted down.	+
-		+
-	=== Promotion / Demotion ===	+
	An application is promoted to Extras when these criteria are fulfilled:		An application is promoted to Extras when these criteria are fulfilled:
	* At least 10 days of quarantine		* At least 10 days of quarantine
-	* Karma 10 or more (1 vote up = 1 karma point / 1 vote down = -4 karma points)	+	* Karma >10 (1 vote up = 1 karma point / 1 vote down = -1 karma points)
-	* At least 3 members of the testers group have voted. (((what is the testers group needs to be defined, but they are meant to be senior and reliable contributors)))	+	* At least 3 members of the testers group have voted. (proposed improvement, not implemented/final)
-	An application is demoted to Extras-devel when, for some reason, "extras-devel issues" still show up: missing dependencies, install/uninstall problems, "Other" section (what else? are they documented anywhere?).	+	== Demotion ==
		+	An application is demoted from Extras-testing to Extras-devel when these criteria are fulfilled:
		+	* At least 10 days of quarantine have expired and at least one of the following conditions is true
		+	* Karma < -5 (1 vote up = 1 karma point / 1 vote down = -1 karma points)
		+	* When the maintainer votes down the application
		+	*At least 3 members of the testers group have voted down. (proposed improvement, not implemented/final)
	== Tools for testers ==		== Tools for testers ==
-	Testing tools help finding non-obvious flaws and getting useful data from visible problems.	+	{{main|Documentation/devtools/maemo5}}
-		+
-	=== Crash Reporter ===	+
-	Install this tool in your device (((once the Fremantle version is public))) in order to register data from crashes suffered by an application or its dependent components.	+
-		+
-	== Promoting packages to extras-testing ==	+
-		+
-	Developers can tinker as much as they want in the [[extras-devel repository. Once they think their an application is ready to go public they need to promote it to extras-testing following these steps:	+
-	* Go to the armel version of your application.	+	Testing tools help finding non-obvious flaws and getting useful data from visible problems. Several things can be observed from the application to give an idea of whether it is performing as it should. For all these areas tools exist to analyse the application. To get an idea of what kind of tools are in the Maemo 5 SDK tools repository, see the list of [[Documentation/devtools/maemo5#Categories|Maemo 5 developer tools]].
-	* Check if you are listed as maintainer for the package.	+
-	* Check if there is a 'Promote package' link, if there is: click it	+
-	If there are no errors shown your application will be promoted to extras-testing. This can take some time.	+	Some the tools mentioned below are available from the Maemo 5 SDK tools repository. Even though the repository is intended for SDK use, all tools in the repository can be run on-device as well. To activate the tools repository, [[Documentation/devtools/maemo5#Installation|follow the installation instructions]].
-	In case there are warning or errors on the details page of your package, please try to resolve these issues first. If the interface doesn't detect any problem, it will unlock the 'Promote package' link automatically.
-	You don't have to promote the i386 versions of your packages, they will be promoted automatically.	+	=== Process tracing ===
		+	Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces.
-	A promotion can fail because your application depends on an other 'user/*' category application. In this case you need to promote the other application first and wait until that package shows up in Extras-testing.	+	=== CPU usage ===
		+	* Terminal applications: top, [[Documentation/devtools/maemo5/htop|htop]]
		+	* Applet: [http://maemo.org/downloads/product/Maemo5/load-applet/ load-applet] or [http://maemo.org/downloads/product/Maemo5/cpumem-applet/ cpumem-applet] from Extras are handy graphical tools to monitor CPU usage from the Status area on the desktop. The blinking column on the left indicates CPU activity.
		+	* Applications: [http://maemo.org/downloads/product/Maemo5/conky/ conky] can also be used to monitor cpu usage.
-	=== Promotion checks ===	+	=== Memory usage ===
		+	Tracking memory use for leaks.
-	The package interface will try to prevent promotion for packages with known issues. The following tests are in place:	+	[http://maemo.org/downloads/product/Maemo5/load-applet/ load-applet from Extras] is a handy graphical tool to monitor memory usage from the Status area on the desktop. The blinking column on the right indicates memory use.
-	* Application is using one of the official 'user/*' sections.	+	=== Network traffic ===
-	* Promoting person is maintainer for the package.	+	Network use can also be looked at to see whether the application is behaving as expected.
-	* All specified dependencies can be found in the origin repository or neighbour repositories.	+
-	* All specified version dependencies are met.	+
-	* Application doesn't depend on another 'user/*' application which hasn't been promoted yet.	+
-	* Check if all dependencies which are in origin and not in target can be promoted.	+
-	* Check dependencies of dependencies. (Check complete tree)	+
-	* Check if the application hasn't been promoted already	+
-	Extras-testing -> Extras specific tests	+	=== Power consumption ===
-	* Application karma > X	+	For widgets:
-	* TO BE ADDED: Age > X days	+	Users should [[Root access|run as root]] the command "[http://maemo.org/packages/view/powertop/ powertop]" before and after adding the widget to the desktop.  Symptoms of a power-hungry application are a large increase in the number of overall wakeups from the "hildon-home" process.
-	* TO BE ADDED: X voters from tester group	+
-	=== Maemian/Minimae checks after builds ===	+	== Testing marathons ==
-	Jeremiah is implementing the following checks in minimae for this Sprint:	+	Every now and again, we hold testing marathons, to help clear the queue of applications awaiting approval for extras-testing. You can find out more about the [[Extras-testing/Testing marathon | testing marathons]] here.
-	* MD5 sums match, not missing, etc.	+	[[Category:Development]]
-	* .dsc file (Description file) is correctly formatted, has info, etc. Should I check for icons?	+	[[Category:Quality Assurance]]
-	* Use of trademarked names in package (not in version string.) i.e. A package cannot be called MaemoCamera or MaemoMusic since that is use of a registered trademark.	+
-	* Make sure the name of the binary package that gets built is the same as the package name. i.e. a package called foo should build a binary called foo, not bar.	+
-	* Each package has to have a copyright file	+
-	* Copyright file cannot be compressed	+
-	* Remove helper files from copyright, check for properly assigned copyright	+

---

Latest revision as of 21:00, 12 April 2013

The software hosted in extras-testing is not ready for normal users! PLEASE use it only for testing purposes. Be ready to file proper bug reports instead of posting complaints. Potential problems: crashes, battery drain, poor system performance, full disk space & more - SERIOUSLY! Backing up your data is recommended. In case of trouble you might need to re-flash your device.

The maemo.org extras-testing repository hosts community applications and related packages that are waiting to be evaluated and promoted to Maemo 5 Extras. The packages hosted here are promoted from extras-devel and have gone through an automated quality check. Now they need real humans assessing that they are indeed ready to be released. If this article is hard to understand but you want to test software, please see Help testing software -article.

Contents 1 Community Quality Assurance 2 How it works in practice 2.1 Follow new packages on Twitter (@maemoextras) 3 The extras-testing QA queue & you 3.1 Report soon, report often 3.2 Try to find the blockers first 3.3 One after another 4 How to activate Extras-testing 5 Vote on Packages waiting to be evaluated 6 Quality Assurance criteria 6.1 Thumbs Up 6.2 Thumbs Down 7 Command line applications 8 Promotion 9 Demotion 10 Tools for testers 10.1 Process tracing 10.2 CPU usage 10.3 Memory usage 10.4 Network traffic 10.5 Power consumption 11 Testing marathons

Community Quality Assurance

Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software developed by community projects can match commercial software in terms of features, usability, reliability and fun. But we also face the risk of getting maemo.org Extras associated with beta quality software made by geeks for geeks only, without the last mile of polishing.

This is why we have put a community QA process in place in order to help developers to get their software ready for end users. Free community certification for free software.

You can help improve this QA process. Feel free to check the discussion page and the Talk discussion thread.

We run occasional testing marathons, where we get together for a day and work on getting through the extras-testing queue. Join in!

How it works in practice

Developers upload their software to Extras-devel, the unstable repository. extras-devel is where anything can break and where no end users are expected unless they know perfectly well what they are doing. When the developer of an application thinks that it's ready for the masses, he or she promotes it to extras-testing. There is a series of automatic tests to filter the jump from extras-devel to extras-testing already. If everything is ok, the application ends up in extras-testing, and from that point it will be subject to a human evaluation.

Follow new packages on Twitter (@maemoextras)

If you want to be informed when new packages enter the Extras-Testing repository, you might want to follow maemoextras on Twitter.

The extras-testing QA queue & you

The list of applications waiting to be evaluated in Extras-testing can be found at

http://maemo.org/packages/repository/qa/fremantle_extras-testing/ 

They are sorted by age: the first application is the oldest in the extras-testing queue and the last is the youngest. When a developer sends a new version of an application the old version disappears from the list and the new one is located at the end of the list.

There are three basic types of beta testers:

• The ones that concentrate on the applications on top of the list (the apps that have been longest in the queue). Do this unless you have a better agenda. It is important to give feedback to developers soon. Either they get their software approved to Extras or they get bad reports so work on a new version in extras-devel.
• The ones that concentrate on the applications they regularly use. By following this path you become an expert in specific applications, able to evaluate new releases quickly and get involved in the project as a regular.
• The ones that concentrate on a type of testing. Some prefer to check that the features work and there are no crashes. Others might prefer to look at performance or power management metrics. Testing is a complex activity and nobody is expected to be an expert in all fields.

Report soon, report often

Testing a specific version of an application thoroughly takes time, and most of us are busy. This is why you are encouraged to handle testing in chunks, testing a specific aspect of one application and reporting back to the page where the evaluations are made. If you report that the content of ExampleApp is correct (something that took you only ten minutes to check), then someone else after you can concentrate on something else. Ten people can look at one thing each, instead of ten people each having to go through the ten things.

Try to find the blockers first

The QA Checklist contains several points that block an app from promotion to Extras. If there is a flagrant and evident blocker (e.g. wrong use of the Maemo trademark) then there is no immediate need to go further on testing more complex and subtle aspects like power management or system reliability. Just file the blocker bug and raise the flag asking the developer to prepare a fixed version in Extras-devel.

One after another

If you are testing aspects like power management and system stability it is recommended that you install and try out applications one by one (this is a good idea for all users anyway). If you install many applications at once then it will be difficult to find out which one is causing trouble.

Below you have a modular checklist to help you evaluate apps soon and often.

How to activate Extras-testing

Users willing to give a hand testing and evaluating community applications must start activating the extras-testing repository in the Application Manager of their Maemo 5 compatible device. Note, the word 'fremantle' contains only two letter 'e'. Note the trailing slash in the URL

Application manager, menu, Application catalogs, New
Catalog name: maemo.org extras-testing
Web address: http://repository.maemo.org/extras-testing/
Distribution: fremantle
Components: free non-free

Vote on Packages waiting to be evaluated

• Voting for fremantle extras-testing packages takes place at Fremantle Repository QA Queue after completing your QA evaluation.
• Developers might want to have a look at the full list of packages as well.

Quality Assurance criteria

There are several elements to be considered before voting Up/Down. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry, perhaps let others do the testing.

If you want to help evaluating one application follow the Extras-testing QA Checklist

Thumbs Up

Don't vote up an application before having a good understanding of its quality based on your own testing and the opinion of other contributors involved.

Thumbs Down

Whenever you decide to vote an application down, please leave a comment describing the issues you found. The best feedback is a bug report from the bug-tracking system of the application because it allows the developers and testers to track and discuss the problems effectively. Voting thumbs down without any explanation doesn't help the developer in providing better software for you and the end users. Remember, this is not about "like" nearly as much as it is about "doesn't break things".

There is also a mailing comments list that contains all the traffic from the comments on QA pages - it is a good reference to see how testing communication works, what the common issues are, etc.

Command line applications

Command line apps are a special-case in apps and app testing. They have their own special icon, and should have detailed descriptions and documentation describing their function.

Promotion

An application is promoted to Extras when these criteria are fulfilled:

• At least 10 days of quarantine
• Karma >10 (1 vote up = 1 karma point / 1 vote down = -1 karma points)
• At least 3 members of the testers group have voted. (proposed improvement, not implemented/final)

Demotion

An application is demoted from Extras-testing to Extras-devel when these criteria are fulfilled:

• At least 10 days of quarantine have expired and at least one of the following conditions is true
• Karma < -5 (1 vote up = 1 karma point / 1 vote down = -1 karma points)
• When the maintainer votes down the application
• At least 3 members of the testers group have voted down. (proposed improvement, not implemented/final)

Tools for testers

Testing tools help finding non-obvious flaws and getting useful data from visible problems. Several things can be observed from the application to give an idea of whether it is performing as it should. For all these areas tools exist to analyse the application. To get an idea of what kind of tools are in the Maemo 5 SDK tools repository, see the list of Maemo 5 developer tools.

Some the tools mentioned below are available from the Maemo 5 SDK tools repository. Even though the repository is intended for SDK use, all tools in the repository can be run on-device as well. To activate the tools repository, follow the installation instructions.

Process tracing

Tracing process activity can tell whether the application is working as it should, but it is also a good indicator of battery usage. If an application wakes up unnecessarily (i.e. no user interaction or absolutely mandatory timed operations) this can be seen in traces.

CPU usage

• Terminal applications: top, htop
• Applet: load-applet or cpumem-applet from Extras are handy graphical tools to monitor CPU usage from the Status area on the desktop. The blinking column on the left indicates CPU activity.
• Applications: conky can also be used to monitor cpu usage.

Memory usage

Tracking memory use for leaks.

load-applet from Extras is a handy graphical tool to monitor memory usage from the Status area on the desktop. The blinking column on the right indicates memory use.

Network traffic

Network use can also be looked at to see whether the application is behaving as expected.

Power consumption

For widgets: Users should run as root the command "powertop" before and after adding the widget to the desktop. Symptoms of a power-hungry application are a large increase in the number of overall wakeups from the "hildon-home" process.

Testing marathons

Every now and again, we hold testing marathons, to help clear the queue of applications awaiting approval for extras-testing. You can find out more about the testing marathons here.