Testing Squad

(The application has no bugtracker, should I thumb it down?)
(testingsquad-comments mailing list)
 
(12 intermediate revisions not shown)
Line 6: Line 6:
* You can help testing beta software available in the [[Extras-testing]] repository, in order to provide high quality software to the Maemo users.
* You can help testing beta software available in the [[Extras-testing]] repository, in order to provide high quality software to the Maemo users.
-
* If you don't feel ready to join the testing Squad or don't have enough skills, you can still help by testing the applications from a user point of view (functionalities, description, etc...).
+
* If you don't feel ready to join the testing Squad or don't have enough skills, you can still help by testing the applications from a user point of view (functionalities, description, etc...). For more information read article [[Help testing software]].
* Don't be shy to give feedback (both positive and negative) to author of an application, using [http://maemo.org/packages/ packages interface] or [http://bugs.maemo.org Bugzilla] if you've found obvious bugs. Such activity does not require you to become Testing Squad member.
* Don't be shy to give feedback (both positive and negative) to author of an application, using [http://maemo.org/packages/ packages interface] or [http://bugs.maemo.org Bugzilla] if you've found obvious bugs. Such activity does not require you to become Testing Squad member.
==How to join the Testing Squad==
==How to join the Testing Squad==
* Read the "[[Testing Squad#How_everybody_can_help|How everybody can help]]" section above.
* Read the "[[Testing Squad#How_everybody_can_help|How everybody can help]]" section above.
-
* Join the Testing Squad group at [https://garage.maemo.org/projects/testingsquad/ maemo.garage].
 
* Subscribe the Testing Squad [https://garage.maemo.org/mail/?group_id=1273 mailing list].
* Subscribe the Testing Squad [https://garage.maemo.org/mail/?group_id=1273 mailing list].
 +
* Attend QA team meetings
 +
** Read minutes of [[QA meeting | our last meeting]]
* Welcome to the team! :-)
* Welcome to the team! :-)
Line 18: Line 19:
Mailing list for Testing Squad members is meant to keep track of apps in testing queue, so every member is aware which apps require most attention at given time. If you want to become Testing Squad member, [https://garage.maemo.org/mail/?group_id=1273 subscribing] to mailing list is essential.
Mailing list for Testing Squad members is meant to keep track of apps in testing queue, so every member is aware which apps require most attention at given time. If you want to become Testing Squad member, [https://garage.maemo.org/mail/?group_id=1273 subscribing] to mailing list is essential.
 +
 +
There is also a mailing [https://garage.maemo.org/mailman/listinfo/testingsquad-comments comments list] that contains all the traffic from the comments on QA pages - it is a good reference to see how testing communication works, what the common issues are, etc.
==How Testing Squad works==
==How Testing Squad works==
Line 26: Line 29:
* The testers test the applications freely in their free time.
* The testers test the applications freely in their free time.
* In the end of the week a evaluation is made and any pending issue is discussed among the team.
* In the end of the week a evaluation is made and any pending issue is discussed among the team.
 +
 +
==Super-testers==
 +
 +
Super-testers are members of the testing-squad with a proven track record (currently testers with more than 100 package testing karma). Their votes (can) count a bit differently than those of regular testers. Three (3) votes down from the testers will demote an application when the 20 day limit is reached. This should take care of the obviously broken lingering packages. When 3 super-tester votes are up and 20 days have passed -> promotion unlock. This will only be able to influence promote/reject when at least 3 votes are the same and less than 3 differ.
 +
 +
See https://garage.maemo.org/pipermail/testingsquad-list/2010-July/000089.html for more details on the initial setup and some procedural detail on super-testers.
==Packages marked for demotion==
==Packages marked for demotion==
Let's keep the testing queue clean:  
Let's keep the testing queue clean:  
-
[[/Packages Marked for Demotion| Packages Marked for Demotion]]
+
[[/Packages Marked for Demotion| Packages Marked for Demotion]].
 +
So everyone please spend couple seconds and give thumbs down for those.
== Code of Conduct ==
== Code of Conduct ==
Line 48: Line 58:
Follow instructions at [[Extras-testing/QA_Checklist| Q&A Checklist]] to ensure consistent testing procedure.
Follow instructions at [[Extras-testing/QA_Checklist| Q&A Checklist]] to ensure consistent testing procedure.
 +
 +
[[Help testing software]] is more thorough article about the same matter.
=== '''The application has a usable but very ugly UI, should I thumb it down?''' ===
=== '''The application has a usable but very ugly UI, should I thumb it down?''' ===
Line 53: Line 65:
No, developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.  
No, developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.  
-
=== '''The application has no bugtracker, should I thumb it down?''' ===
+
=== '''The application has no bugtracker, or an invalid one, should I thumb it down?''' ===
 +
 
 +
Yes: the bugtracker field is mandatory, applications without that field or with an invalid one can't be promoted to extras.
-
Yes: the bugtracker field is mandatory, applications without that field can't be promoted to extras.
+
=== '''How should I decide if a bugtracker link is valid?''' ===
-
You should also check if that field is valid (e.g. some developers just use "http://bugs.maemo.org" without a product); e-mail addresses are valid for small applications (themes, widgets,...).
+
-
'''Requirement proposal''': After clicking bugtracker link, you can start filing a bug right away or just after clicking one link at maximum at the page that opens. --[[User:ossipena|ossipena]] 09:38, 24 February 2010 (UTC)
+
We are currently deciding some objective criteria for answering this question. Read article [[Tracking bugs in Extras]] for more info.
=== '''I found a critical bug during testing, what should I do? ===
=== '''I found a critical bug during testing, what should I do? ===

Latest revision as of 22:56, 28 July 2010

Contents

[edit] What is a Testing Squad?

Testing Squad is a team of Maemo community members that are involved in testing applications queued in Extras-testing repository in regular and systematic manner. The idea is to ensure high quality applications are promoted to end-user oriented Extras repository, instead of being stuck in Extras-testing forever, just because there is nobody willing to put enough effort to test these apps and make sure they are mature enough to hit Extras for everyone to use.

[edit] How everybody can help

  • You can help testing beta software available in the Extras-testing repository, in order to provide high quality software to the Maemo users.
  • If you don't feel ready to join the testing Squad or don't have enough skills, you can still help by testing the applications from a user point of view (functionalities, description, etc...). For more information read article Help testing software.
  • Don't be shy to give feedback (both positive and negative) to author of an application, using packages interface or Bugzilla if you've found obvious bugs. Such activity does not require you to become Testing Squad member.

[edit] How to join the Testing Squad

[edit] Mailing list

Mailing list for Testing Squad members is meant to keep track of apps in testing queue, so every member is aware which apps require most attention at given time. If you want to become Testing Squad member, subscribing to mailing list is essential.

There is also a mailing comments list that contains all the traffic from the comments on QA pages - it is a good reference to see how testing communication works, what the common issues are, etc.

[edit] How Testing Squad works

The Testing Squad follows a very simple process:

  • Each week a member of the Testing Squad posts a list of applications that should be tested during that week.
  • The testers test the applications freely in their free time.
  • In the end of the week a evaluation is made and any pending issue is discussed among the team.

[edit] Super-testers

Super-testers are members of the testing-squad with a proven track record (currently testers with more than 100 package testing karma). Their votes (can) count a bit differently than those of regular testers. Three (3) votes down from the testers will demote an application when the 20 day limit is reached. This should take care of the obviously broken lingering packages. When 3 super-tester votes are up and 20 days have passed -> promotion unlock. This will only be able to influence promote/reject when at least 3 votes are the same and less than 3 differ.

See https://garage.maemo.org/pipermail/testingsquad-list/2010-July/000089.html for more details on the initial setup and some procedural detail on super-testers.

[edit] Packages marked for demotion

Let's keep the testing queue clean: Packages Marked for Demotion. So everyone please spend couple seconds and give thumbs down for those.

[edit] Code of Conduct

  • Don't vote up an application before having a good understanding of its quality.
  • Don't vote an application based in others comments, test it by yourself.
  • Leave a comment or fill a bug report (preferred) every time you vote down an application.
  • Don't vote down an application based in your personal preferences, always respect the Q&A rules.

[edit] Testers FAQ

[edit] Do I need special skills to become a tester?

No, anyone could become a tester (in fact you already are if you for example report bugs or give thumbs up/down to certain applications), however before becoming a member of Testing Squad you need to be ready to test various Extras-testing applications on more regular basis.

[edit] How to test applications properly?

Follow instructions at Q&A Checklist to ensure consistent testing procedure.

Help testing software is more thorough article about the same matter.

[edit] The application has a usable but very ugly UI, should I thumb it down?

No, developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.

[edit] The application has no bugtracker, or an invalid one, should I thumb it down?

Yes: the bugtracker field is mandatory, applications without that field or with an invalid one can't be promoted to extras.

[edit] How should I decide if a bugtracker link is valid?

We are currently deciding some objective criteria for answering this question. Read article Tracking bugs in Extras for more info.

[edit] I found a critical bug during testing, what should I do?

First you should report the bug in the application bugtracker, after that ask the developer to thumb down the application(in order to demote it) and add it to this list.

[edit] Where to report bugs?

If the application has a regular product section in Maemo bugzilla, just use it: Enter a new bug report -> Extras -> application-name. Otherwise follow the link in the application bugtracker (or write an email to the developer if it contains a mailto link).

[edit] Should I always comment?

You don't need to comment if you don't have anything to add to the discussion; for example if someone already commented that the package needs a bugtracker or is not optified, you don't need to repeat the same, just vote down.