Extras-testing/QA Checklist/QA Improvements

(Thumbs Up)
(Speed Promotion: Alternative)
 
(16 intermediate revisions not shown)
Line 4: Line 4:
* Maintainer - the owner of the package under testing
* Maintainer - the owner of the package under testing
* Tester - Any community member
* Tester - Any community member
-
* Master - selected persons, for example with high testing karma or testing squad members.
+
* Master/Admin - selected testing squad members.
== Automatic checks/Autobuilder ==
== Automatic checks/Autobuilder ==
-
* Bugtracker field
+
* <s>Bugtracker field</s> - '''Done'''
-
* Optified and dependencies too
+
* <s>That the description field is not empty</s> - '''Done'''
 +
* Require description field content check only if description has changed
 +
* Optified and dependencies are optified too
* License files included and headers have copyright/license.
* License files included and headers have copyright/license.
Line 15: Line 17:
== Package Interface ==
== Package Interface ==
-
* Changelog should be displayed.
+
* <s>Changelog should be displayed</s> - '''Done'''
-
* Votes should be changeable.
+
* A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
 +
* If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
 +
* <s>Votes should be changeable</s> - '''Done'''
* Each package that enters or leaves testing triggers a e-mail for the testing squad list
* Each package that enters or leaves testing triggers a e-mail for the testing squad list
* Link to Wiki so that details of test criteria are always accessible to new testers
* Link to Wiki so that details of test criteria are always accessible to new testers
Line 31: Line 35:
=== Demotion ===
=== Demotion ===
-
* Packages can be demoted at any time by his maintainers or by a member of the testing squad (demotions should be advertised in the testing squad list).
+
* Packages can be demoted at any time by their maintainers ('''Implemented''') or by a member of the testing squad (demotions should be advertised in the testing squad list).
-
* When demoting a package there's a option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
+
* When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
 +
 
 +
=== Speed Promotion ===
 +
* Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...);
 +
* Changes must be easily visible/documented (run diff agains version in Extras)
 +
* Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. [[User:ossi1967|ossi1967]]
 +
 
 +
=== Speed Promotion (alternative) ===
 +
Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing.
 +
 
 +
However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --[[User:jaffa|Jaffa]] 13:12, 12 May 2010 (UTC)
== Check List ==
== Check List ==
Line 76: Line 90:
== Testing Squad ==
== Testing Squad ==
-
* Can demote packages when there's blockers.
+
* Can demote packages when there are known blockers.
* Can promote packages when they are stuck in the testing queue for a while without any known blocker.
* Can promote packages when they are stuck in the testing queue for a while without any known blocker.
 +
* Can promote packages in speed promotion process [[User:ossi1967|ossi1967]]
=== Testing Squad mailing list ===
=== Testing Squad mailing list ===
-
* Public mailing list where are discussed concerns/improvements to the QA process.
+
* Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
-
* Receives a notification for each packages that enters testing, is demoted or is promoted.
+
* Receives an automatic notification for each package that enters testing, is demoted or is promoted.
 +
[[Category:Quality Assurance]]

Latest revision as of 13:12, 12 May 2010

Note: Work in progress, none of the action points below are definitive.

Contents

[edit] Roles

  • Maintainer - the owner of the package under testing
  • Tester - Any community member
  • Master/Admin - selected testing squad members.

[edit] Automatic checks/Autobuilder

  • Bugtracker field - Done
  • That the description field is not empty - Done
  • Require description field content check only if description has changed
  • Optified and dependencies are optified too
  • License files included and headers have copyright/license.

[edit] Tool for easy on device testing

  • Some easy way to capture a log of power usage, used files and open ports.

[edit] Package Interface

  • Changelog should be displayed - Done
  • A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
  • If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
  • Votes should be changeable - Done
  • Each package that enters or leaves testing triggers a e-mail for the testing squad list
  • Link to Wiki so that details of test criteria are always accessible to new testers

[edit] Thumbs Up

  • Collapsable check list will appear, testers should mark the fields tested.
  • Promotion should occur at +10 karma (if there's at least one completed check list ???)
    • or maybe it would be better to have one check per testing task expect for "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 )

[edit] Thumbs Down

  • Testers must comment on thumbs down.
  • Maintainers thumb down will demote the application from the testing queue.
  • X (fix me) thumbs down will demote the app.

[edit] Demotion

  • Packages can be demoted at any time by their maintainers (Implemented) or by a member of the testing squad (demotions should be advertised in the testing squad list).
  • When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.

[edit] Speed Promotion

  • Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...);
  • Changes must be easily visible/documented (run diff agains version in Extras)
  • Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. ossi1967

[edit] Speed Promotion (alternative)

Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing.

However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --Jaffa 13:12, 12 May 2010 (UTC)

[edit] Check List

1. [ ] Licensing ok.
2. [ ] Description field ok.
--
3. [ ] Announced features available.
4. [ ] Working provided features.
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments:
  • Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
  • Vote up if there were no FAILs. If there was even one FAIL, vote down.
  • UI usability issues cannot be used as a reason for vote down.
  • Always test functionality - that is, run the program and try if it works as it should.

imaginary example:

1. [x] Licensing ok.
2. [x] Description field ok.
 FAIL: the package doesn't have a description.
--
3. [ ] Announced features available.
4. [x] Working provided features.
 FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). 
 FAIL: When exporting file the program crashes (see bug: http://url/456)
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments: 
I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678

[edit] Testing Squad

  • Can demote packages when there are known blockers.
  • Can promote packages when they are stuck in the testing queue for a while without any known blocker.
  • Can promote packages in speed promotion process ossi1967

[edit] Testing Squad mailing list

  • Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
  • Receives an automatic notification for each package that enters testing, is demoted or is promoted.