Extras-testing/QA checklist
(→QA checklist) |
|||
| Line 17: | Line 17: | ||
Developers: filling the ''XSBC-Bugtracker'' field in your packages address this problem (((FIXME: link to documentation?))). Check [[Bugs:Adding_Extra_products|this wiki page]] to get your bugtracker at bugs.maemo.org. | Developers: filling the ''XSBC-Bugtracker'' field in your packages address this problem (((FIXME: link to documentation?))). Check [[Bugs:Adding_Extra_products|this wiki page]] to get your bugtracker at bugs.maemo.org. | ||
| - | ==== | + | ==== Licensing and legal issues ==== |
Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks. | Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks. | ||
| - | |||
| - | |||
| - | |||
==== Illegal or dubious content ==== | ==== Illegal or dubious content ==== | ||
To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity. | To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity. | ||
| - | ==== | + | ==== Broken functionality or reproducible crashes ==== |
| - | + | There is some functionality that does not work or produces crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem. | |
| - | ==== | + | ==== Missing announced features ==== |
| - | + | Core features advertised through UI or product page don't work or are missing. | |
==== Too much memory used in root partition ==== | ==== Too much memory used in root partition ==== | ||
| Line 38: | Line 35: | ||
Feel free to add those packages to [[Opt_Problem/Non-Optified_packages]]. | Feel free to add those packages to [[Opt_Problem/Non-Optified_packages]]. | ||
| + | |||
| + | ==== System performance compromised ==== | ||
| + | Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time. | ||
==== Power management issues ==== | ==== Power management issues ==== | ||
| Line 48: | Line 48: | ||
===Warnings=== | ===Warnings=== | ||
| - | If one of these | + | If one of these warning criterias is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised. |
* Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info. | * Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info. | ||
| Line 68: | Line 68: | ||
1. [ ] Bug database exist. | 1. [ ] Bug database exist. | ||
2. [ ] Licensing ok. | 2. [ ] Licensing ok. | ||
| - | 3. [ ] Working provided features. | + | 3. [ ] No illegal/dubious content. |
| - | + | 4. [ ] Working provided features. | |
| - | + | 5. [ ] No missing announced features. | |
| - | + | 6. [ ] Optification ok. | |
| - | + | 7. [ ] No performance problems. | |
| - | + | 8. [ ] No power management issues. | |
9. [ ] No known security risks. | 9. [ ] No known security risks. | ||
</pre> | </pre> | ||
| Line 87: | Line 87: | ||
1. [x] Bug database exist. | 1. [x] Bug database exist. | ||
2. [ ] Licensing ok. | 2. [ ] Licensing ok. | ||
| - | 3. [x] Working provided features. | + | 3. [ ] No illegal/dubious content. |
| + | 4. [x] Working provided features. | ||
FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). | FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). | ||
FAIL: When exporting file the program crashes (see bug: http://url/456) | FAIL: When exporting file the program crashes (see bug: http://url/456) | ||
| - | + | 5. [ ] No missing announced features. | |
| - | + | 6. [x] Optification ok. | |
FAIL: the package uses 1512kb from root. | FAIL: the package uses 1512kb from root. | ||
| - | + | 7. [ ] No performance problems. | |
| - | + | 8. [ ] No power management issues. | |
| - | + | ||
9. [ ] No known security risks. | 9. [ ] No known security risks. | ||
</pre> | </pre> | ||
Revision as of 21:38, 17 December 2009
The software hosted in extras-testing is not ready for normal users!
PLEASE use it only for testing purposes. Be ready to file proper bug reports instead of posting complaints. Potential problems: crashes, battery drain, poor system performance, full disk space & more - SERIOUSLY! Backing up your data is recommended. In case of trouble you might need to re-flash your device
There are several elements to be considered by developers before promoting software to extras-testing. The same criteria are useful for beta testers before approving applications to go to Extras. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry just let others do the job.
Please check the points of this list. Don't rush to give thumbs up to an application before it is tested. But don't try to delay the release of an app with minor bugs or enhancement proposals separate from this discrete QA checklist.
Blockers
If one of these criteria is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.
Lack of bug reporting database
http://bugs.maemo.org/ is the preferred option. Otherwise it needs to be identified in the http://maemo.org/packages/ page.
Developers: filling the XSBC-Bugtracker field in your packages address this problem (((FIXME: link to documentation?))). Check this wiki page to get your bugtracker at bugs.maemo.org.
Licensing and legal issues
Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.
Illegal or dubious content
To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.
Broken functionality or reproducible crashes
There is some functionality that does not work or produces crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.
Missing announced features
Core features advertised through UI or product page don't work or are missing.
Too much memory used in root partition
The application or its dependencies ignore the recommendation to use the eMMC to install as much files as possible, filling the root partition with 500kb or more.
dpkg -L packagename shows all files installed by a package. Also see Opt Problem.
Feel free to add those packages to Opt_Problem/Non-Optified_packages.
System performance compromised
Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.
Power management issues
Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging shouldn't be compromised.
If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.
Security risks
The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to report it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.
Warnings
If one of these warning criterias is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.
- Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
- Application icon missing or not visible in the device. This is only optional for command-line applications.
Testers with good technical knowledge are invited to look deeper in the Maemo Quality considerations (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.
Non-blockers
The extras-testing QA process shouldn't be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.
- Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
- Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.
QA checklist
Use this checklist until the extras-testing approval interface is fixed. It helps to see what you have done, and what should be still tested.
Testing checklist: 1. [ ] Bug database exist. 2. [ ] Licensing ok. 3. [ ] No illegal/dubious content. 4. [ ] Working provided features. 5. [ ] No missing announced features. 6. [ ] Optification ok. 7. [ ] No performance problems. 8. [ ] No power management issues. 9. [ ] No known security risks.
- Copy&paste this to the comment box.
- Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
- Vote up if there were no FAILs, if there was even one FAIL vote down.
- UI usability issues cannot be used as reason for vote down.
- Always test functionality - that is, run the program and try if it works as it should.
imaginary example:
1. [x] Bug database exist. 2. [ ] Licensing ok. 3. [ ] No illegal/dubious content. 4. [x] Working provided features. FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). FAIL: When exporting file the program crashes (see bug: http://url/456) 5. [ ] No missing announced features. 6. [x] Optification ok. FAIL: the package uses 1512kb from root. 7. [ ] No performance problems. 8. [ ] No power management issues. 9. [ ] No known security risks.