Extras-testing/QA checklist

(→Lack of bug reporting database)
(→Blockers)
 
(70 intermediate revisions not shown)
Line 1: Line 1:
-
== Community Quality Assurance ==
+
{{Extras-testing}}
-
Offering good quality community software to owners of Maemo devices is a top priority. We have a chance to show the world that open source software developed by community projects can match commercial software in terms of features, usability, reliability and fun. But we also face the risk of getting maemo.org Extras associated with beta quality software made by geeks for geeks only, without the last mile of polishing.
+
-
This is why we have put a community QA process in place in order to help developers to get their software ready for end users. Free community certification for free software.
+
There are several elements to be considered by developers before promoting software to extras-testing. The same criteria are useful for beta testers before approving applications to go to Extras. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry just let others do the job.
-
You can help improve this QA process. Feel free to check the [[Talk:Extras-testing/QA_Checklist|discussion page]] and the [http://talk.maemo.org/showthread.php?t=33317 Talk discussion thread].
+
Please check the points of this list. Don't rush to give thumbs up to an application before it is tested. But don't try to delay the release of an app with minor bugs or enhancement proposals separate from this discrete QA checklist.
-
== How it works in practice ==
+
==QA Improvements==
-
Developers upload their software to extras-devel, the unstable repository. extras-devel is where anything can break and where no end users are expected unless they know perfectly well what they are doing. When the developer of an application thinks that it's ready for the masses, he or she promotes it to extras-testing. There is a series of automatic tests to filter the jump from extras-devel to extras-testing already. If everything is ok, the application ends up in extras-testing, and from that point it will be subject to a human evaluation.
+
-
=== The extras-testing QA queue & you ===
+
There is a ongoing QA (Quality Assurance) improvements discussion in talk.maemo.org (see [http://talk.maemo.org/showthread.php?t=41179 here] and [http://talk.maemo.org/showthread.php?t=33317 here]) and the [http://lists.maemo.org/pipermail/maemo-community/2010-January/003834.html mailing list]. One task is to [http://talk.maemo.org/showthread.php?t=42055 establish a testing squad].
-
The list of applications waiting to be evaluated in extras-testing can be found at http://maemo.org/packages/repository/qa/fremantle_extras-testing/ . They are sorted by age: the first application is the oldest in the extras-testing queue and the last is the youngest. When a developer sends a new version of an application the old version disappears from the list and the new one is located at the end of the list.
+
The results can be found at this wiki page: [[Extras-testing/QA_Checklist/QA_Improvements|QA improvements]]
-
There are three basic types of beta testers:
+
===QA checklist===
-
* The ones that concentrate on the applications on top of the list (the apps that have been longest in the queue). Do this unless you have a better agenda. It is important to give feedback to developers soon. Either they get their software approved to Extras or they get bad reports so work on a new version in extras-devel.
+
-
* The ones that concentrate on the applications they regularly use. By following this path you become an expert in specific applications, able to evaluate new releases quickly and get involved in the project as a regular.
+
-
* The ones that concentrate on a type of testing. Some prefer to check that the features work and there are no crashes. Others might prefer to look at performance or power management metrics. Testing is a complex activity and nobody is expected to be an expert in all fields.
+
-
=== Report soon, report often ===
+
Use this checklist until the extras-testing approval interface is fixed. It helps to see what you have tested and what other testers should still test. This checklist condenses the blockers explained below, read the explanation below before you mark a test as done.
-
Testing a specific version of an application thoroughly takes time, and most of us are busy. This is why you are encouraged to handle testing in chunks, testing a specific aspect of one application and reporting back to the page where the evaluations are made. If you report that the content of ExampleApp is correct (something that took you only ten minutes to check), then someone else after you can concentrate on something else. Ten people can look at one thing each, instead of ten people each having to go through the ten things.
+
-
=== One after another ===
+
<pre>
-
If you are testing aspects like power management and system stability it is recommended that you install and try out applications one by one (this is a good idea for all users anyway). If you install many applications at once then it will be difficult to find out which one is causing trouble.
+
1. [ ] Bug database exist.
 +
2. [ ] Licensing ok.
 +
3. [ ] No illegal/dubious content.
 +
4. [ ] Working provided features.
 +
5. [ ] No missing announced features.
 +
6. [ ] Optification ok.
 +
7. [ ] No performance problems.
 +
8. [ ] No power management issues.
 +
9. [ ] No known security risks.
-
Below you have a modular checklist to help you evaluate apps soon and often.
+
Additional comments:
 +
</pre>
-
==The checklist==
+
'''Note: Only paste this template if there is some problem with the application or if you have not tested some of the points above, let's not spam the developers.'''
-
There are several elements to be considered by developers before promoting software to extras-testing. The same criteria are useful for beta testers before approving applications to go to Extras. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry just let others do the job.
+
* Copy&paste this to the comment box.
 +
* Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
 +
* Vote up if there were no FAILs. If there was even one FAIL, vote down.
 +
* UI usability issues cannot be used as a reason for vote down.
 +
* Always test functionality - that is, run the program and try if it works as it should.
-
Please check the points of this list. Don't rush to give thumbs up to an application before it is tested. But don't try to delay the release of an app with minor bugs or enhancement proposals separate from this discrete QA checklist.
+
imaginary example:
 +
<pre>
 +
1. [x] Bug database exist.
 +
2. [ ] Licensing ok.
 +
3. [ ] No illegal/dubious content.
 +
4. [x] Working provided features.
 +
FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123).  
 +
FAIL: When exporting file the program crashes (see bug: http://url/456)
 +
5. [ ] No missing announced features.
 +
6. [x] Optification ok.
 +
FAIL: the package uses 1512kb from root.
 +
7. [ ] No performance problems.
 +
8. [ ] No power management issues.
 +
9. [ ] No known security risks.
-
=== Blockers ===
+
Additional comments:
 +
I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678
 +
</pre>
-
If one of these criteria is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.
+
==== Blockers ====
-
==== Lack of bug reporting database ====
+
A "blocker" is any item not met in the QA checklist.  If any one of the criteria in the QA checklist is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.
-
http://bugs.maemo.org/ is the preferred [[Bugs:Adding_Extra_products|option]]. Otherwise it needs to be [http://talk.maemo.org/showthread.php?t=31019&highlight=xsbc-bugtracker identified] in the http://maemo.org/packages/ page.
+
-
Developers: filling the ''XSBC-Bugtracker'' field in your packages address this problem (((FIXME: link to documentation?))). Check [[Bugs:Adding_Extra_products|this wiki page]] to get your bugtracker at bugs.maemo.org.
+
#  Lack of bug reporting database
 +
#: It needs to be [[Packaging#Bugtracker location|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.
 +
#: '''A missing bugtracker is NOT a blocker''' as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.
 +
#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: <nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki> Resolutions for bugtracking are found at [[Packaging#Bugtracker_location]]''
 +
# Licensing and legal issues
 +
#: Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.
 +
# Illegal or dubious content
 +
#: To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.
 +
# Working features, broken functionality or reproducible crashes
 +
#: There is some functionality that does not work or produces a crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.
 +
# Missing announced features
 +
#:Core features advertised through UI or product page don't work or are missing.
 +
# Optification
 +
#: '''''Test using the ''df'' command:'''''
 +
#: In Xterminal, run <code>df -h | grep -e rootfs -e mmcblk0p2</code> before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
 +
#: '''''Test using the ''dpkg'' command:'''''
 +
#: After installation of the package being tested, run in Xterminal <code>dpkg -L packagename</code> It will show all the files installed by the package.  Inspect the result to confirm that some of the files are placed in the /opt or MyDocs directories.
 +
#: Please add those packages which fail optification to the list at [[Opt_Problem/Non-Optified_packages]].
 +
#: ''Developers: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem.  If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker.  Also see [[Opt Problem]].''
 +
# System performance compromised
 +
#: Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.
 +
# Power management issues
 +
#: Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging should not be compromised.
 +
#: If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.
 +
# Security risks
 +
#: The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.
-
==== Legal issues ====
+
====Non-blockers====
-
Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.
+
-
==== Missing announced features ====
+
The extras-testing QA process should not be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.
-
Core features advertised through UI or product page don't work or are missing.
+
-
==== Illegal or dubious content ====
+
* Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
-
For the avoidance of doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.  
+
* Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.
-
==== Reproducible crashes ====
+
====Warnings====
-
You can provide a list of steps that lead to an application crash.
+
-
==== System performance compromised ====
+
If one of these warning criteria are not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.
-
Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.
+
-
==== Too much memory used in root partition ====
+
* Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
-
The application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more.
+
* Application icon missing or not visible in the device. This is only optional for command-line applications.
-
dpkg -L packagename shows all files installed by a package. Also see [[Opt Problem]].
+
Testers with good technical knowledge are invited to look deeper in the [http://maemo.org/maemo_release_documentation/maemo4.1.x/node16.html Maemo Quality considerations] (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.
-
==== Power management issues ====
+
===Command Line applications===
-
Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging shouldn't be compromised.
+
-
If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.
+
Command line applications are a special case, so two extra checks should be performed, these are blockers:
-
==== Security risks ====
+
* The application uses the [[Extras-testing/Command_line_applications | CLI icon]] in the application manager.
-
The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.
+
* The application [[Extras-testing/Command_line_applications#Description|description]] clearly says that the application only runs from the command line.
-
===Warnings===
+
==Testing software==
-
If one of these criteria is not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.
+
-
* Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
+
===Power use===
-
* Application icon missing or not visible in the device. This is only optional for command-line applications.
+
-
Testers with good technical knowledge are invited to look deeper in the [http://maemo.org/maemo_release_documentation/maemo4.1.x/node16.html Maemo Quality considerations] (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.
+
The [[Nokia N900|N900]] has extensive powersaving technologies. It can achieve in some circumstances very good battery life. For example, with a wifi AP that supports powersaving properly, connected over ssh and to a GSM network, battery life can be around 5 days idle.
-
===Non-blockers===
+
Leaving an application inactive (accidentally or not) is not uncommon, and the application must not use significant power when doing nothing with the screen locked.
-
The extras-testing QA process shouldn't be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.
+
-
* Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
+
This is a list of some applications that may be of use to ensure that the application you are testing does not damage suspend time.  
-
* Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.
+
 
 +
* Powertop - is available in [http://maemo.org/packages/package_instance/view/fremantle_extras-devel_non-free_armel/powertop/1.0/ the extras-devel] repository.
 +
 
 +
(for best results, run with screen locked, over ssh, with nothing plugged into USB, as this minimises normal system activity)
 +
 
 +
C#      | Ratio  | Avg/dura | Frequency | Ratio
 +
--------+--------+----------+-----------+--------+
 +
      C0 |  57.8% |          |  600 MHz |  0.0% |
 +
      C1 |  22.4% |    0.9ms |  550 MHz |  0.0% |
 +
      C2 |  19.8% |    3.3ms |  500 MHz | 100.0% |
 +
      C3 |  0.0% |          |  250 MHz |  0.0% |
 +
      C4 |  0.0% |          |
 +
 
 +
This is an example of an app that when idle does not powersave at all.
 +
The CPU is continually active.
 +
 
 +
The 'C0' state is the state where the CPU is actively running - 60% of the time, and it is spending 100% of its time when running at 500 MHz.
 +
 
 +
The battery may last 10-12 hours.
 +
 
 +
C1 through C4 are various powersaving states - with C4 being the deepest, it is spending at most 3 ms - 1/300th of a second in shallow powersaving.
 +
 
 +
In comparison - the same phone when idle - the stats look like:
 +
 
 +
C#      | Ratio  | Avg/dura | Frequency | Ratio
 +
--------+--------+----------+-----------+--------+
 +
      C0 |  1.0% |          |  600 MHz |  0.0% |
 +
      C1 |  0.0% |    0.2ms |  550 MHz |  0.0% |
 +
      C2 |  3.0% |    5.9ms |  500 MHz |  0.0% |
 +
      C3 |  17.6% |  252.0ms |  250 MHz | 100.0% |
 +
      C4 |  78.3% |  810.5ms |
 +
 
 +
The phone spends most of its time in the deepest powersaving state, is only actually running 1% of the time, and when it does, is running at 250 MHz.
 +
 
 +
The battery in this case will last over 110 hours.
 +
 
 +
===top===
 +
 
 +
Top or [http://maemo.org/packages/view/htop/ htop in extras] can also be very handy. If when idle, the package you're testing is using 50% CPU - it is never going to be very good for battery life!
 +
 
 +
Cumulative time can be of use here - if every time you have looked, it has been using .4% CPU, but over half an hour it has used 10 min of CPU, something is wrong.
 +
 
 +
[[Category:Quality Assurance]]

Latest revision as of 10:53, 28 May 2012

Image:Ambox_notice.png
 The software hosted in extras-testing is not ready for normal users!
PLEASE use it only for testing purposes. Be ready to file proper bug reports instead of posting complaints.

Potential problems: crashes, battery drain, poor system performance, full disk space & more - SERIOUSLY!

Backing up your data is recommended. In case of trouble you might need to re-flash your device.


There are several elements to be considered by developers before promoting software to extras-testing. The same criteria are useful for beta testers before approving applications to go to Extras. Please take your time looking at them. Don't evaluate an application lightly! If you are busy or in a hurry just let others do the job.

Please check the points of this list. Don't rush to give thumbs up to an application before it is tested. But don't try to delay the release of an app with minor bugs or enhancement proposals separate from this discrete QA checklist.

Contents

[edit] QA Improvements

There is a ongoing QA (Quality Assurance) improvements discussion in talk.maemo.org (see here and here) and the mailing list. One task is to establish a testing squad. The results can be found at this wiki page: QA improvements

[edit] QA checklist

Use this checklist until the extras-testing approval interface is fixed. It helps to see what you have tested and what other testers should still test. This checklist condenses the blockers explained below, read the explanation below before you mark a test as done. 

1. [ ] Bug database exist.
2. [ ] Licensing ok.
3. [ ] No illegal/dubious content.
4. [ ] Working provided features.
5. [ ] No missing announced features.
6. [ ] Optification ok.
7. [ ] No performance problems.
8. [ ] No power management issues.
9. [ ] No known security risks.

Additional comments:

Note: Only paste this template if there is some problem with the application or if you have not tested some of the points above, let's not spam the developers.

  • Copy&paste this to the comment box.
  • Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
  • Vote up if there were no FAILs. If there was even one FAIL, vote down.
  • UI usability issues cannot be used as a reason for vote down.
  • Always test functionality - that is, run the program and try if it works as it should.

imaginary example: 

1. [x] Bug database exist.
2. [ ] Licensing ok.
3. [ ] No illegal/dubious content.
4. [x] Working provided features.
 FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). 
 FAIL: When exporting file the program crashes (see bug: http://url/456)
5. [ ] No missing announced features.
6. [x] Optification ok.
 FAIL: the package uses 1512kb from root.
7. [ ] No performance problems.
8. [ ] No power management issues.
9. [ ] No known security risks.

Additional comments: 
I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678

[edit] Blockers

A "blocker" is any item not met in the QA checklist. If any one of the criteria in the QA checklist is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.

  1. Lack of bug reporting database
    It needs to be identified in the http://maemo.org/packages/ page.
    http://bugs.maemo.org/ is the preferred option.
    A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.
    Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: http://maemo.org/packages/view/<package>(/<version>) Resolutions for bugtracking are found at Packaging#Bugtracker_location
  2. Licensing and legal issues
    Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.
  3. Illegal or dubious content
    To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.
  4. Working features, broken functionality or reproducible crashes
    There is some functionality that does not work or produces a crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.
  5. Missing announced features
    Core features advertised through UI or product page don't work or are missing.
  6. Optification
    Test using the df command:
    In Xterminal, run df -h | grep -e rootfs -e mmcblk0p2 before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
    Test using the dpkg command:
    After installation of the package being tested, run in Xterminal dpkg -L packagename It will show all the files installed by the package. Inspect the result to confirm that some of the files are placed in the /opt or MyDocs directories.
    Please add those packages which fail optification to the list at Opt_Problem/Non-Optified_packages.
    Developers: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem. If the application or its dependencies ignore the recommendation to use the eMMC to install as much files as possible, filling the root partition with 500kb or more, this can be considered a blocker. Also see Opt Problem.
  7. System performance compromised
    Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.
  8. Power management issues
    Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging should not be compromised.
    If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.
  9. Security risks
    The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to report it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.

[edit] Non-blockers

The extras-testing QA process should not be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.

  • Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
  • Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.

[edit] Warnings

If one of these warning criteria are not met, the developers will be asked to fix them urgently. Still, the app can make it slowly to Extras unless a blocker is raised.

  • Details at http://maemo.org/packages/ are missing or incomplete: summary, URL to project, updates info.
  • Application icon missing or not visible in the device. This is only optional for command-line applications.

Testers with good technical knowledge are invited to look deeper in the Maemo Quality considerations (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.

[edit] Command Line applications

Command line applications are a special case, so two extra checks should be performed, these are blockers:

  • The application uses the CLI icon in the application manager.
  • The application description clearly says that the application only runs from the command line.

[edit] Testing software

[edit] Power use

The N900 has extensive powersaving technologies. It can achieve in some circumstances very good battery life. For example, with a wifi AP that supports powersaving properly, connected over ssh and to a GSM network, battery life can be around 5 days idle.

Leaving an application inactive (accidentally or not) is not uncommon, and the application must not use significant power when doing nothing with the screen locked.

This is a list of some applications that may be of use to ensure that the application you are testing does not damage suspend time.

(for best results, run with screen locked, over ssh, with nothing plugged into USB, as this minimises normal system activity) 

C#      | Ratio  | Avg/dura | Frequency | Ratio
--------+--------+----------+-----------+--------+
     C0 |  57.8% |          |   600 MHz |   0.0% |
     C1 |  22.4% |    0.9ms |   550 MHz |   0.0% |
     C2 |  19.8% |    3.3ms |   500 MHz | 100.0% |
     C3 |   0.0% |          |   250 MHz |   0.0% |
     C4 |   0.0% |          |

This is an example of an app that when idle does not powersave at all. The CPU is continually active.

The 'C0' state is the state where the CPU is actively running - 60% of the time, and it is spending 100% of its time when running at 500 MHz.

The battery may last 10-12 hours.

C1 through C4 are various powersaving states - with C4 being the deepest, it is spending at most 3 ms - 1/300th of a second in shallow powersaving.

In comparison - the same phone when idle - the stats look like: 

C#      | Ratio  | Avg/dura | Frequency | Ratio
--------+--------+----------+-----------+--------+
     C0 |   1.0% |          |   600 MHz |   0.0% |
     C1 |   0.0% |    0.2ms |   550 MHz |   0.0% |
     C2 |   3.0% |    5.9ms |   500 MHz |   0.0% |
     C3 |  17.6% |  252.0ms |   250 MHz | 100.0% |
     C4 |  78.3% |  810.5ms |

The phone spends most of its time in the deepest powersaving state, is only actually running 1% of the time, and when it does, is running at 250 MHz.

The battery in this case will last over 110 hours.

[edit] top

Top or htop in extras can also be very handy. If when idle, the package you're testing is using 50% CPU - it is never going to be very good for battery life!

Cumulative time can be of use here - if every time you have looked, it has been using .4% CPU, but over half an hour it has used 10 min of CPU, something is wrong.

Retrieved from "http://wiki.maemo.org/Extras-testing/QA_checklist"