Editing Documentation/Maemo 5 Developer Guide/Kernel and Debugging Guide/Maemo Debugging Guide

Warning: You are not logged in. Your IP address will be recorded in this page's edit history.

Warning: This page is 60 kilobytes long; some browsers may have problems editing pages approaching or longer than 32kb. Please consider breaking the page into smaller sections.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Learn more about Contributing to the wiki.

= Debugging =

= Introduction =

The following code examples are used in this chapter:

* [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/gdb_example gdb_example]
* [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/maemopad MaemoPad]
* [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/hello-world-app/ hello-world-app]
* [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/valgrind_example/ valgrind_example]

= Maemo Debugging Guide =

This section is targeted at beginner-level Maemo developers who need to know how to perform debugging in the Maemo environment.

This section covers the two basic debugging tools available in the Maemo environment and shows how to use them. The tools are:

* [[Documentation/devtools/maemo5/gdb|gdb]] - The Gnu Project Debugger. General tool for various debugging needs.
* [[Documentation/devtools/maemo5/valgrind|valgrind]] - Debugger and profiler. Valgrind works only in the X86 environment under Scratchbox, so this tool cannot be used on the device itself.

This section assumes that the developer already knows how to:

* develop software in the Linux environment using the C language
* install software to the tablet device
* [[root access|gain root access]] to the device
* install ssh to the device
* configure repositories in the <code>/etc/apt/sources.list</code> file
* set up USB networking between a Linux PC and the Tablet (the device can also be used over a local WLAN connection instead)
* work with the Scratchbox environment and Scratchbox targets

== Pre-Requisites ==

To follow the debugging examples you need to have the following:

* [[Documentation/Maemo_5_Final_SDK_Installation|Maemo SDK installed on a Linux PC]]
* Nokia Internet Tablet device running [[Open development/Maemo roadmap/Fremantle|Maemo 5]].
* USB cable to connect the device with the Linux PC
* Internet access both for the tablet and for the Linux PC
* USB networking (or WLAN) set up between the Linux PC and the device
* root login access to the device over [[SSH]]
* package <code>maemo-sdk-debug</code> installed
* [[terminal|xterm]] installed on the device
* SSH software installed on the device

== General Notes on Debugging ==

=== Debugging Issues on ARM Architecture ===

When debugging the ARM architecture, pay attention to the following issues:

* To make backtraces work properly on the ARM side, the dbg packages need to be installed for the libraries the application is using. Profiling and debugging (gdb) tools require the code to have either framepointers or debugging symbols to unwind the stack. This is necessary for showing backtraces or call graphs.
* C language functions with the <code>__attribute__((__noreturn__))</code> statement need to be compiled with the gcc option: <code>-mapcs-frame</code> otherwise the compiler excludes their function frame. Maemo Glib and C-library noreturn functions like <code>abort()</code> already have this. Without a function frame, gdb cannot backtrace through "noreturn" functions.
* In addition, for gdb to be able to display the correct function names during debugging, it also needs to have access to the debug symbols. Without them, gdb shows the preceding exported function name for a given address.

=== Debugging Issues in Scratchbox ===

It is recommended to use the native gdb in the target, not the Scratchbox host gdb.

When debugging threads in an application, gdb needs to be linked against the same thread library that the application is using. For this reason, the gdb provided in Scratchbox is not suitable for threads debugging, and the native gdb must be used instead. If you see errors such as:

warning: Cannot initialize thread debugging library: unknown thread_db error '22'

in gdb output, use the native gdb. For instructions on how to start using the native gdb, see the next section.

== Using Gdb Debugger ==

The Gnu Project Debugger, or gdb for short, is a general purpose debugger that can be used for various debugging purposes.

This section does not explain how to use the gdb debugger itself, nor does it explain the specific commands in gdb to perform particular actions. There are other [http://www.gnu.org/software/gdb/documentation/ tutorials and documentation] readily available on the Internet for that purpose. This section focuses on explaining how to set up and perform the basic debugging steps with [[Documentation/devtools/maemo5/gdb|gdb in the Maemo environment]].

=== Setting up Environment ===

Both the Internet Tablet device, described in section [[Documentation/Maemo PC Connectivity Tutorial|Maemo PC Connectivity]], and the Scratchbox environment, described in section [[../../Development Environment/Maemo SDK|Maemo SDK]], need to be set up.

=== Preparing Scratchbox Environment for Debugging ===

If Maemo SDK is not yet installed, it must be installed before continuing. After [[Documentation/Maemo 5 Final SDK Installation|installing the Maemo SDK]], the default target names are FREMANTLE_ARMEL and FREMANTLE_X86. These are the names used in this example.

Scratchbox provides a version of gdb, which is used when executing <code>gdb</code>. It is also possible to use the host system (non-Scratchbox) version of gdb, by executing <code>native-gdb</code>.

Next, install gdb to Scratchbox from the Maemo repositories.
 
 [sbox-FREMANTLE_x86: ~] > fakeroot apt-get install maemo-sdk-debug

Now two gdb programs are installed. The one used can be checked with the following:
 
 [sbox-FREMANTLE_x86: ~] > which gdb
 /targets/links/arch_tools/bin/gdb

Start briefly both gdb debuggers to see that they start properly in the Scratchbox environment. First just run gdb:
 
 [sbox-FREMANTLE_x86: ~] > gdb
 GNU gdb 6.8
 Copyright (C) 2008 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "i686-pc-linux-gnu".
 (gdb) q
 [sbox-FREMANTLE_x86: ~] >

Then run the Maemo version of gdb:
 
 [sbox-FREMANTLE_x86: ~] > native-gdb
 GNU gdb (GDB) 6.8.50.20090417-debian
 Copyright (C) 2009 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "i486-linux-gnu".
 For bug reporting instructions, please see:
 &lt;http://www.gnu.org/software/gdb/bugs/&gt;.
 (gdb) q
 [sbox-FREMANTLE_x86: ~] >

To quit from gdb, type 'q' and press ENTER.

The native-gdb is used here because it allows threads to be debugged. The native-gdb should always be run instead of the Scratchbox version.

=== Preparing Internet Tablet for Debugging ===

{{ambox|text=Because additional software is going to be installed and debugging is performed in the real device that is running the official Sales Image, take a full back-up of the device and any important files before continuing.}}

Gdb needs to be installed to the Internet Tablet device. Install [[SSH|SSH server]] on the device before continuing. OpenSSH server can be installed from maemo [[extras]] repository or from the SDK repository like the gdb below:

# [[Terminal#Open the terminal|Open a console]].
# Gain [[root access]] to the device.
# Edit the <code>/etc/apt/sources.list</code> file on the device so that it contains the lines: 
 
 deb http://repository.maemo.org/ fremantle/sdk free non-free
 deb http://repository.maemo.org/ fremantle/tools free non-free

Important: Performing device software updates from the Maemo SDK repositories (for example, 'apt-get upgrade') is not recommended. The reason for this is that there can be some software packages in the SDK repositories that are so-called ''sdk variants''. They can create problems if installed directly to the actual device. In this example, only gdb software is used from the repository.
# Perform an <code>apt-get update</code> on the device. The update command refreshes the package list database on the device.
# Perform an <code>apt-get install gdb</code>
# Disable the SDK repository by removing or commenting out the lines from your  <code>/etc/apt/sources.list</code> file

You should now have gdb and gdbserver (included in the gdb package) installed to the device.

=== Debugging Use Cases with Gdb ===

==== Debugging Command Line Application in Scratchbox X86 Environment with Gdb ====

One of the most common debugging case is performing debugging in the Scratchbox X86 environment.

By now, the gdb is installed in the FREMANTLE_X86 target. Next, download the [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/gdb_example/ gdb_example] sources.

The example apps are:

* <code>gdb_example.c</code> is a very simple C application that has some functions that call each other in a row. This is used here to demonstrate how to get backtraces.
* <code>gdb_example2.c</code> is a simple variant of the <code>gdb_example.c</code> that has some additional <code>sleep()</code> calls. This is used to demonstrate simple core dump debugging.

Next, compile the small <code>gdb_example.c</code> file as shown below, and start the gdb debugger. This simple example shows how to set breakpoints, and how to get a backtrace from the program. Backtrace tells what functions have been called and what parameters have been used.

<ol>
<li>Compile the <code>gdb_example.c</code> application with the <code>-g</code> option like this:

<pre> 
[sbox-FREMANTLE_x86: ~/src/gdb_example] > gcc gdb_example.c -o gdb_example -g -O2 -Wall
</pre>

It is recommended to compile debug binaries with options -g -O2 -Wall, although only -g is necessary as that leaves debugging information in the binaries. Sometimes optimizations (-O2) might cause trouble for gdb, leaving it out can help in that case. Without optimizations the size and execution time of the binaries might increase a lot. Generally, production binaries should be compiled with -O2 -Wall. However, production binaries are typically Debian-packaged and should be compiled with -g -O2 -Wall, stripping the debug symbols into a separate package to facilitate debugging. See section [[#Making a Debian Debug Package]].
</li>

<li>Next, start gdb with the <code>gdb_example</code> application as a parameter.

<pre>
[sbox-FREMANTLE_x86: ~/src/gdb_example] &gt; native-gdb gdb_example
GNU gdb (GDB) 6.8.50.20090417-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
&lt;http://www.gnu.org/software/gdb/bugs/&gt;...
</pre></li>

<li>Set the breakpoint (br) to function <code>example_3</code>

<pre> 
(gdb) br example_3
Breakpoint 1 at 0x8048424: file gdb_example.c, line 50.
</pre>
</li>

<li>
Run the application under gdb giving a command line parameter foobar to the application.

<pre>
(gdb) run foobar
Starting program: /home/user/src/gdb_example/gdb_example foobar
Now in main().
Now in example_1(2) function.
Now in example_2(parameter here) function.

Breakpoint 1, example_3 () at gdb_example.c:50
50        printf("Now in example_3()\n");
</pre></li>
</ol>

The above shows that the running of the application stopped in function <code>example_3</code>. This happened because the breakpoint (<code>br example_3</code>) was set. Note that the line numbers in the examples may vary if there have been changes to the example sources.

Now the backtrace (bt) can be listed from application to see what functions have been called. The list goes from recent to older. The oldest function was naturally the <code>main()</code> function at the end of the list. It shows also the parameters to the called functions.

(gdb) bt
 #0 example_3 () at gdb_example.c:50
 #1 0x08048417 in example_2 (a=0x8048548 "parameter here") at gdb_example.c:41
 #2 0x080483f1 in example_1 (x=2) at gdb_example.c:32
 #3 0x080483b3 in main (argc=2, argv=0xbffff5d4) at gdb_example.c:20
 (gdb)

It can be convenient to see what the source code is for a line mentioned in the output:
 
 (gdb) list 41
 36	 *
 37	 */
 38	int example_2(char* a)
 39	{
 40	  printf("Now in example_2(%s) function.\n",a);
 41	  example_3();
 42	  return 0;
 43	}
 44	/*
 45	 *
 (gdb)

Also the values of the variables can be inspected:
 
 (gdb) br example_2
 Breakpoint 2 at 0x80483ff: file gdb_example.c, line 40.
 (gdb) run
 The program being debugged has been started already.
 Start it from the beginning? (y or n) y
 Starting program: /home/user/src/gdb_example
 Now in main().
 Now in example_1(2) function.
 
 Breakpoint 2, example_2 (a=0x8048548 "parameter here") at gdb_example.c:40
 40	  printf("Now in example_2(%s) function.\n",a);

To see the value of variable 'a', just type:
 
 (gdb) print a
 $1 = 0x8048548 "parameter here"
 (gdb)

Essentially, debugging with gdb in the Scratchbox X86 target is similar to debugging with gdb in any Linux host. In this example, only a small subset of gdb's functionality has been used.

==== Debugging Command Line Application in Internet Tablet Device ====

It is possible to debug an application in the Internet tablet device itself using gdb. Before starting, log in to the device and install (if you have not done so yet) the gdb debugger to the device:
 
 /home/user # apt-get install gdb
 ... etc ...
 /home/user #

Here it is assumed that <code>ssh</code> and <code>xterm</code> are already installed in the tablet and that it is possible to log in on the device using ssh from a Linux PC. In addition, the maemo repository entries should be set in the <code>/etc/apt/sources.list</code> file as explained previously.

Debugging with the gdb debugger in the device is similar to using gdb in a normal Linux PC environment. The limitations are mostly related to the available free RAM, meaning that in the worst case, memory might run out while trying to debug an application in the device.

This example follows the basic logic of the first example, but this time is performed in the device.

<ol>
<li> Compile the <code>gdb_example.c</code> application in the Scratchbox for armel architecture. 
<pre> 
[sbox-FREMANTLE_X86: ~] > sb-conf select FREMANTLE_ARMEL
Hangup
Shell restarting...
[sbox-FREMANTLE_ARMEL: ~] > pwd
/home/user
[sbox-FREMANTLE_ARMEL: ~] > cd src/gdb_example
[sbox-FREMANTLE_ARMEL: ~/src/gdb_example] > gcc gdb_example.c -o gdb_example -g -O2 -Wall
[sbox-FREMANTLE_ARMEL: ~/src/gdb_example] > file gdb_example
gdb_example: ELF 32-bit LSB executable, ARM, version 1 (SYSV), for GNU/Linux 2.6.14,
dynamically linked (uses shared libs), not stripped
[sbox-FREMANTLE_ARMEL: ~/src/gdb_example] >
</pre></li>

<li> Copy the armel version of the <code>gdb_example</code> to the tablet. You need to have the <code>sshd</code> daemon up and running in the device before you can copy files with <code>scp</code>. The IP address is an example, and your device IP address can be different.

<pre> 
[sbox-FREMANTLE_ARMEL: ~/src/gdb_example] > scp gdb_example.c \
gdb_example user@192.168.2.15:
user@192.168.2.15's password:  ...........
gdb_example.c                             100%  828     0.8KB/s   00:00
gdb_example                               100% 7118     7.0KB/s   00:00
</pre></li>

<li> Log in on the device with ssh with username user.

<pre> 
[sbox-FREMANTLE_ARMEL: ~/src/gdb_example] > ssh root@192.168.2.15
root@192.168.2.15's password: ..........
 
BusyBox v1.10.2 (Debian 3:1.10.2.legal-1osso26+0m5) built-in shell (ash)
Enter 'help' for a list of built-in commands.
 
~# cd ~user
</pre></li>

<li> Start the gdb debugger with the <code>gdb_example</code> application.

<pre> 
/home/user# gdb ./gdb_example
... snip ...
(gdb)

</pre></li>
</ol>

You should now be able to debug this small example application in a similar way as in the previous example.

==== Debugging Core Files in Device ====

This section explains how to debug core files in the Internet Tablet. The default location for core dumps is <code>/home/user/MyDocs/core-dumps</code>. There can be a limit for core dump size. You can check this by executing 
 
 Nokia-NXX-31-10:/home/user# ulimit -c
 unlimited

If the limit is too low, set it bigger, for example:

Nokia-NXX-31-10:/home/user# ulimit -c unlimited
 unlimited

Also, if you don't have the aforementioned <code>core-dumps</code> directory created and <code>sp-rich-core</code> installed, cores may not be created. The package can be installed by

Nokia-NXX-31-10:/home/user# apt-get install sp-rich-core

If needed, create the folder

~ $ mkdir MyDocs/core-dumps/

Next, the small example application <code>gdb_example2</code> will be used to demonstrate how to debug the core file.
<ol>
<li> Compile the <code>gdb_example2</code> in the Scratchbox armel target, and just copy the file to the device using <code>scp</code>. Then start the <code>gdb_example2</code> in the device like this:

<pre> 
Nokia-NXX-31-10:/home/user # ./gdb_example2 &amp;
Now in main().
Now in example_1(1) function.
</pre></li>

<li> The <code>gdb_example2</code> is now running in the background, and it starts to dump its output to the screen. There are some <code>sleep()</code> calls in <code>gdb_example2</code>, so that you have time to kill it with the SIGSEGV signal. Now just make it to generate a core dump. Assuming that the process is referred to as %1, use the kill command as below and press the ENTER key a couple of times:

<pre> 
Nokia-NXX-31-10:/home/user # kill -SIGSEGV %1
Nokia-NXX-31-10:/home/user #
[1] + Segmentation fault (core dumped) ./gdb_example2
</pre></li>

<li> You now have a compressed core dump data file under the <code>/home/user/MyDocs/core-dumps/</code> directory. Its name includes the name of the file and end with the PID number of the <code>gdb_example2</code> program. Check that you got it (the number shown below, 1581, is naturally different in your environment):

<pre> 
/home/user # ls -l MyDocs/core-dumps/
-rw-r--r--    1 user     root        52048 Aug  7 12:08 gdb_example2-782
9-11-1581.rcore.lzo
</pre></li>

<li> Extract the compressed data before you can use it with gdb. Install the sp-rich-core-postproc package and run the extract command which creates a new directory (given as a second parameter) where to extract data:

<pre> 
Nokia-NXX-31-10:/home/user # apt-get install sp-rich-core-postproc
... snip ...
Nokia-NXX-31-10:/home/user # rich-core-extract \
/media/mmc1/core-dumps/gdb_example2-7829-11-1581.rcore.lzo coredir
</pre>

The newly-created coredir includes a lot of information in different files about the system. The file to pass for the gdb is named <code>coredump</code>.</li>

<li> The <code>gdb_example2</code> is linked against the <code>libc</code> library. If you want to be able to resolve symbols also for the library during debugging, install <code>libc6-dbg</code> package in the device. ''The same rule applies to other libraries that your application might be linked against.''  See the further notes about the DBG packages in this material. Now install the <code>libc6-dbg</code> package to get symbols for the library.

<pre> 
Nokia-NXX-31-10:/home/user #  apt-get install libc6-dbg
Reading Package Lists... Done
Building Dependency Tree... Done
The following NEW packages will be installed:
  libc6-dbg

.... snip ...

/home/user #
</pre></li>

<li> Now you can debug the core file together with the <code>gdb_example2</code> binary that you compiled with the -g flag. Try and see where the execution of the <code>gdb_example2</code> was when you used the <code>-SIGSEGV</code> signal. Start gdb and give <code>gdb_example2</code> as the first parameter, and the core file as the second parameter:

<pre>
Nokia-NXX-31-10:/home/user # gdb ./gdb_example2 MyDocs/core-dumps/coredir/coredump
GNU gdb (GDB) 6.8.50.20090417-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabi".
For bug reporting instructions, please see:
&lt;http://www.gnu.org/software/gdb/bugs/&gt;...
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debu
g/lib/libc-2.5.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.3...Reading symbols from /usr/lib/
debug/lib/ld-2.5.so...done.
done.
Loaded symbols for /lib/ld-linux.so.3
Core was generated by `./gdb_example2'.
Program terminated with signal 11, Segmentation fault.
#0  0x400be97c in nanosleep () from /lib/libc.so.6
0x400be97c &lt;nanosleep+28&gt;:	mov	r7, r12
(gdb)
</pre>

The example above shows that now gdb is using debug symbols from <code>/usr/lib/debug/lib/libc-2.5.so</code>. Had the libc6-dbg package not been installed, gdb would not have information available about the libraries' debug symbols. Now, gdb is waiting for a command, so just give the bt (backtrace) command. You should see something similar to this:

<pre>
(gdb) bt
#0  0x400be97c in nanosleep () from /lib/libc.so.6
#1  0x400be73c in sleep () from /lib/libc.so.6
#2  0x0000844c in example_1 (x=1) at gdb_example2.c:33
#3  0x00008408 in main (argc=1, argv=0xbe8b2674) at gdb_example2.c:21
</pre>

Note that for this simple example, the available <code>libc6-dbg</code> package was installed before starting to debug the <code>gdb_example2</code> application. Backtraces work only if debug packages are installed also for the libraries through which the backtrace goes.

Depending on at what point the <code>kill -SIGSEGV</code> command was given, the output may be different. In this example, the process was hit when it was calling the sleep function inside the <code>example_1</code> function in file <code>gdb_example2.c</code>. It can also be seen that the <code>sleep()</code> function has further called the <code>nanosleep()</code> function, and that is when it got the <code>-SIGSEGV</code> signal (see the note above).
</li>
</ol>

==== Debugging Core File from Device Inside Scratchbox ====

Because the binaries and libraries in the device are prelinked, successful debugging inside the Scratchbox environment using the programs core file (from the device) would require that:

* you copy the relevant libraries (the ones that the application is using) from the device to Scratchbox. Otherwise, addresses in the prelinked and unprelinked libraries would not match, and gdb backtraces would not load the library.

* If the core file debugging is performed in the Scratchbox ARMEL environment, the native gdb program needs to be used instead of the one provided by Scratchbox. See the previous section on how to set the native gdb as the default one.

After copying the <code>/lib/libc6</code> library from the device and setting the native gdb to be used, the application can be debugged normally. Keep in mind that generally the Scratchbox tools override the target tools.

==== Debugging UI Applications in Scratchbox X86 ====

Many maemo applications use the graphical UI, and debugging these applications differs slightly from debugging simple command line applications.

Here maemopad will be used as an example application to debug in the X86 target with gdb.

If the Scratchbox environment is set up correctly as explained above, these steps should be easy to follow to perform UI debugging with the maemopad application.
<ol>
<li> Activate the X86 target, go to the testing directory and download the source package of maemopad.

<pre> 
[sbox-FREMANTLE_ARMEL] > sb-conf select FREMANTLE_X86
[sbox-FREMANTLE_X86] > cd ~/src/
[sbox-FREMANTLE_X86 ~/src] > mkdir maemopad
[sbox-FREMANTLE_X86 ~/src/maemopad] > cd maemopad
[sbox-FREMANTLE_X86 ~/src/maemopad] > apt-get source maemopad
Reading package lists... Done
Building dependency tree... Done
Need to get 347kB of source archives.
Get:1 http://repository.maemo.org fremantle/sdk/free maemopad 2.5 (dsc) [465B]
Get:2 http://repository.maemo.org fremantle/sdk/free maemopad 2.5 (tar) [346kB]
Fetched 347kB in 1s (231kB/s)
dpkg-source: warning: extracting unsigned source package (./maemopad_2.5.dsc)
dpkg-source: extracting maemopad in maemopad-2.5
dpkg-source: unpacking maemopad_2.5.tar.gz
</pre></li>

<li> Check that you have the correct files.

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad] > ls -l
total 352
drwxrwxr-x  6 maemo maemo   4096 Jul 28 14:09 maemopad-2.5
-rw-rw-r--  1 maemo maemo    465 Jul  6 16:02 maemopad_2.5.dsc
-rw-rw-r--  1 maemo maemo 346129 Jul  6 16:02 maemopad_2.5.tar.gz
</pre></li>

<li> Go to the <code>maemopad-2.5</code> source directory, and set the <code>DEB_BUILD_OPTIONS</code> environment variable, so that the generated binaries are not stripped. Then build the maemopad package with <code>dpkg-buildpackage</code> command as shown here:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad] > cd maemopad-2.5
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5] > export DEB_BUILD_OPTIONS=debug,\
nostrip
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5] > dpkg-buildpackage -rfakeroot -d
dpkg-buildpackage: source package is maemopad
dpkg-buildpackage: source version is 2.5
dpkg-buildpackage: source changed by Soumya Bijjal &lt;soumya.3.bijjal@nokia.com&gt;
dpkg-buildpackage: host architecture i386
dpkg-buildpackage: source version without epoch 2.3
.... etc ....
</pre>

In this example, the standard <code>export DEB_BUILD_OPTIONS=debug,nostrip</code> environment variable is used, but there might be source packages that do not support these debug,nostrip options. In that case, one must make sure that the source is compiled with <code>-g</code> flag (usually this option can be added to the <code>CFLAGS</code> variable in the <code>debian/rules</code> file), and that the produced binaries will not be stripped. In the long run, it is better to modify the source package to generate a separate debug symbol (<code>-dbg</code>) package. This requires modifying both the <code>debian/rules</code> and <code>debian/control</code> files. In fact, maemopad already builds a separate debug package, but this example makes no use of it to demonstrate the simpler case.

<li> You should now have maemopad binaries generated so that they have the debug symbols in them. Check that you get a <code>not stripped</code> flag from the maemopad binary:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5] > file debian/maemopad/usr/bin/\
maemopad
debian/tmp/usr/bin/maemopad: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
for GNU/Linux 2.0.0, dynamically linked (uses shared libs), not stripped
</pre></li>

<li> You should now have a <code>maemopad_2.5_i386.deb</code> file in the <code>~/src/maemopad</code> directory. Check that you have it:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad] > ls -l
total 420
drwxrwxr-x  6 maemo maemo   4096 Jul 28 14:18 maemopad-2.5
-rw-r--r--  1 maemo maemo   3022 Jul 28 14:18 maemopad-dbg_2.5_i386.deb
-rw-rw-r--  1 maemo maemo    465 Jul 28 14:17 maemopad_2.5.dsc
-rw-rw-r--  1 maemo maemo 348814 Jul 28 14:17 maemopad_2.5.tar.gz
-rw-rw-r--  1 maemo maemo    807 Jul 28 14:18 maemopad_2.5_i386.changes
-rw-r--r--  1 maemo maemo  50242 Jul 28 14:18 maemopad_2.5_i386.deb
</pre></li>

<li> Install the newly compiled <code>maemopad_2.5_i386.deb</code> file inside the Scratchbox environment:

<pre>
[sbox-FREMANTLE_X86: ~/src/maemopad] > dpkg -i maemopad_2.5_i386.deb
... output from dpkg ...
</pre></li>

<li> Start the <code>Xephyr</code> server outside the Scratchbox:

<pre> 
Linux-PC$ Xephyr :2 -host-cursor -screen 800x480x16 -dpi 96 -ac -kb &amp;
</pre></li>

<li> Start the Application Framework from inside the Scratchbox X86 target:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.1] > export DISPLAY=:2
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.1] > af-sb-init.sh start
... lots of output from various programs ...
</pre>

You should now have the application framework up and running and the <code>Xephyr</code> window should contain the SDK UI.</li>

<li> Move to the source directory:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5] > cd src
</pre></li>

<li> In the src directory, you should have the files:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5/src] > ls -l
total 280
-rw-rw-r--  1 maemo maemo 25491 Jul 28 14:18 Makefile 
-rw-rw-r--  1 maemo maemo   491 Jul 28 14:09 Makefile.am
-rw-rw-r--  1 maemo maemo 23238 Jul 28 14:09 Makefile.in
-rwxrwxr-x  1 maemo maemo 83433 Jul 28 14:18 maemopad
-rw-rw-r--  1 maemo maemo 82928 Jul 28 14:18 maemopad-maemopad-window.o
-rw-rw-r--  1 maemo maemo 18564 Jul 28 14:18 maemopad-main.o
-rw-rw-r--  1 maemo maemo 20400 Jul 28 14:09 maemopad-window.c
-rw-rw-r--  1 maemo maemo  2391 Jul 28 14:09 maemopad-window.h
-rw-rw-r--  1 maemo maemo  2193 Jul 28 14:09 main.c
</pre></li>

<li> In this example, a debugging breakpoint will be set at the callback function <code>maemopad_window_on_menu_file_new()</code> located in the <code>maemopad-window.c</code> file. This function is called when the user clicks the New button from the maemopad menu. Set the debugging breakpoint like this:

<pre> 
[sbox-FREMANTLE_X86: ~/src/maemopad/maemopad-2.5/src] > run-standalone.sh \
gdb maemopad
(gdb) br maemopad_window_on_menu_file_new
Breakpoint 1 at 0x804b457: file maemopad-window.c, line 283.
</pre></li>

<li> Start the maemopad application from <code>gdb</code>.

<pre> 
(gdb) run
Starting program: /targets/FREMANTLE_X86/usr/bin/maemopad
[Thread debugging using libthread_db enabled]
</pre>

Note that for thread-debugging the native gdb program is used. If you need to debug threads in the application, you need to use the native gdb linked against the same thread library that your application is using. Remember that the Scratchbox <code>gdb</code> is not suitable for this purpose. To use the native <code>gdb</code>, <code>native-gdb</code> needs to be used, as mentioned earlier in this material.
</li>

<li> Now you should be able to see the maemopad application inside the Xephyr window, and you should be able to use it normally. Next, click the upper menu and select the item HELP. In your gdb terminal window you should now see :

<pre> 
Breakpoint 1, maemopad_window_on_menu_file_new (button=0x80cf0a8,
    data=0x8077018) at maemopad-window.c:283
283	  MaemopadWindow *self = MAEMOPAD_WINDOW (data);
</pre></li>

<li> The breakpoint that you set above is now reached, and execution of maemopad is stopped. The debugger waits for your command now. Try something simple, like using the list command to see where the execution of the application is going. You should get:

<pre>
(gdb) list
278	}
279
280	/* new */
281	static void maemopad_window_on_menu_file_new (GtkButton *button, gpointer data)
282	{
283	  MaemopadWindow *self = MAEMOPAD_WINDOW (data);
284	  g_assert (self);
285
286	  /* save changes note if file is edited */
287	  if (self->file_edited) {
</pre></li>

<li> You can now debug maemopad normally. Try and see, if you can execute the maemopad step by step so that you can get the new writing area on screen. Enter the s command like this:

<pre>
(gdb) s
maemopad_window_get_type () at /usr/include/glib-2.0/glib/gthread.h:344
344	  if G_LIKELY ((gpointer) g_atomic_pointer_get (value_location) != NULL)
(gdb) s
55	G_DEFINE_TYPE (MaemopadWindow, maemopad_window, HILDON_TYPE_WINDOW)
(gdb)
</pre></li>

<li> You can also try the bt (backtrace) command to see what functions were called. You should see a list of functions that have been called, similar to the following:

<pre>
(gdb) bt
#0  maemopad_window_get_type () at maemopad-window.c:55
#1  0x0804b45c in maemopad_window_on_menu_file_new (button=0x80cf0a8,
    data=0x8077018) at maemopad-window.c:283
#2  0xb76c441d in IA__g_cclosure_marshal_VOID__VOID (closure=0x80c7db0,
    return_value=0x0, n_param_values=1, param_values=0x8162390,
    invocation_hint=0xbfffe35c, marshal_data=0x804b450)
    at /home/bifh5/fremantle-i386-prereleased.cs2007q3/work/glib2.0-2.20
.1/gobject/gmarshal.c:77
....
<snip>
....
#30 0xb76136f8 in IA__g_main_loop_run (loop=0x80be058)
    at /home/bifh5/fremantle-i386-prereleased.cs2007q3/work/glib2.0-2.20
.1/glib/gmain.c:2678
#31 0xb7cd43e9 in IA__gtk_main () at gtkmain.c:1200
#32 0x0804a0f7 in main (argc=0, argv=0x60203006) at main.c:78
</pre>

At the top of the backtrace list are functions that were called last, and at the bottom of the list are the applications <code>main()</code> function and then the <code>gtk_main()</code>, etc.</li>

<li> If you now continued debugging with the <code>s</code> (step) command, you would end up looping the gtk main event loop. However, it is better to just give the <code>c</code> (continue) command:

<pre> 
(gdb) c
Continuing.
</pre></li>

<li> Now the maemopad application is running. If you select the New menu item again, the breakpoint is still set. So if you click New you would get again:

<pre> 
Breakpoint 1, maemopad_window_on_menu_file_new (button=0x80cf0a8,
    data=0x8077018) at maemopad-window.c:283
283	  MaemopadWindow *self = MAEMOPAD_WINDOW (data);
(gdb)
</pre></li>

<li> You can clear a specific breakpoint using the clear command. Remove the breakpoint in <code>maemopad_window_on_menu_file_new</code> function:

<pre> 
(gdb) clear maemopad_window_on_menu_file_new
Deleted breakpoint 1
(gdb) c
Continuing.
</pre></li>
</ol>

Because the breakpoint is now cleared, you can use the application normally under <code>Xephyr</code>.

===== Alternative method for starting gdb =====

Rather than starting the application framework, followed by gdb, it is possible to do both at the same time by passing an extra argument to the <code>af-sb-init.sh</code> script. For example:
 af-sb-init.sh start --gdb=my_app
where <code>my_app</code> is the application to debug.

== Debugging Hildon Desktop Plug-ins ==

This section explains how to debug Hildon Desktop plug-ins in maemo environment.

Desktop and Control Panel applications are both launched by the maemo-launcher daemon. Their plug-ins are all shared libraries (i.e. .so files). Writing these plug-ins is explained in maemo.org documentation.

=== Downloading and Compiling the Example Apps ===

For this example, you need to first download, compile, and install the hello-world-app package.
<ol>
<li> Download the source code of hello-world-app:

<pre> 
[sbox-FREMANTLE_X86: ~/src] > svn export https://vcs.maemo.org/svn/\
maemoexamples/tags/maemo_5.0/hello-world-app/
</pre></li>

<li> Build the package like this:

<pre>
[sbox-FREMANTLE_X86: ~/src] > cd hello-world-app
[sbox-FREMANTLE_X86: ~/src/hello-world-app] >
[sbox-FREMANTLE_X86: ~/src/hello-world-app] > export DEB_BUILD_OPTIONS=debug,nostrip
[sbox-FREMANTLE_X86: ~/src/hello-world-app] > ./autogen.sh
... etc ...
[sbox-FREMANTLE_X86: ~/src/hello-world-app] > dpkg-buildpackage -rfakeroot
dpkg-buildpackage: source package is hello-world-app
dpkg-buildpackage: source version is 2.2
dpkg-buildpackage: source changed by Maemo Integration &lt;integration@maemo.org&gt;
dpkg-buildpackage: host architecture i386
dpkg-buildpackage: source version without epoch 2.2
dpkg-checkbuilddeps: Using Scratchbox tools to satisfy builddeps
 fakeroot debian/rules clean
dh_testdir
dh_testroot
rm -f build-stamp configure-stamp
# Add here commands to clean up after the build process.
/scratchbox/tools/bin/make clean

... snip, output from compilation ...

dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is in
cluded)

[sbox-FREMANTLE_X86: ~/src/hello-world-app] >
</pre></li>

<li> You should now have:

<pre>
[sbox-FREMANTLE_X86: ~/src/hello-world-app] > cd ..
[sbox-FREMANTLE_X86: ~/src] > ls -lt
total 348
-rw-rw-r--  1 maemo maemo   1001 Aug 17 14:30 hello-world-app_2.2_i386.changes
-rw-r--r--  1 maemo maemo  76188 Aug 17 14:30 hello-world-app_2.2_i386.deb
drwxr-xr-x  6 maemo maemo   4096 Aug 17 14:30 hello-world-app
-rw-rw-r--  1 maemo maemo    384 Aug 17 14:30 hello-world-app_2.2.dsc
-rw-rw-r--  1 maemo maemo 240773 Aug 17 14:29 hello-world-app_2.2.tar.gz
</pre></li>

<li> Install the newly-compiled debug version of the package hello-world-app_2.2_i386.deb:

<pre>
[sbox-FREMANTLE_X86: ~/src] > fakeroot dpkg -i hello-world-app_2.2_i386.deb
(Reading database ... 24822 files and directories currently installed.)
Unpacking hello-world-app (from hello-world-app_2.2_i386.deb) ...
... snip ...
</pre></li>
</ol>

Now the Hildon desktop plug-ins example applications should be installed in the Scratchbox X86 target.

=== How to Debug Applications Started by Maemo-Launcher ===

Basically, these applications can be debugged by:

* attaching to an already running process
* starting the application using maemo-summoner

==== Attaching to Maemo-Launched Application with Gdb ====

With maemo-launched applications, it is necessary to give the maemo-launcher binary to <code>gdb</code> and attach to the already running process.

[sbox-FREMANTLE_X86: ~/ ] > export DISPLAY=:2
 [sbox-FREMANTLE_X86: ~/ ] > af-sb-init.sh start
 ... snip ...
 [sbox-FREMANTLE_X86: ~/ ] > pidof hildon-desktop | cut -d' ' -f1
 22961
 #
 # this would take the first (largest) PID value from the returned list.
 # The number 22961 is just an example.
 # smallest PID value is maemo-invoker which had requested maemo-launcher to
 # start hildon-desktop.
 #
 [sbox-FREMANTLE_X86: ~/ ] > native-gdb maemo-launcher
 ... snip ...
 (gdb) attach 22961
 Attaching to program: /targets/FREMANTLE_X86/usr/bin/maemo-launcher, process
  22961
 ... snip...
 (gdb)

Now it should be possible to debug the application normally with gdb.

==== Starting Maemo-Launched Application with Maemo-Summoner ====

The following will start the Control Panel under (native) gdb, and the newly installed Control Panel applet called <code>hello-world-app</code> will be debugged. The breakpoint will be set to function <code>hello_world_dialog_show()</code>. Pay attention to the question about the "pending shared library load".
 
 [sbox-FREMANTLE_X86: ~/ ] > export DISPLAY=:2
 [sbox-FREMANTLE_X86: ~/ ] > af-sb-init.sh start
 ... snip ...
 [sbox-FREMANTLE_X86: ~/ ] > run-standalone.sh native-gdb maemo-summoner
 ... snip ...
 (gdb) br hello_world_dialog_show
 Make breakpoint pending on future shared library load? (y or [n]) y
 Breakpoint 1 (hello_world_dialog_show) pending.
 (gdb) run /usr/bin/controlpanel.launch
 Starting program: /targets/FREMANTLE_X86/usr/bin/maemo-summoner /usr/bin/cont
 rolpanel.launch
 ... snip ...

This should start the Control Panel in the Xephyr screen. When clicking the hello world plug-in in the Control Panel, the execution should stop at the given breakpoint. Try, for example, to get a backtrace:

Breakpoint 2, hello_world_dialog_show () at libhelloworld.c:68
 68        GtkWidget *dialog = hello_world_dialog_new ();
 (gdb) bt
 #0  hello_world_dialog_show () at libhelloworld.c:80
 #1  0xb6b73528 in execute (osso=0x8604000, data=0x8613048, user_activated=1)
     at hello-world-applet.c:22
 ...
 (gdb) list
 75	}
 76
 77	void
 78	hello_world_dialog_show ()
 79	{
 80	  GtkWidget *dialog = GTK_WIDGET(hello_world_dialog_new ());
 81	  gtk_dialog_run (GTK_DIALOG(dialog));
 82	  gtk_widget_destroy (dialog);
 83	}
 84
 (gdb) c
 Continuing.

The backtrace shows which functions were called before the breakpoint was reached at <code>hello_world_dialog_show()</code>. Now the plug-in can be debugged normally with gdb.

Because the hello world plug-in was compiled with the <code>-g</code> option, the source code listing can be viewed with the <code>list</code> command.

To do this same for desktop, the <code>hildon-desktop</code> process needs to be killed first. The desktop can be killed with the command:

[sbox-FREMANTLE_X86: ~] > kill $(pidof hildon-desktop)
 #
 # after this you could start the hildon-desktop under gdb like this:
 #
 [sbox-FREMANTLE_X86: ~] > run-standalone.sh native-gdb maemo-summoner
 ... snip ...
 (gdb) run /usr/bin/hildon-desktop.launch
 ... snip ...

Doing this on the device would require first disabling the software lifeguard with the flasher tool. If this is not done, the device will reboot. The flag is:

--set-rd-flags=no-lifeguard-reset

== Running Out of Memory During Debugging on the Device ==

If running out of RAM during debugging on the device, there are a couple of options:

<ul>
<li> Add (more) swap to the device by increasing the size of the swap partition.
If there is enough memory, but gdb is still abruptly terminated, you can try setting it OOM-protected as [[root access|root]]:
<pre> 
/home/user # echo -17 > /proc/[PID of your gdb]/oom_adj
</pre>

By default, processes have an OOM (out-of-memory) adjustment value of zero. The value <code>-17</code> disables the kernel OOM killing for the given process. '''N.B.'''  As a result of this, some other processes might be killed by the kernel.</li>

<li> Use <code>gdbserver</code> to debug.</li>
</ul>

=== Notes on Using Gdbserver ===

<code>gdbserver</code> is a debugging tool that can be started and run in the Internet Tablet device. Then a connection can be made to this running instance of gdbserver program from a Linux PC with a gdb program. Gdbserver uses a lot less memory than a full scale gdb program, thus making it possible to perform debugging in devices that have limited RAM, such as PDAs.

The gdbserver does not care about symbols in the binary, but instead the Linux PC side gdb expects to have a local copy of the binary being debugged so that the binaries in the device can be stripped. Debugging core files from the device in Scratchbox with <code>gdbserver</code> has the same issues. In practice, it is easier to perform the debugging in the device itself. See the section above about prelinked binaries and libraries.

For further information about using <code>gdbserver</code>, <code>gdb</code> and <code>DDD</code>, see:

* [http://www.scratchbox.org/documentation/user/scratchbox-1.0/html/ideintegration.html#AEN76 scratchbox.org: Running the cross-compiled programs in a debugger]
* [http://www.scratchbox.org/documentation/user/scratchbox-1.0/html/debugging.html scratchbox.org: Debugging in Scratchbox]

== Valgrind Debugger ==

[[Documentation/devtools/maemo5/valgrind|Valgrind]] is a CPU simulator with different debugging and analyzing plug-ins. The Valgrind plug-ins are:

* '''<code>memcheck</code>:''' This plug-in tool is used to debug memory leaks and deallocation errors in applications. The following example will mainly focus on this.

* '''<code>massif</code>:''' This plug-in produces PostScript graph of process memory usage as a function of time. This also produces an ASCII or HTML report of allocation backtraces.

* '''<code>callgrind</code>:''' This can be used for profiling performance and getting call traces from programs. These can be further visualized with the <code>Kcachegrind</code> tool.

* '''<code>helgrind drd</code>:''' These help in finding race conditions in threaded programs.

{{ambox|text=In the maemo environment, you can use the Valgrind debugger only in the Scratchbox X86 target.}}

Valgrind has many options, but only the basic ones are covered here with examples. The link to the full Valgrind manual is given at the end of this section.

=== Installing Valgrind Tool ===

Installing Valgrind is simple. Log in to Scratchbox and run the following commands:
<ol>
<li> Get Valgrind from the repository: 
<pre> 
[sbox-FREMANTLE_ARMEL: ~] > sb-conf select FREMANTLE_X86
[sbox-FREMANTLE_X86: ~] > fakeroot apt-get install valgrind

...

Setting up valgrind (3.4.1-0maemo1+0m5) ...

[sbox-FREMANTLE_X86: ~] >
</pre>

The Maemo version of valgrind depends on the <code>libc6-dbg</code> package. On desktop Linux, some of the debug symbols are included in the libc6 library itself. If the debug symbols are missing from the libraries, Valgrind cannot match the error suppressions to the internal library functions. In the Maemo <code>libc6</code> case, it would show lots of errors for the dynamic linker.

If using a non-Maemo version of valgrind, the following environment variable needs to be set before valgrinding programs using Glib:

[sbox-FREMANTLE_X86: ~] > export G_SLICE="always-malloc"

Without this, valgrind will report bogus leaks from Glib. The GNOME wiki has [http://live.gnome.org/Valgrind more information on the use of valgrind with Glib].
</li>

<li> Get the two small example applications written in the C language to demonstrate the basic usage of valgrind tool. You can download these from maemo.org: [https://vcs.maemo.org/svn/maemoexamples/tags/maemo_5.0/valgrind_example/ valgrind_example].

<pre> 
[sbox-FREMANTLE_X86: ~/src] > svn export https://vcs.maemo.org/svn/\
maemoexamples/tags/maemo_5.0/valgrind_example/
[sbox-FREMANTLE_X86: ~/src] > cd valgrind_example
[sbox-FREMANTLE_X86: ~/src/valgrind_example] >
</pre>

</li>

<li> Compile the two small example applications in the following way: 
<pre>
[sbox-FREMANTLE_X86: ~/src/valgrind_example] > gcc valgrind_example.c \
-o valgrind_example -g -O2 -Wall
[sbox-FREMANTLE_X86: ~/src/valgrind_example] > gcc valgrind_example2.c \
-o valgrind_example2 -g -O2 -Wall
</pre></li>
</ol>

=== Using Valgrind Memory Debugger Tool ===

After compiling the small example application, it can be run under valgrind with the command:
 
 [sbox-FREMANTLE_X86: ~/src/valgrind_example] &gt; valgrind --tool=memcheck \
 --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes \
 ./valgrind_example

Explanation of the parameters and their meanings:

* '''<code>--tool=memcheck</code>:''' Defines which tool Valgrind should use. In this example, the memory checker tool is used.

* '''<code>--leak-check=yes</code>:''' With this option, Valgrind checks the code for potential memory leaks.

* '''<code>--show-reachable=yes</code>:''' This option creates a stack trace of the creation of the reachable but unfreed memory when the program exits.

* '''<code>--num-callers=20</code>:''' With this option, the number of function call levels that Valgrind displays can be changed. The default value is 12. Giving higher number takes a bit more memory. It is advisable to tune this option, because for example the gtk callstack can consist of more than 100 items.

* '''<code>--track-fds=yes</code>:''' <code>Track-fds</code> means <code>Track FileDescriptors</code>. If the application is opening and closing files with <code>fopen()</code> or <code>open()</code>, this will report which files are not closed.

The output of the example application should look like:

==6526== Memcheck, a memory error detector.
 ==6526== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
 ==6526== Using LibVEX rev 1884, a library for dynamic binary translation.
 ==6526== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
 ==6526== Using valgrind-3.4.1-maemo, a dynamic binary instrumentation framework.
 ==6526== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
 ==6526== For more details, rerun with: -v
 ==6526==
 ==6526==
 ==6526== FILE DESCRIPTORS: 3 open at exit.
 ==6526== Open file descriptor 2: /dev/pts/3
 ==6526==    &lt;inherited from parent&gt;
 ==6526==
 ==6526== Open file descriptor 1: /dev/pts/3
 ==6526==    &lt;inherited from parent&gt;
 ==6526==
 ==6526== Open file descriptor 0: /dev/pts/3
 ==6526==    &lt;inherited from parent&gt;
 ==6526==
 ==6526==
 ==6526== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 13 from 1)
 ==6526== malloc/free: in use at exit: 40 bytes in 2 blocks.
 ==6526== malloc/free: 3 allocs, 1 frees, 60 bytes allocated.
 ==6526== For counts of detected errors, rerun with: -v
 ==6526== searching for pointers to 2 not-freed blocks.
 ==6526== checked 56,184 bytes.
 ==6526==
 ==6526== 10 bytes in 1 blocks are definitely lost in loss record 1 of 2
 ==6526==    at 0x4020C19: malloc (in /targets/FREMANTLE_X86/usr/lib/valgrind/
 x86-linux/vgpreload_memcheck.so)
 ==6526==    by 0x8048402: main (valgrind_example.c:26)
 ==6526==
 ==6526==
 ==6526== 30 bytes in 1 blocks are definitely lost in loss record 2 of 2
 ==6526==    at 0x4020C19: malloc (in /targets/FREMANTLE_X86/usr/lib/valgrind/
 x86-linux/vgpreload_memcheck.so)
 ==6526==    by 0x8048470: main (valgrind_example.c:48)
 ==6526==
 ==6526== LEAK SUMMARY:
 ==6526==    definitely lost: 40 bytes in 2 blocks.
 ==6526==      possibly lost: 0 bytes in 0 blocks.
 ==6526==    still reachable: 0 bytes in 0 blocks.
 ==6526==         suppressed: 0 bytes in 0 blocks.
 [sbox-FREMANTLE_X86: ~/src/valgrind_example] >

The output of the Valgrind tells that there were 40 unallocated bytes for the application ("definitely lost"). This means that the example application is leaking memory, and cannot free it anymore. This means that the code should be studied closely.

Valgrind also tells the lines in the code where these allocations that are not freed are performed. In this example, the lines in question are 48 and 26.

==== Alternative method for starting Valgrind ====

[[#Alternative method for starting gdb|As with gdb]], Valgrind can be started together with the application framework, which is especially useful for graphical applications that must normally be run with <code>run-standalone.sh</code>. For example, to start the application framework an an application <code>my_app</code> underValgrind:
 af-sb-init.sh start --valgrind=my_app
If other Valgrind frontends are required, they can be set with the environment variables:
* <code>VALGRINDCMD</code>: the command you want to run inside valgrind
* <code>VALGRIND</code>: the valgrind command and parameters, including the frontend to use
For example, to run <code>massif</code> on <code>maemo-af-desktop</code>:
 VALGRINDCMD="maemo-af-desktop" VALGRIND="valgrind --tool=massif --num-callers=50 \
        --trace-children=yes --depth=5 --format=html -log-file=/tmp/massif \
        --alloc-fn=g_malloc --alloc-fn=g_malloc0 --alloc-fn=g_realloc \
        --alloc-fn=g_slice_alloc --alloc-fn=g_try_malloc \
        --alloc-fn=g_slice_alloc0" af-sb-init.sh start
 ...
 af-sb-init.sh stop

=== Official Valgrind Manual ===

Valgrind.org has an [http://valgrind.org/docs/manual/manual.html Valgrind manual], which explains the many options and debugging practices that Valgrind can support.

== Other Debugging Tools ==

The SDK provides a number of other tools that are useful in debugging. See the linked documentation pages to learn more about them.

* ''strace''  - a system call tracer. Prints out all the system calls made by the traced process.
** [[Documentation/devtools/maemo5/strace|strace documentation]]
* ''ltrace''  - a library call tracer. Prints out all library calls made by the traced process.
** [[Documentation/devtools/maemo5/ltrace|ltrace documentation]]
* ''latrace'' - traces dynamic library calls.
** [[Documentation/devtools/maemo5/latrace|latrace documentation]]
* ''sp-rich-core''  - produce ''rich cores''  that will provide a snapshot of the system's state at the time of the crash in addition to the core dump file.
** [[Documentation/devtools/maemo5/sp-rich-core|sp-rich-core documentation]]
* ''syslog''  - a logging system that can collect logs in a centralized way.
** [[Documentation/devtools/maemo5/syslog|syslog documentation]]

Please also have a look at the tools main page [[Documentation/devtools/maemo5|Maemo 5 SDK Tools]] for even more tools that could be useful for you.

= Making a Debian Debug Package =

Application developers should provide debug packages corresponding to their application packages. This section shows how to do that.

Debian debug packages contain ''debug symbol''  files that debuggers (like <code>gdb</code>) will automatically load when the application is debugged.

In an ARM environment, the debugger cannot do backtracing or show correct function names without debugging symbols, making it impossible to debug optimized binaries or libraries. All libraries in the device are optimized.

In an X86 target, debugging can be done without debug symbols.

If the package maintainer provides no debug package, then the other developers in the community need to spend extra time and effort to recompile the application (or the library) with debugging symbols. This extra (time-consuming) step can be simply eliminated if the package maintainer provides a ready-compiled debug package.

This section covers creating a Debian debug package from the Maemo/Scratchbox point of view.

It is the package maintainers role and decision to create and provide a debug package for their application or library. This means that you as the package owner are responsible to modify the debian package configurations, so that the <code>dpkg-buildpackage</code> tool will produce the additional <code>dbg</code>-package.

If you are a maintainer of a library or binary package, it is strongly recommended to create a corresponding debug package. This is needed by anybody wanting to either debug or profile your software or something using it.

If the package is otherwise fine (and <code>debian/rules</code> uses debhelper <code>dh_* scripts</code>), creating debug packages is easy, just:

* Add <code>*-dbg</code> package descriptions to the <code>debian/control</code> file

* Make sure that "<code>-g</code>" is included in the compiler flags and Makefiles do not strip binaries (for example, give "<code>-s</code>" to "<code>install</code>" command)

* Add "<code>--dbg-package=&lt;package name&gt;</code>" argument to "<code>dh_strip</code>" invocation in <code>debian/rules</code> for each package containing binaries (<code>dh_strip</code> will then extract the debug symbols from the given binary package to a separate debug package)

* Rebuild/test source package

More detailed instructions are below.

== Creating DBG Packages ==

Steps to create a debian DBG package are:
<ol>
<li> '''Clean up any previous dbg configurations from your package''' 
If your package already provides the debugging symbols the "hard, old way" then you should clean these configurations first.
If you use <code>dh_strip --dbg-package</code> option in <code>debian/rules</code> file, then it is not necessary to build or copy anything to the <code>-dbg</code> package build directory anymore. Check your packages <code>debian/rules</code> file and remove all lines that have statements to create dbg packages.
If you have any files named <code>debian/*-dbg.*</code> then just remove these files. These were required with the old way to use <code>dh_strip</code>.
The following steps are the only ones needed to create the debug packages with newer versions of <code>dh_strip</code>.
After you have cleaned up any previous dbg configurations from your debian files move to the next step.</li>
<li> '''Define DBG package(s)''' 
For every package listed in your <code>debian/control</code> file that contains libraries or binaries, you need to add corresponding <code>&lt;package&gt;-dbg</code> entry. If you have <code>debian/control.in</code> then modify that instead.
Make the debug package(s) to depend from the corresponding binary/library package. Here is an example:

<pre>
.
. other package definitions above
.

Package: libgtk2.0-0-dbg
Section: libdevel
Architecture: any
Depends: libgtk2.0-0 (= ${binary:Version})
Description: Debug symbols for the Gtk library
</pre>

If the package contains binaries instead of libraries, the <code>Section</code> should be <code>devel</code>, not <code>libdevel</code>. The new <code>-dbg</code> package may have a different name from the old style debug package (for example <code>libgtk2.0-0-dbg</code>, not <code>libgtk2.0-dbg</code>). If there are earlier (old style) debug packages in the repositories, the new debug package should replace or conflict with the old one.</li>

<li> '''Add option -g to CFLAGS''' 
Make sure you have set option <code>-g</code> in <code>CFLAGS</code> in the <code>debian/rules</code> file, and that this option is effective and always enabled. Otherwise the debug package will not contain the required debug symbols.
For example, use a line like this for <code>CFLAGS</code>

<pre> 
CFLAGS = -Wall -g -O2
</pre></li>
<li> '''Use dh_strip''' 
You can either provide all debugging information for your package, or just a minimal debugging information required to debug something else using your library.
You want to select the latter option only if you want to reveal as little of your code as possible (for example) for contract reasons.
<ol><li> '''Providing all debug information in your dbg package.''' 
<code>debian/compat</code> levels smaller than 3 should not be used (1 is default!). If your package sets <code>debian/compat</code> level '''below'''  5, give the following arguments to dh_strip in <code>debian/rules</code>:

<pre>
dh_strip --dbg-package=&lt;package1&gt; [--dbg-package=&lt;package2&gt; ...]
</pre>

For example:

<pre>
dh_strip --dbg-package=libgtk2.0-0
</pre>

If compat level is 5 or higher, use syntax:

<pre>
dh_strip --dbg-package=&lt;package1&gt;-dbg\
 [--dbg-package=&lt;package2&gt;-dbg ...]
</pre>

For example:

<pre>
dh_strip --dbg-package=libgtk2.0-0-dbg
</pre>

If you are using cdbs instead of debhelper in your <code>debian/rules</code> file, use this instead:

<pre>
DEB_DH_STRIP_ARGS := &lt;dh_strip arguments&gt;
</pre>

For example (for compat levels below 5):

<pre>
DEB_DH_STRIP_ARGS := --dbg-package=&lt;package1&gt; ...
</pre>

</li>

<li> '''Providing just minimal debug information''' 
In case you do not want to reveal all information about your binary (for contractual reasons, as an example), you can provide a debug symbol file just with the <code>.debug_frame</code> section (which is the minimal information needed by <code>gdb</code> to show working backtraces in an ARM environment). In addition to information provided by the binary file, it will reveal only static function names and the number of function arguments. To do this, replace the above <code>dh_strip</code> line in <code>debian/rules</code> file <code>binary-arch</code> target with:

<pre>
chmod +x $(shell pwd)/debian/wrapper
export PATH=$(shell pwd)/debian/wrapper:$$PATH; \
dh_strip  --dbg-package=&lt;package1&gt; [--dbg-package=&lt;package2&gt; ...]
</pre>

Notice that the wrapper script is set executable because the <code>dpkg-source</code> does not preserve the exec permissions. Store the following wrapper script as <code>debian/wrapper/objcopy</code>:

#!/bin/sh
 #

case " $* " in
   *" --only-keep-debug "*)
     exec /usr/bin/objcopy -R .debug_info -R .debug_aranges \
          -R .debug_pubnames -R .debug_abbrev -R .debug_line \
          -R .debug_str -R .debug_ranges -R .comment -R .note "$@"
     ;;
 esac
 exec /usr/bin/objcopy "$@"

With this <code>dh_strip</code> will use <code>objcopy</code> through this wrapper (and remove the other debug sections).</li></ol>
<li> Verify the package(s)
Update the Debian changelog with <code>dch -i</code> and build the package with <code>dpkg-buildpackage -rfakeroot</code>. This will create new Debian source package (dsc + diff) and binary package(s) which you can install on the target for additional testing.</li>
</ol>
See also <code>dh_strip (1)</code> and <code>debhelper (7)</code> manual pages for details about the helper scripts.

== Using and Installing DBG Packages ==

The debug packages are easy to use, just <code>apt-get install</code> them and on target gdb load the new style debug symbol files (installed to subdirectories under <code>/usr/lib/debug/</code>) automatically.

Inside Scratchbox, debuggers and profilers search for the debug symbol files under scratchbox target directory under <code>/usr/lib/debug</code>, so link a target directory under it.
 
 mkdir /usr/lib/debug
 cd /usr/lib/debug
 mkdir targets
 ln -s /usr/lib/debug \
  targets/$(sh -c '. /targets/links/scratchbox.config;echo $SBOX_TARGET_NAME')

The reason for this is that realpath on libraries and binaries inside Scratchbox returns <code>/scratchbox/targets/.../usr/lib/...</code> instead of just normal <code>/usr/lib/...</code>.

Installing the <code>maemo-debug-scripts</code> package does this link automatically.

And for thread debugging to work, install <code>libc6-dbg</code> package and <code>gdb</code> package, the Scratchbox provided host <code>gdb</code> doesn't work with threads (or ARM core dumps). To use the native gdb in Scratchbox, you need to start gdb the following way:
 
 SBOX_REDIRECT_IGNORE=/usr/bin/gdb /usr/bin/gdb /usr/bin/my-binary

The <code>maemo-debug-scripts</code> package provides <code>native-gdb</code> script for this.

For gdb to find old style debug symbol files (installed directly into <code>/usr/lib/debug/</code>), use <code>LD_LIBRARY_PATH</code> or load them manually in <code>gdb</code>.

Files in these old style debug symbol files contain both the binary and debug symbol sections, so they are also larger than the new style debug symbols that <code>dh_strip</code> instructions above will create.

== For Further Reading ==

* [http://www.debian.org/doc/manuals/maint-guide/index.en.html Debian New Maintainers' Guide]
* [http://www.debian.org/doc/developers-reference/best-pkging-practices.html Debian Developer's Reference, Chapter 6 - Best Packaging Practices]

[[Category:Development]]
[[Category:Documentation]]
[[Category:Fremantle]]

Please note that all contributions to maemo.org wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see maemo.org wiki:Copyrights for details). Do not submit copyrighted work without permission!

Summary:

Cancel | Editing help (opens in new window)

Templates used on this page:

Template:Ambox (view source) (semi-protected)

Retrieved from "http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Kernel_and_Debugging_Guide/Maemo_Debugging_Guide"

Personal tools

Navigation

Views

Editing Documentation/Maemo 5 Developer Guide/Kernel and Debugging Guide/Maemo Debugging Guide

@@ Line 49: / Line 49: @@
 When debugging the ARM architecture, pay attention to the following issues:
-* To make backtraces work properly on the ARM side, the debug packages need to be installed for the libraries the application is using. Profiling and debugging (gdb) tools require the code to have either framepointers or debugging symbols to unwind the stack. This is necessary for showing backtraces or call graphs.
+* To make backtraces work properly on the ARM side, the dbg packages need to be installed for the libraries the application is using. Profiling and debugging (gdb) tools require the code to have either framepointers or debugging symbols to unwind the stack. This is necessary for showing backtraces or call graphs.
-* C language functions with the <code>__attribute__((__noreturn__))</code> statement need to be compiled with the gcc option: <code>-mapcs-frame</code> otherwise the compiler excludes their function frame. On Maemo, the Glib and C library noreturn functions like <code>abort()</code> already have this. Without a function frame, gdb cannot backtrace through "noreturn" functions.
+* C language functions with the <code>__attribute__((__noreturn__))</code> statement need to be compiled with the gcc option: <code>-mapcs-frame</code> otherwise the compiler excludes their function frame. Maemo Glib and C-library noreturn functions like <code>abort()</code> already have this. Without a function frame, gdb cannot backtrace through "noreturn" functions.
 * In addition, for gdb to be able to display the correct function names during debugging, it also needs to have access to the debug symbols. Without them, gdb shows the preceding exported function name for a given address.