Building packages with sbdmock

(sbdmock configuration: -- fix table formatting)
(wikify slightly)
 
(9 intermediate revisions not shown)
Line 1: Line 1:
-
'''''Note: this page is work in progress documenting my experiments with sbdmock'''''  --[[User:gcobb|gcobb]] 18:17, 12 July 2009 (UTC)
+
''sbdmock'' builds binary "deb" packages from source, it is part of Maemo's automatic build chain.
-
Before submitting packages to the autobuilder, it is important to test that it will build.  The first step of that is to use ''dpkg-buildpackage'' in the scratchbox environment to create the packages (or a tool like [[MUD]] can be used, which automatically calls ''dpkg-buildpackage'').
+
''sbdmock'' is a tool which provides a clean scratchbox environment, allowing you to test that your package can build in that environment before submitting it to the autobuilder. It can either be used manually, or it can be used as part of an automatic script (for example, I use for my nightly builds of GPE, OpenSync and Xsisusb).
-
 
+
-
However, just because a package builds in your normal scratchbox development environment is no guarantee that it will build in the autobuilder.  The main reason is that the autobuilder starts with no packages installed and only installs the packages defined in your package control file (using the ''BuildDepends:'' line).
+
-
 
+
-
''sbdmock'' is a tool which provides a clean scratchbox environment, allowing you to test that your package can build in that environment before submitting it to the autobuilder. It can either be used manually, or it can be used as part of an automatic script (for example, I intend to use it as part of my nightly builds of GPE).
+
-
 
+
-
Of course, if the build in sbdmock fails you will need to debug it.  sbdmock can be used for that as well.
+
The sbdmock home page is: http://bifh.org/wiki/sbdmock
The sbdmock home page is: http://bifh.org/wiki/sbdmock
Line 13: Line 7:
The sbdmock git source tree can be accessed at: http://github.com/kad/sbdmock/tree/master
The sbdmock git source tree can be accessed at: http://github.com/kad/sbdmock/tree/master
-
= Getting sbdmock =
+
== Why use sbdmock? ==
-
This is the procedure I used, using Debian squeezePackages for sbdmock may be available for your platform, however I wanted to use the latest development version and, as I expected to make changes to it, I did not want to install it into system directories.
+
Before submitting a package to the autobuilder, it is important to test that it will buildThe first step of that is to use ''dpkg-buildpackage'' in the scratchbox environment to create the packages (or a tool like [http://mud-builder.garage.maemo.org/ mud-builder] can be used, which automatically calls ''dpkg-buildpackage'').
-
1. Install Debian packages for ''git-core'', ''python-pip'', ''python-setuptools'' and ''python-virtualenv'' using:
+
However, just because a package builds in your normal scratchbox development environment is no guarantee that it will build in the autobuilder. The most common problem is that the autobuilder starts with no packages installed and only installs the packages defined in your package control file (using the ''Build-Depends:'' line).
-
'''sudo aptitude install git-core python-pip python-setuptools python-virtualenv'''
+
''sbdmock'' allows you to test your build in a clean scratchbox target, just like the autobuilder uses.  Of course, if the build in sbdmock fails you will need to debug it.  sbdmock can be used for that as well.
-
2. Create the python virtual environment:
+
== Getting sbdmock ==
-
  '''virtualenv sbdmock'''
+
This is the procedure I used, using Debian squeeze. Packages for sbdmock may be available for your platform, however I wanted to use the latest development version and, as I expected to make changes to it, I did not want to install it into system directories.
-
3. Install minideblib, which is a prerequisite for using sbdmock:
+
<ol>
 +
<li>
 +
Install Debian packages for ''git-core'', ''python-pip'', ''python-setuptools'' and ''python-virtualenv'' using:<pre>sudo aptitude install git-core python-pip python-setuptools python-virtualenv</pre>
 +
</li>
 +
<li>
 +
Create the Python virtual environment:<pre>virtualenv sbdmock</pre>
 +
</li>
 +
<li>
 +
Install minideblib, which is a prerequisite for using sbdmock:<pre>pip install -E sbdmock minideblib</pre>
 +
</li>
 +
<li>
 +
Install sbdmock from the git repository:<pre>pip install -E sbdmock -e git://github.com/kad/sbdmock.git#egg=sbdmock</pre>
 +
</li>
 +
<li>
 +
Copy the config files to your sbdmock config directory:
 +
<pre>
 +
mkdir ~/.sbdmock/
 +
cp sbdmock/src/sbdmock/etc/*.cfg ~/.sbdmock/
 +
</pre>
 +
</li>
 +
<li>
 +
Test the sbdmock installation:<pre>sbdmock/bin/sbdmock --help</pre>
 +
</li>
 +
</ol>
-
'''pip install -E sbdmock minideblib'''
+
Alternatively, if you are on a Debian based system, you might prefer to build and install Debian packages instead:
-
4. Install sbdmock from the git repository:
+
<ol>
-
 
+
<li>
-
'''pip install -E sbdmock -e git://github.com/kad/sbdmock.git#egg=sbdmock'''
+
Install ''git-core'', ''python-setuptools'' and ''dpkg-dev'':<pre>sudo aptitude install git-core python-setuptools dpkg-dev</pre>
-
 
+
</li>
-
5. Copy the config files to your sbdmock config directory:
+
<li>
-
 
+
Download latest minideblib tarball from http://pypi.python.org/pypi/minideblib
-
'''mkdir ~/.sbdmock/'''
+
</li>
-
'''cp sbdmock/src/sbdmock/etc/*.cfg ~/.sbdmock/'''
+
<li>
-
 
+
Unpack it and build the .deb package:
-
6. Test the sbdmock installation:
+
<pre>
-
 
+
tar -xvzf minideblib-*.tar.gz
-
'''sbdmock/bin/sbdmock --help'''
+
cd minideblib-*/
 +
dpkg-buildpackage -us -uc -rfakeroot
 +
</pre>
 +
</li>
 +
<li>
 +
Download sbdmock sources:<pre>git clone git://github.com/kad/sbdmock.git sbdmock</pre>
 +
</li>
 +
<li>
 +
Build sbdmock .deb package:<pre>dpkg-buildpackage -us -uc -rfakeroot -i.git -I.git</pre>
 +
</li>
 +
<li>
 +
Install the built .deb packages:<pre>dpkg -i python-minideblib*.deb sbdmock*.deb</pre>
 +
</li>
 +
<li>
 +
Copy the config files to your sbdmock config directory:
 +
<pre>
 +
mkdir ~/.sbdmock/
 +
cp /usr/share/doc/sbdmock/examples/*.cfg ~/.sbdmock/
 +
</pre>
 +
</li>
 +
<li>
 +
Test the sbdmock installation:<pre>sbdmock --help</pre>
 +
</li>
 +
</ol>
-
= sbdmock documentation =
+
== sbdmock documentation ==
There is no documentation.  ''sbdmock --help'' gives:
There is no documentation.  ''sbdmock --help'' gives:
Line 75: Line 115:
</pre>
</pre>
-
= sbdmock configuration =
+
== sbdmock configuration ==
The most important part of the configuration is to put the rootstraps where sbdmock can find them.   
The most important part of the configuration is to put the rootstraps where sbdmock can find them.   
-
The rootstrap file names are specified in the configuration files (for example the standard chinook configuration file specifies ''maemo-sdk-rootstrap_4.0_armel.tgz'' as the rootstrap file name). Be aware that the rootstrap file names are relative to your scratchbox home directory. So, you have three options:
+
The rootstrap file names are specified in the configuration files (for example the standard chinook configuration file specifies ''maemo-sdk-rootstrap_4.0_armel.tgz'' as the rootstrap file name). Be aware that the rootstrap file names are relative to your scratchbox home directory. So, you have three options:
-
1) Put the rootstrap files in your scratchbox home directory.  This is easiest and means no changes to the configuration files. However, depending on how you installed the SDK, they are almost certainly not in that directory at the moment.
+
# Put the rootstrap files in your scratchbox home directory.  This is easiest and means no changes to the configuration files. However, depending on how you installed the SDK, they are almost certainly not in that directory at the moment.
-
 
+
# Put a softlink in the scratchbox home directory by logging in to Scratchbox and using something like:<pre>ln -s /some/directory/maemo-sdk-rootstrap_4.0_armel.tgz</pre>Note that this directory has to be accessible from within the Scratchbox environment, of course.
-
2) Put a softlink in the scratchbox home directory by logging in to scratchbox'' and using something like:
+
# Edit the configuration file to change the location.  The configuration file is in the <code>~/.sbdmock</code> directory and the line to change is the one which defines ''<nowiki>config_opts['rootstrap']</nowiki>''.  Again, the file has to be accessible from within the scratchbox environment.
-
 
+
-
'''ln -s /some/directory/maemo-sdk-rootstrap_4.0_armel.tgz'''
+
-
 
+
-
Note that this directory has be be accessible from within the scratchbox environment, of course.
+
-
 
+
-
3) Edit the configuration file to change the location.  The configuration file is in the ''~/.sbdmock'' directory and the line to change is the one which defines ''<nowiki>config_opts['rootstrap']</nowiki>''.  Again, the file has to be accessible from within the scratchbox environment.
+
Currently the following config files are shipped with sbdmock, although others may be added soon:
Currently the following config files are shipped with sbdmock, although others may be added soon:
-
{| class="wikitable" border="1"
+
{| class="wikitable"
! Configuration name !! SDK version !! Architecture !! Repository !! Rootstrap
! Configuration name !! SDK version !! Architecture !! Repository !! Rootstrap
|-
|-
Line 113: Line 147:
|}
|}
-
= Test build =
+
'''Note:''' the ''extras'' configurations are not used by the autobuilder: all building is done in extras-devel.
 +
 
 +
== Test build ==
To test the package defined in ''something.dsc'' will build using the chinook autobuilder use the following command:
To test the package defined in ''something.dsc'' will build using the chinook autobuilder use the following command:
-
  '''sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel -u ''something.dsc'''''
+
  sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel -u ''something.dsc''
This command must be issued from '''outside''' the scratchbox environment.
This command must be issued from '''outside''' the scratchbox environment.
Line 123: Line 159:
''sbdmock'' displays a message at the end of the build saying where the results (including logs) can be found.  That filespec is designed to be used from outside the scratchbox environment.  From inside the scratchbox environment the results are in ''<nowiki>~/maemo-chinook-armel-extras-devel/results</nowiki>''.
''sbdmock'' displays a message at the end of the build saying where the results (including logs) can be found.  That filespec is designed to be used from outside the scratchbox environment.  From inside the scratchbox environment the results are in ''<nowiki>~/maemo-chinook-armel-extras-devel/results</nowiki>''.
-
Once the ''-armel-extras-devel'' build is working, it is worth testing the other configurations: ''-i386-extras-devel'' (which the autobuilder will build automatically unless your package defines that it is only for armel), ''-armel-extras'' (so that you know the build will work if/when the package is promoted to extras) and ''-i386-extras''.
+
Once the ''-armel-extras-devel'' build is working, it is worth testing ''-i386-extras-devel'', which the autobuilder will build automatically unless your package defines that it is only for armel. Note that the autobuilder never uses the ''-extras'' configurations.
-
= Debugging build problems =
+
== Debugging build problems ==
The test builds described above use the ''sbdmock -u'' option.  This option causes sbdmock to create a completely new scratchbox target, with a unique name, run the build and delete the target again.  This is ideal for automatic builds and test builds but it is not useful when you want to debug a failing build.  In that case, you would like the scratchbox target to remain around so that you can select it and do debugging to see why your build failed, and fix it.
The test builds described above use the ''sbdmock -u'' option.  This option causes sbdmock to create a completely new scratchbox target, with a unique name, run the build and delete the target again.  This is ideal for automatic builds and test builds but it is not useful when you want to debug a failing build.  In that case, you would like the scratchbox target to remain around so that you can select it and do debugging to see why your build failed, and fix it.
Line 133: Line 169:
For example:
For example:
-
  '''sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel --sbtarget TEST_40_ARMEL ''something.dsc'''''
+
  sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel --sbtarget TEST_40_ARMEL ''something.dsc''
'''Note:''' currently sbdmock does not actually recreate the target completely, so it is important that you specify a target which has the correct compiler, devkit and CPU transparency settings for the configuration you are using.  In other words, use a target which was set up for the correct version of the SDK and the correct architecture.
'''Note:''' currently sbdmock does not actually recreate the target completely, so it is important that you specify a target which has the correct compiler, devkit and CPU transparency settings for the configuration you are using.  In other words, use a target which was set up for the correct version of the SDK and the correct architecture.
If the build fails you can go into scratchbox, select the target (if necessary) and debug the problem.  You can then use ''dpkg-buildpackage'' to recreate the package files and re-try the build in ''sbdmock''.
If the build fails you can go into scratchbox, select the target (if necessary) and debug the problem.  You can then use ''dpkg-buildpackage'' to recreate the package files and re-try the build in ''sbdmock''.
 +
 +
== Example ==
 +
 +
'''Todo:''' create a worked example showing how to debug and fix a missing build dependency.
 +
 +
'''''Note: this page is work in progress documenting my experiments with sbdmock'''''  -- [[User:gcobb|gcobb]] 22:04, 16 July 2009 (UTC)
 +
[[Category:Development]]
[[Category:Development]]

Latest revision as of 13:05, 4 February 2011

sbdmock builds binary "deb" packages from source, it is part of Maemo's automatic build chain.

sbdmock is a tool which provides a clean scratchbox environment, allowing you to test that your package can build in that environment before submitting it to the autobuilder. It can either be used manually, or it can be used as part of an automatic script (for example, I use for my nightly builds of GPE, OpenSync and Xsisusb).

The sbdmock home page is: http://bifh.org/wiki/sbdmock

The sbdmock git source tree can be accessed at: http://github.com/kad/sbdmock/tree/master

Contents

[edit] Why use sbdmock?

Before submitting a package to the autobuilder, it is important to test that it will build. The first step of that is to use dpkg-buildpackage in the scratchbox environment to create the packages (or a tool like mud-builder can be used, which automatically calls dpkg-buildpackage).

However, just because a package builds in your normal scratchbox development environment is no guarantee that it will build in the autobuilder. The most common problem is that the autobuilder starts with no packages installed and only installs the packages defined in your package control file (using the Build-Depends: line).

sbdmock allows you to test your build in a clean scratchbox target, just like the autobuilder uses. Of course, if the build in sbdmock fails you will need to debug it. sbdmock can be used for that as well.

[edit] Getting sbdmock

This is the procedure I used, using Debian squeeze. Packages for sbdmock may be available for your platform, however I wanted to use the latest development version and, as I expected to make changes to it, I did not want to install it into system directories.

  1. Install Debian packages for git-core, python-pip, python-setuptools and python-virtualenv using:
    sudo aptitude install git-core python-pip python-setuptools python-virtualenv
  2. Create the Python virtual environment:
    virtualenv sbdmock
  3. Install minideblib, which is a prerequisite for using sbdmock:
    pip install -E sbdmock minideblib
  4. Install sbdmock from the git repository:
    pip install -E sbdmock -e git://github.com/kad/sbdmock.git#egg=sbdmock
  5. Copy the config files to your sbdmock config directory:
    mkdir ~/.sbdmock/
    cp sbdmock/src/sbdmock/etc/*.cfg ~/.sbdmock/
    
  6. Test the sbdmock installation:
    sbdmock/bin/sbdmock --help

Alternatively, if you are on a Debian based system, you might prefer to build and install Debian packages instead:

  1. Install git-core, python-setuptools and dpkg-dev:
    sudo aptitude install git-core python-setuptools dpkg-dev
  2. Download latest minideblib tarball from http://pypi.python.org/pypi/minideblib
  3. Unpack it and build the .deb package:
    tar -xvzf minideblib-*.tar.gz
    cd minideblib-*/
    dpkg-buildpackage -us -uc -rfakeroot
    
  4. Download sbdmock sources:
    git clone git://github.com/kad/sbdmock.git sbdmock
  5. Build sbdmock .deb package:
    dpkg-buildpackage -us -uc -rfakeroot -i.git -I.git
  6. Install the built .deb packages:
    dpkg -i python-minideblib*.deb sbdmock*.deb
  7. Copy the config files to your sbdmock config directory:
    mkdir ~/.sbdmock/
    cp /usr/share/doc/sbdmock/examples/*.cfg ~/.sbdmock/
    
  8. Test the sbdmock installation:
    sbdmock --help

[edit] sbdmock documentation

There is no documentation. sbdmock --help gives:

Usage: 
    usage: sbdmock [options] /path/to/dsc
    optional commands:
        clean - clean out the specified chroot
        init - initialize the chroot, do not build anything

Options:
  -h, --help            show this help message and exit
  -r TARGET             compilation target name/config file name default: default
  --no-clean            do not clean chroot before building
  --debug               Output copious debugging information
  --resultdir=RESULTDIR
                        path for resulting files to be put
  --statedir=STATEDIR   path for state dirresulting files to be put
  --sbtarget=SBTARGET   Overrides scratchbox target name from configuration file
  --addrepo=ADDREPO     Additional repository to sources.list
  --insertrepo=INSERTREPO
                        Additional repository to sources.list. Will be added on first place
  --uniqueext=UNIQUEEXT
                        Arbitrary, unique extension to append to target name
  -u                    Generate unique extension (based on package name)
  -b                    indicates that no source files are to be built
  -B                    same as '-b', but no architecture-independent binary package files are to be build either
  -S                    only the source should be build and no binary packages need to be made
  --dbo=DBO             Modifiers for DEB_BUILD_OPTIONS (e.g. '+debug,-parallel')
  --define=DEFINE       Define variables which might be used in other variables. Format: name=value

[edit] sbdmock configuration

The most important part of the configuration is to put the rootstraps where sbdmock can find them.

The rootstrap file names are specified in the configuration files (for example the standard chinook configuration file specifies maemo-sdk-rootstrap_4.0_armel.tgz as the rootstrap file name). Be aware that the rootstrap file names are relative to your scratchbox home directory. So, you have three options:

  1. Put the rootstrap files in your scratchbox home directory. This is easiest and means no changes to the configuration files. However, depending on how you installed the SDK, they are almost certainly not in that directory at the moment.
  2. Put a softlink in the scratchbox home directory by logging in to Scratchbox and using something like:
    ln -s /some/directory/maemo-sdk-rootstrap_4.0_armel.tgz
    Note that this directory has to be accessible from within the Scratchbox environment, of course.
  3. Edit the configuration file to change the location. The configuration file is in the ~/.sbdmock directory and the line to change is the one which defines config_opts['rootstrap']. Again, the file has to be accessible from within the scratchbox environment.

Currently the following config files are shipped with sbdmock, although others may be added soon:

Configuration name SDK version Architecture Repository Rootstrap
maemo-chinook-armel-extras-devel Chinook (Maemo 4.0) armel (Internet Tablet) extras-devel maemo-sdk-rootstrap_4.0_armel.tgz
maemo-chinook-armel-extras Chinook (Maemo 4.0) armel (Internet Tablet) extras maemo-sdk-rootstrap_4.0_armel.tgz
maemo-chinook-i386-extras-devel Chinook (Maemo 4.0) i386 (scratchbox PC) extras-devel maemo-sdk-rootstrap_4.0_i386.tgz
maemo-chinook-i386-extras Chinook (Maemo 4.0) i386 (scratchbox PC) extras maemo-sdk-rootstrap_4.0_i386.tgz

Note: the extras configurations are not used by the autobuilder: all building is done in extras-devel.

[edit] Test build

To test the package defined in something.dsc will build using the chinook autobuilder use the following command:

sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel -u something.dsc

This command must be issued from outside the scratchbox environment.

sbdmock displays a message at the end of the build saying where the results (including logs) can be found. That filespec is designed to be used from outside the scratchbox environment. From inside the scratchbox environment the results are in ~/maemo-chinook-armel-extras-devel/results.

Once the -armel-extras-devel build is working, it is worth testing -i386-extras-devel, which the autobuilder will build automatically unless your package defines that it is only for armel. Note that the autobuilder never uses the -extras configurations.

[edit] Debugging build problems

The test builds described above use the sbdmock -u option. This option causes sbdmock to create a completely new scratchbox target, with a unique name, run the build and delete the target again. This is ideal for automatic builds and test builds but it is not useful when you want to debug a failing build. In that case, you would like the scratchbox target to remain around so that you can select it and do debugging to see why your build failed, and fix it.

To do this, use the sbdmock --sbtarget command to specify a target to use. This should be a target which already exists but one which you do not mind sbdmock cleaning. Cleaning involves re-initialising the target so things like changes to /etc/apt/sources.list and packages you have installed will be lost. But don't worry, no files in your home directory will be modified.

For example:

sbdmock/bin/sbdmock -r maemo-chinook-armel-extras-devel --sbtarget TEST_40_ARMEL something.dsc

Note: currently sbdmock does not actually recreate the target completely, so it is important that you specify a target which has the correct compiler, devkit and CPU transparency settings for the configuration you are using. In other words, use a target which was set up for the correct version of the SDK and the correct architecture.

If the build fails you can go into scratchbox, select the target (if necessary) and debug the problem. You can then use dpkg-buildpackage to recreate the package files and re-try the build in sbdmock.

[edit] Example

Todo: create a worked example showing how to debug and fix a missing build dependency.

Note: this page is work in progress documenting my experiments with sbdmock -- gcobb 22:04, 16 July 2009 (UTC)