Task:PR1.2 autobuilder

(New page: == Background == After the PR1.2 SDK was released, Niels deployed it to the autobuilder; enacting a plan which was discussed on this list. This has caused problems with libhildon dependen...)
({{task|completed}})
 
(6 intermediate revisions not shown)
Line 1: Line 1:
-
== Background ==
+
{{task|completed}}
-
After the PR1.2 SDK was released, Niels deployed it to the
+
== Background ==
-
autobuilder; enacting a plan which was discussed on this list. This
+
-
has caused problems with libhildon dependencies, but the problems
+
-
aren't just limited to that library:
+
-
  http://lists.maemo.org/pipermail/maemo-developers/2010-March/025668.html
+
After the PR1.2 SDK was released, Niels deployed it to the autobuilder; enacting a plan which was discussed on this list. This has caused problems with libhildon dependencies, but the problems aren't just limited to that library [http://lists.maemo.org/pipermail/maemo-developers/2010-March/025668.html].
Both Niels and the Council believe we should do something to assist.
Both Niels and the Council believe we should do something to assist.
Line 13: Line 10:
== What we should do ==
== What we should do ==
-
We *need* to do something, both to improve the situation in -devel and
+
We *need* to do something, both to improve the situation in [[extras-devel]] and [[extras-testing]] today, and test an approach for the next upgrade.
-
-testing today, and test an approach for the next upgrade.
+
The main requirements here are, I think:
The main requirements here are, I think:
Line 25: Line 21:
will be much appreciated.
will be much appreciated.
 +
 +
== Progress ==
 +
 +
Fremantle autobuilder has been switched to the Squeeze devkit; idea [[#Use the improved dpkg-shlibdeps]] has been implemented. [http://lists.maemo.org/pipermail/maemo-developers/2010-April/025870.html] [http://lists.maemo.org/pipermail/maemo-developers/2010-April/025905.html]. -- 21:50, 16 April 2010 (UTC)
== Options ==
== Options ==
Line 30: Line 30:
=== Deploy SDKs as they are released; treat -devel and -testing as trunk ===
=== Deploy SDKs as they are released; treat -devel and -testing as trunk ===
-
This is what we have now, I think we can say it doesn't work if
+
This is what we have now, I think we can say it doesn't work if there's going to be more than a few days between SDK release and device upgrade. Since Nokia doesn't pre-announce release dates, and last minute bugs could cause problems; I think we can rule this one out.
-
there's going to be more than a few days between SDK release and
+
-
device upgrade. Since Nokia doesn't pre-announce release dates, and
+
-
last minute bugs could cause problems; I think we can rule this one
+
-
out.
+
=== Revert the builder ===
=== Revert the builder ===
-
This is basically a step backwards. "Changing the builder config is
+
This is basically a step backwards. "Changing the builder config is easy. Cleaning up -devel and rebuilding the affected packages is quite some work as we don't have any scripts for that made yet."
-
easy. Cleaning up -devel and rebuilding the affected packages is quite
+
-
some work as we don't have any scripts for that made yet."
+
=== Hack the SDK, create some kind of hybrid ===
=== Hack the SDK, create some kind of hybrid ===
-
This'd be bad enough if limited to just libhildon, but isn't viable
+
This'd be bad enough if limited to just libhildon, but isn't viable and WILL cause unforeseen problems. Veto :-)
-
and WILL cause unforeseen problems. Veto :-)
+
=== Create separate repos, build queues for pre- and post-1.2 ===
=== Create separate repos, build queues for pre- and post-1.2 ===
-
Similar to what's been done for Extras proper. "Creating the repos
+
Similar to what's been done for Extras proper. "Creating the repos will be about a day work, but the part that sits on top of that (management scripts, Packages interface, QA queue) will probably take a week of work."
-
will be about a day work, but the part that sits on top of that
+
-
(management scripts, Packages interface, QA queue) will probably take
+
-
a week of work."
+
There are hard issues around QA here.
There are hard issues around QA here.
Line 58: Line 48:
=== Try building in each SDK in turn ===
=== Try building in each SDK in turn ===
-
My suggestion; when someone uploads to "fremantle" auto-builder it
+
My suggestion; when someone uploads to "fremantle" auto-builder it attempts to build the package against the PR1.1 SDK. If it succeeds, it goes into Extras-devel. If it fails to build, it gets built with the PR1.2 SDK, and so on.
-
attempts to build the package against the PR1.1 SDK. If it succeeds,
+
-
it goes into Extras-devel. If it fails to build, it gets built with
+
-
the PR1.2 SDK, and so on.
+
-
For an app with compile time symbol resolution (e.g. C/C++), this'd
+
For an app with compile time symbol resolution (e.g. C/C++), this'd handle both cases quite nicely. Python apps would have to depend on the specific versions of bindings which gave them the features they wanted - again, not too much of a problem.
-
handle both cases quite nicely. Python apps would have to depend on
+
-
the specific versions of bindings which gave them the features they
+
-
wanted - again, not too much of a problem.
+
-
At extras(-stable) promotion time you could even decide, based on
+
At extras(-stable) promotion time you could even decide, based on actual binary package dependencies, whether to put in fremantle-1.2 or both fremantle-1.2 & fremantle extras repos. This would fix another common complain, "what if I don't upgrade for a few weeks".
-
actual binary package dependencies, whether to put in fremantle-1.2 or
+
-
both fremantle-1.2 & fremantle extras repos. This would fix another
+
-
common complain, "what if I don't upgrade for a few weeks".
+
-
Larger packages could be prevent from being tried to built twice by
+
Larger packages could be prevent from being tried to built twice by stating a forced build dependency on a PR1.2 version of any system package.
-
stating a forced build dependency on a PR1.2 version of any system
+
-
package.
+
-
Some software won't install from -devel and -testing as it does now,
+
Some software won't install from -devel and -testing as it does now, but that will be reduced to the set of software which is actually using features that are unavailable. A HAM patch could grey out applications which were uninstallable, or show some other indication.
-
but that will be reduced to the set of software which is actually
+
-
using features that are unavailable. A HAM patch could grey out
+
-
applications which were uninstallable, or show some other indication.
+
-
This is, effectively, reinventing the more intelligent dpkg-shlibdeps:
+
This is, effectively, reinventing the more intelligent [http://wiki.debian.org/Projects/ImprovedDpkgShlibdeps dpkg-shlibdeps].
-
 
+
-
http://wiki.debian.org/Projects/ImprovedDpkgShlibdeps
+
However, it deals with the following three circumstances:
However, it deals with the following three circumstances:
Line 91: Line 65:
#Application uses API which is unchanged in later SDK. This will be built with PR1.1 SDK, succeeds and goes into Extras-devel. Can be promoted to Extras-testing and tested on any device with PR1.1 or PR1.2. Once promoted it COULD go into fremantle and fremantle-1.2.
#Application uses API which is unchanged in later SDK. This will be built with PR1.1 SDK, succeeds and goes into Extras-devel. Can be promoted to Extras-testing and tested on any device with PR1.1 or PR1.2. Once promoted it COULD go into fremantle and fremantle-1.2.
#Application uses API which is introduced in later SDK. This will be built with PR1.1 SDK and fail. It will    be rebuilt with PR1.2 SDK, succeed and go into Extras-devel. Can be promoted to Extras-testing and tested by testers using the most recent release. Once promoted it will go into fremantle-1.2.
#Application uses API which is introduced in later SDK. This will be built with PR1.1 SDK and fail. It will    be rebuilt with PR1.2 SDK, succeed and go into Extras-devel. Can be promoted to Extras-testing and tested by testers using the most recent release. Once promoted it will go into fremantle-1.2.
 +
 +
=== Use the improved dpkg-shlibdeps ===
 +
 +
This is a big topic and as such might require more careful consideration, but seems way more future proof than the rest of options.
 +
 +
Stuff that would need to be done:
 +
#Add the debian-squeeze devkit to the autobuilder. At this point I don't know if it's usable, or not. The Lenny devkit does not contain dpkg-shlibdeps.
 +
#Ensure required .symbols files are in the SDK. We could either
 +
#*Fix upstream. Make them generate .symbols files, make them use the updated devkit. Probably impossible for PR1.2.
 +
#*Ship symbol files ourselves in an external package. A bit ugly, requires maintenance.
 +
#Put the built packages into a common -devel repo, or specific -devel-1.0 and -devel-1.2 repositories?
 +
#If single -devel & -testing repositories, decide whether to put packages in -1.0 or -1.2 at -stable promotion time.
 +
 +
Some cons:
 +
*Just enabling debian-squeeze devkit in autobuilder might cause problems. Need to check on that. Might need testing.
 +
*Need to ensure that local SDK installations can replicate the autobuilder setup. This means debian-squeeze devkit, and .symbols files have to be distributed.
 +
 +
A nice added benefit is that we get updated debhelper in the SDK, and more Debian compatibility; might be worth to push.
 +
 +
==== Initial testing ====
 +
 +
There will be a temporary autobuilder with the debian-squeeze devkit installed where we will experiment with .symbols files. We have thrown a few packages to it with no visible problems so far; if someone thinks his package might break when built under a newer Debian tooling please say so before it breaks.
 +
 +
.symbols file generation is currently being done externally, using the publicly released library binaries and dpkg-gensymbols. A sample PR1.2 libhildon1 .symbols file looks like [http://depot.javispedro.com/nit/syms/libhildon1.symbols this], and will be placed in /etc/dpkg/symbols/ .
=== Case-by-case basis ===
=== Case-by-case basis ===

Latest revision as of 08:37, 11 May 2010

Image:Ambox_notice.png
 This task is completed and this page stays here for documentation purposes only. Please file bugs or propose a new, related task if you want to push this forward.
Please see the talk page for discussion.

Contents

[edit] Background

After the PR1.2 SDK was released, Niels deployed it to the autobuilder; enacting a plan which was discussed on this list. This has caused problems with libhildon dependencies, but the problems aren't just limited to that library [1].

Both Niels and the Council believe we should do something to assist.


[edit] What we should do

We *need* to do something, both to improve the situation in extras-devel and extras-testing today, and test an approach for the next upgrade.

The main requirements here are, I think:

  • It's not an excessive amount of work
  • It's a viable long term strategy
  • The QA process doesn't get broken

Thoughts and comments from developers, or anyone else with a idea, will be much appreciated.


[edit] Progress

Fremantle autobuilder has been switched to the Squeeze devkit; idea #Use the improved dpkg-shlibdeps has been implemented. [2] [3]. -- 21:50, 16 April 2010 (UTC)

[edit] Options

[edit] Deploy SDKs as they are released; treat -devel and -testing as trunk

This is what we have now, I think we can say it doesn't work if there's going to be more than a few days between SDK release and device upgrade. Since Nokia doesn't pre-announce release dates, and last minute bugs could cause problems; I think we can rule this one out.

[edit] Revert the builder

This is basically a step backwards. "Changing the builder config is easy. Cleaning up -devel and rebuilding the affected packages is quite some work as we don't have any scripts for that made yet."

[edit] Hack the SDK, create some kind of hybrid

This'd be bad enough if limited to just libhildon, but isn't viable and WILL cause unforeseen problems. Veto :-)

[edit] Create separate repos, build queues for pre- and post-1.2

Similar to what's been done for Extras proper. "Creating the repos will be about a day work, but the part that sits on top of that (management scripts, Packages interface, QA queue) will probably take a week of work."

There are hard issues around QA here.

[edit] Try building in each SDK in turn

My suggestion; when someone uploads to "fremantle" auto-builder it attempts to build the package against the PR1.1 SDK. If it succeeds, it goes into Extras-devel. If it fails to build, it gets built with the PR1.2 SDK, and so on.

For an app with compile time symbol resolution (e.g. C/C++), this'd handle both cases quite nicely. Python apps would have to depend on the specific versions of bindings which gave them the features they wanted - again, not too much of a problem.

At extras(-stable) promotion time you could even decide, based on actual binary package dependencies, whether to put in fremantle-1.2 or both fremantle-1.2 & fremantle extras repos. This would fix another common complain, "what if I don't upgrade for a few weeks".

Larger packages could be prevent from being tried to built twice by stating a forced build dependency on a PR1.2 version of any system package.

Some software won't install from -devel and -testing as it does now, but that will be reduced to the set of software which is actually using features that are unavailable. A HAM patch could grey out applications which were uninstallable, or show some other indication.

This is, effectively, reinventing the more intelligent dpkg-shlibdeps.

However, it deals with the following three circumstances:

  1. Application uses API which has changed in later SDK. This will be built with PR1.1 SDK, succeeds and goes into Extras-devel. Can be promoted to Extras-testing but there's no clear way for a tester to know it won't work if their device is running the latest OS.
  2. Application uses API which is unchanged in later SDK. This will be built with PR1.1 SDK, succeeds and goes into Extras-devel. Can be promoted to Extras-testing and tested on any device with PR1.1 or PR1.2. Once promoted it COULD go into fremantle and fremantle-1.2.
  3. Application uses API which is introduced in later SDK. This will be built with PR1.1 SDK and fail. It will be rebuilt with PR1.2 SDK, succeed and go into Extras-devel. Can be promoted to Extras-testing and tested by testers using the most recent release. Once promoted it will go into fremantle-1.2.

[edit] Use the improved dpkg-shlibdeps

This is a big topic and as such might require more careful consideration, but seems way more future proof than the rest of options.

Stuff that would need to be done:

  1. Add the debian-squeeze devkit to the autobuilder. At this point I don't know if it's usable, or not. The Lenny devkit does not contain dpkg-shlibdeps.
  2. Ensure required .symbols files are in the SDK. We could either
    • Fix upstream. Make them generate .symbols files, make them use the updated devkit. Probably impossible for PR1.2.
    • Ship symbol files ourselves in an external package. A bit ugly, requires maintenance.
  3. Put the built packages into a common -devel repo, or specific -devel-1.0 and -devel-1.2 repositories?
  4. If single -devel & -testing repositories, decide whether to put packages in -1.0 or -1.2 at -stable promotion time.

Some cons:

  • Just enabling debian-squeeze devkit in autobuilder might cause problems. Need to check on that. Might need testing.
  • Need to ensure that local SDK installations can replicate the autobuilder setup. This means debian-squeeze devkit, and .symbols files have to be distributed.

A nice added benefit is that we get updated debhelper in the SDK, and more Debian compatibility; might be worth to push.

[edit] Initial testing

There will be a temporary autobuilder with the debian-squeeze devkit installed where we will experiment with .symbols files. We have thrown a few packages to it with no visible problems so far; if someone thinks his package might break when built under a newer Debian tooling please say so before it breaks.

.symbols file generation is currently being done externally, using the publicly released library binaries and dpkg-gensymbols. A sample PR1.2 libhildon1 .symbols file looks like this, and will be placed in /etc/dpkg/symbols/ .

[edit] Case-by-case basis

Developer complains: add a temporary exception to autobuilder to build $APPNAME with PR1.0/1.1 SDK, but goes into the shared extras-devel as now.

Retrieved from "http://wiki.maemo.org/Task:PR1.2_autobuilder"