Maemo.org Infrastructure/Autobuilder
Contents |
[edit] Autobuilder and friends
maemo autobuilder setup
autobuilder consists of multiple VMs
[edit] drop VM
this VM has /etc/passwd synchronised with garage and ~ folders mounted via NFS from garage
account synchronisation is handled by scripts running on garage VM and then sync is triggered using ssh and scripts in /usr/local/bin
packages are uploaded to /mnt/incoming-builder via SCP
[edit] garage VM
this is the VM where stuff happens
password/account sync to gforge/postgresql is done using
*/10 * * * * root /usr/local/bin/add_groups_users_git_ssh.sh > /tmp/add_groups_users_git_ssh.log dev/null 2>&1
this also updates ~/.ssh/authorized_keys
garage also handles web extras-uploader (/var/lib/extras-assistant/) - package is uploaded and then moved to the same folder as packages uploaded to drop and then chowned using
/var/lib/extras-assistant/bin/copy_package_files_to_autobuilder.sh
A lot of jobs on garage VM is done using local root crontab (/var/spool/cron/crontabs/root)
after package is uploaded it's processed by buildME
buildME runs as builder user and it's started from cron every minute
* * * * * builder /home/builder/buildme
buildme is configured using /etc/buildme.conf
buildme takes care of couple things
- verify that .tar.gz and other files are correct (checked using checksum from .dsc file)
- select free destination (buildme can handle parallel builds on multiple hosts/users)
- scp all required files to selected destination
- start sbdmock on the destination
- copy results back and resulting .deb to repository incoming folder (result_dir = /mnt/builder/%(product)s and repo_queue = /mnt/incoming/extras-devel/%(product)s/)
- send emails to list and user uploading package
[edit] builder VM
this VM has standard installation of scratchbox with no targets configured (it's not required for sbdmock)
when sbdmock is started it cleans up old build folder, creates new target and prepares build enviroment and then runs dpkg-buildpackage
sbdmock also generates logfiles that are parsed by buildme
[edit] repository/stage VM
this is where repository management happens
*/2 * * * * repository /home/repository/queue-manage-extras-devel.sh */5 * * * * repository /home/repository/queue-manage-extras.sh */5 * * * * repository /home/repository/queue-manage-community-testing.sh */5 * * * * repository /home/repository/queue-manage-community.sh
those scripts (and scripts inside /home/repository/queue-manager-extras) check for new packages in repository incoming folder and then move those to /var/repository/staging, regenerate Packages
(using sums that were previously cached) and sign it if required and then if any changes happened
#touch .changed file, so we know that we need to sync to live touch /var/repository/staging/community/.$dist.changed
this file is then checked by
1003 10634 1 0 Mar18 ? 00:00:00 /bin/sh /usr/local/bin/packages/rqp.sh
started by /etc/init.d/repository-qp
this script starts rsync when required to sync to live repository
this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)
[edit] maintenance
builder@garage:~$ ./buildme
>>>> /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3-1.dsc
[2013-11-23 18:22:39] REJECTED: "Can't stat /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3.orig.tar.gz: No such file or directory"
[2013-11-23 18:22:39] Unexpected error:
Error'>: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/buildlib/app.py", line 81, in run
mainfunc(argv, options, self._logger, self.conf)
File "./buildme", line 641, in main
'conf' : conf
File "/usr/lib/python2.7/dist-packages/buildlib/fsm.py", line 72, in run
code = handler(self)
File "/usr/lib/python2.7/dist-packages/buildlib/helpers.py", line 82, in wrapper
rc = func(*args)
File "./buildme", line 341, in setup_build
reject_fname(env, exobj)
File "./buildme", line 193, in reject_fname
shutil.move(name, rejected_dir)
File "/usr/lib/python2.7/shutil.py", line 289, in move
raise Error, "Destination path '%s' already exists" % real_dst
Error: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
builder@garage:~$ ls -al /mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc