Maemo.org Infrastructure/Autobuilder

sixwheeledbeast (Talk | contribs)
(Moved from migration page - new page)
Newer edit →

Revision as of 20:30, 5 June 2013

Contents

Autobuilder and friends

maemo autobuilder setup

autobuilder consists of multiple VMs

drop VM

this VM has /etc/passwd synchronised with garage and ~ folders mounted via NFS from garage

account synchronisation is handled by scripts running on garage VM and then sync is triggered using ssh and scripts in /usr/local/bin

packages are uploaded to /mnt/incoming-builder via SCP

garage VM

this is the VM where stuff happens

password/account sync to gforge/postgresql is done using 

*/10 *   * * *   root    /usr/local/bin/add_groups_users_git_ssh.sh > /tmp/add_groups_users_git_ssh.log dev/null 2>&1

this also updates ~/.ssh/authorized_keys

garage also handles web extras-uploader (/var/lib/extras-assistant/) - package is uploaded and then moved to the same folder as packages uploaded to drop and then chowned using 

/var/lib/extras-assistant/bin/copy_package_files_to_autobuilder.sh

A lot of jobs on garage VM is done using local root crontab (/var/spool/cron/crontabs/root)

after package is uploaded it's processed by buildME

buildME runs as builder user and it's started from cron every minute 

* * * * * builder /home/builder/buildme

buildme is configured using /etc/buildme.conf

buildme takes care of couple things

  • verify that .tar.gz and other files are correct (checked using checksum from .dsc file)
  • select free destination (buildme can handle parallel builds on multiple hosts/users)
  • scp all required files to selected destination
  • start sbdmock on the destination
  • copy results back and resulting .deb to repository incoming folder (result_dir = /mnt/builder/%(product)s and repo_queue = /mnt/incoming/extras-devel/%(product)s/)
  • send emails to list and user uploading package

builder VM

this VM has standard installation of scratchbox with no targets configured (it's not required for sbdmock)

when sbdmock is started it cleans up old build folder, creates new target and prepares build enviroment and then runs dpkg-buildpackage

sbdmock also generates logfiles that are parsed by buildme

repository/stage VM

this is where repository management happens 

*/2 * * * *     repository      /home/repository/queue-manage-extras-devel.sh
*/5 * * * *     repository      /home/repository/queue-manage-extras.sh
*/5 * * * *     repository      /home/repository/queue-manage-community-testing.sh
*/5 * * * *     repository      /home/repository/queue-manage-community.sh

those scripts (and scripts inside /home/repository/queue-manager-extras) check for new packages in repository incoming folder and then move those to /var/repository/staging, regenerate Packages

(using sums that were previously cached) and sign it if required and then if any changes happened 

#touch .changed file, so we know that we need to sync to live
touch /var/repository/staging/community/.$dist.changed

this file is then checked by 

1003     10634     1  0 Mar18 ?        00:00:00 /bin/sh /usr/local/bin/packages/rqp.sh

started by /etc/init.d/repository-qp

this script starts rsync when required to sync to live repository

this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)

Retrieved from "https://wiki.maemo.org/Maemo.org_Infrastructure/Autobuilder"