Distcc

= Distributed Remote-Distributed-Compile from the n900 to native armel targets =

For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as: n900-chroot# The easydebian chroot prompt (you can ssh in too if you like) host-root#   The high-power desktop system to run the ARM VM's under qemu guest-root#  The ARM VM's themselves.

1. Install Easy Debian
Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the n900. Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.

2. Install a few QEMU VMs
On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

Lenny armel qemu images

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system (usually apt-get install qemu and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \ -initrd initrd.img-2.6.26-1-versatile \ -hda debian_lenny_armel_small.qcow2 \ -append "root=/dev/sda1"

3. Install the required packages on the VM
First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:

guest-root# apt-get update guest-root# apt-get upgrade guest-root# apt-get install build-essential ccache distcc

4. Configure distcc and ccache
a. Add the hosts to /etc/distcc/hosts

b. Configure /etc/default/distcc with at least: STARTDISTCC="true" ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
 * 1) configure your subnet

c. create a ccache/distcc wrapper

root# mv /usr/bin/gcc /usr/bin/gcc-orig root# cat< /usr/bin/gcc-ccache export CCACHE_PREFIX=distcc export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103' exec ccache gcc-orig "$@" EOT root# chmod 755 /usr/bin/gcc-ccache
 * 1) !/bin/sh
 * 1) List the distcc hosts here; I had trouble with /etc/distcc/hosts working

Be sure to do this on the n900 too!

d. link the wrapper as the new gcc: root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc

e. test the distcc-ccache-gcc wrapper: root# gcc -version gcc-orig: no input files distcc[2486] ERROR: compile (null) on localhost failed

The distcc error is expected

5. Configure the network to DHCP
guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

6. Halt the vm and launch multiple instances
a. Halt the guest and close the window: guest-root# shutdown -h now

b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).

root# cat< /usr/local/bin/qemu-ifup-br0 iface="$1" bridgeName=`basename "$0" | cut -f3 -d-` echo "bridgeName=$bridgeName iface=$1" /sbin/ifconfig "$iface" 0.0.0.0 up /usr/sbin/brctl addif "$bridgeName" "$iface" EOT chmod 755 /usr/local/bin/qemu-ifup-br0
 * 1) !/bin/sh

host-root# brctl addbr br0 host-root# brctl addif br0 eth0 host-root# ifconfig br0 192.168.99.222 host-root# ip route replace default via 192.168.99.1

c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:

host arm-vm1 { hardware ethernet 00:16:3e:00:00:01; fixed-address 192.168.99.101;       } host arm-vm2 { hardware ethernet 00:16:3e:00:00:02; fixed-address 192.168.99.102;       } host arm-vm3 { hardware ethernet 00:16:3e:00:00:03; fixed-address 192.168.99.103;       }

d. Launch the vm's:

host-root# for i in 1 2 3; do qemu-system-arm -name armvm$i \ -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \ -initrd initrd.img-2.6.26-1-versatile \ -hda debian_lenny_armel_small.qcow2 \ -append "root=/dev/sda1" \ -net nic,macaddr=00:16:3e:00:00:0$arm \ -net tap,script=/usr/local/bin/qemu-ifup-br0 \ -daemonize \ -snapshot ; done

e. The bridge listing should look something like this: host-root# brctl show br0		8000.001cc0b1dadf	no		eth0 tap1 tap2 tap3

Run a distributed build on your n900
a. Download a recent vanilla kernel: cd /usr/src wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv cd linux-2.6.34 [...] make -j3 defconfig *** Default configuration is based on 'versatile_defconfig' # # make -j4 [...] b. Open top and watch your ARM VM's spin at 100% cpu: USER     PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND root     20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ... root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ... root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ... root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
 * 1) configuration written to .config

As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.

--ewheelerinc 03:27, 20 August 2010 (UTC)

About the Author
Eric Wheeler is a Linux consultant and security professional available for hire at http://www.GlobalLinuxSecurity.com. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under Professor Jonathan Walpole.