Distcc

Contents

Distributed Remote-Distributed-Compile from the n900 to native armel targets

For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as:

n900-chroot#  The easydebian chroot prompt (you can ssh in too if you like)
host-root#    The high-power desktop system to run the ARM VM's under qemu
guest-root#   The ARM VM's themselves.

1. Install Easy Debian

Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the n900. Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.


2. Install a few QEMU VMs

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

Lenny armel qemu images

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system (usually apt-get install qemu and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
   -initrd initrd.img-2.6.26-1-versatile \
   -hda debian_lenny_armel_small.qcow2 \
   -append "root=/dev/sda1"


3. Install the required packages on the VM

First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:

guest-root# apt-get update
guest-root# apt-get upgrade
guest-root# apt-get install build-essential ccache distcc

4. Configure distcc and ccache

a. Add the hosts to /etc/distcc/hosts

b. Configure /etc/default/distcc with at least:

STARTDISTCC="true"
# configure your subnet
ALLOWEDNETS="127.0.0.1 192.168.0.0/16"

c. create a ccache/distcc wrapper

root# mv /usr/bin/gcc /usr/bin/gcc-orig
root# cat<<EOT > /usr/bin/gcc-ccache
#!/bin/sh
export CCACHE_PREFIX=distcc
# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
exec ccache gcc-orig "$@"
EOT
root# chmod 755 /usr/bin/gcc-ccache

Be sure to do this on the n900 too!

d. link the wrapper as the new gcc:

root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc

e. test the distcc-ccache-gcc wrapper:

root# gcc -version
gcc-orig: no input files
distcc[2486] ERROR: compile (null) on localhost failed

The distcc error is expected

5. Configure the network to DHCP

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

6. Halt the vm and launch multiple instances

a. Halt the guest and close the window:

guest-root# shutdown -h now

b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).

root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
#!/bin/sh
iface="$1"
bridgeName=`basename "$0" | cut -f3 -d-`
echo "bridgeName=$bridgeName iface=$1"
/sbin/ifconfig "$iface" 0.0.0.0 up
/usr/sbin/brctl addif "$bridgeName" "$iface"
EOT
chmod 755 /usr/local/bin/qemu-ifup-br0
host-root# brctl addbr br0
host-root# brctl addif br0 eth0
host-root# ifconfig br0 192.168.99.222
host-root# ip route replace default via 192.168.99.1

c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:

      host arm-vm1 {
              hardware ethernet 00:16:3e:00:00:01;
              fixed-address 192.168.99.101;        }
      host arm-vm2 {
              hardware ethernet 00:16:3e:00:00:02;
              fixed-address 192.168.99.102;        }
      host arm-vm3 {
              hardware ethernet 00:16:3e:00:00:03;
              fixed-address 192.168.99.103;        }


d. Launch the vm's:

host-root#  for i in 1 2 3; do qemu-system-arm -name armvm$i \
   -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
   -initrd initrd.img-2.6.26-1-versatile \
   -hda debian_lenny_armel_small.qcow2 \
   -append "root=/dev/sda1" \
   -net nic,macaddr=00:16:3e:00:00:0$arm \
   -net tap,script=/usr/local/bin/qemu-ifup-br0 \
   -daemonize \
   -snapshot ; done

e. The bridge listing should look something like this:

host-root# brctl show
br0		8000.001cc0b1dadf	no		eth0
							tap1
							tap2
							tap3


Run a distributed build on your n900

a. Download a recent vanilla kernel:

cd /usr/src
wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
cd linux-2.6.34
   [...]
make -j3 defconfig
*** Default configuration is based on 'versatile_defconfig'
#
# configuration written to .config
#
make -j4
   [...]

b. Open top and watch your ARM VM's spin at 100% cpu:

USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                          
root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ...
root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...


As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.

--ewheelerinc 03:27, 20 August 2010 (UTC)

Forum Thread

Click here for the forum thread on this subject

About the Author

Eric Wheeler is a Linux consultant and security professional available for hire at http://www.GlobalLinuxSecurity.com. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under Professor Jonathan Walpole.