Editing Distcc

Warning: You are not logged in. Your IP address will be recorded in this page's edit history.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Learn more about Contributing to the wiki.

= Distributed Remote-Distributed-Compile from the n900 to native armel targets =

For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel.  It, however, often useful to compile natively on the architecture you will be using.  Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as:
 n900-chroot#  The easydebian chroot prompt (you can ssh in too if you like)
 host-root#    The high-power desktop system to run the ARM VM's under qemu
 guest-root#   The ARM VM's themselves.

== 1. Install Easy Debian ==

Before continuing, you will need to install [http://wiki.maemo.org/Easy_Debian Easy Debian] on your phone.  This is the fastest way to get a working native gcc environment on the n900.  Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.

== 2. Install a few QEMU VMs ==

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

[http://people.debian.org/~aurel32/qemu/armel/ Lenny armel qemu images]

I used the _small version so I can be as specific as possible with packages.  I assume you use the _small version from here.

Install qemu-arm on your system (usually apt-get install [http://www.qemu.org qemu] and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
    -initrd initrd.img-2.6.26-1-versatile \
    -hda debian_lenny_armel_small.qcow2 \
    -append "root=/dev/sda1"

== 3. Install the required packages on the VM ==

First update /etc/apt/sources.list for the country of your choice (defaults to .de).  I recommend commenting the volatile repo unless  you intend to use it.
Then add the packages:

guest-root# apt-get update
 guest-root# apt-get upgrade
 guest-root# apt-get install build-essential ccache distcc

== 4. Configure distcc and ccache ==

a. Add the hosts to /etc/distcc/hosts

b. Configure /etc/default/distcc with at least:
 STARTDISTCC="true"
 # configure your subnet
 ALLOWEDNETS="127.0.0.1 192.168.0.0/16"

c. create a ccache/distcc wrapper

root# mv /usr/bin/gcc /usr/bin/gcc-orig
 root# cat<<EOT > /usr/bin/gcc-ccache
 #!/bin/sh
 export CCACHE_PREFIX=distcc
 # List the distcc hosts here; I had trouble with /etc/distcc/hosts working
 export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
 exec ccache gcc-orig "$@"
 EOT
 root# chmod 755 /usr/bin/gcc-ccache

Be sure to do this on the n900 too!

d. link the wrapper as the new gcc:
 root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc

e. test the distcc-ccache-gcc wrapper:
 root# gcc -version
 gcc-orig: no input files
 distcc[2486] ERROR: compile (null) on localhost failed

The distcc error is expected

== 5. Configure the network to DHCP ==

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
 guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

== 6. Halt the vm and launch multiple instances ==

a. Halt the guest and close the window:
 guest-root# shutdown -h now

b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system.  Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).

root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
 #!/bin/sh
 iface="$1"
 bridgeName=`basename "$0" | cut -f3 -d-`
 echo "bridgeName=$bridgeName iface=$1"
 /sbin/ifconfig "$iface" 0.0.0.0 up
 /usr/sbin/brctl addif "$bridgeName" "$iface"
 EOT
 chmod 755 /usr/local/bin/qemu-ifup-br0

host-root# brctl addbr br0
 host-root# brctl addif br0 eth0
 host-root# ifconfig br0 192.168.99.222
 host-root# ip route replace default via 192.168.99.1

c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03.  If you are using ISC's dhcpd, the config looks something like this:

host arm-vm1 {
               hardware ethernet 00:16:3e:00:00:01;
               fixed-address 192.168.99.101;        }
       host arm-vm2 {
               hardware ethernet 00:16:3e:00:00:02;
               fixed-address 192.168.99.102;        }
       host arm-vm3 {
               hardware ethernet 00:16:3e:00:00:03;
               fixed-address 192.168.99.103;        }

d. Launch the vm's:

host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
    -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
    -initrd initrd.img-2.6.26-1-versatile \
    -hda debian_lenny_armel_small.qcow2 \
    -append "root=/dev/sda1" \
    -net nic,macaddr=00:16:3e:00:00:0$arm \
    -net tap,script=/usr/local/bin/qemu-ifup-br0 \
    -daemonize \
    -snapshot ; done

e. The bridge listing should look something like this:
 host-root# brctl show
 br0		8000.001cc0b1dadf	no		eth0
 							tap1
 							tap2
 							tap3

== Run a distributed build on your n900 ==

a. Download a recent vanilla kernel:
 cd /usr/src
 wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
 cd linux-2.6.34
    [...]
 make -j3 defconfig
 *** Default configuration is based on 'versatile_defconfig'
 #
 # configuration written to .config
 #
 make -j4
    [...]
 
b. Open top and watch your ARM VM's spin at 100% cpu:
 USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                          
 root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
 root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ...
 root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
 root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...

As you can see, I have 4 VM's on my system, one for each core on the i7.  Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible).  This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.

--[[User:ewheelerinc|ewheelerinc]] 03:27, 20 August 2010 (UTC)

=== Forum Thread ===
[http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]

=== About the Author ===
Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com].  He has 14 years of experience in Linux since 1996 with a great breadth of knowledge.  Eric is currently working on a PhD in Computer Science at Portland  State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].

Please note that all contributions to maemo.org wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see maemo.org wiki:Copyrights for details). Do not submit copyrighted work without permission!

Summary:

Cancel | Editing help (opens in new window)

Retrieved from "http://wiki.maemo.org/Distcc"

Personal tools

Navigation

Views

Editing Distcc

@@ Line 1: / Line 1: @@
-For those who have tried to compile on the [[Nokia N900|N900]], you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, is often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the N900 platform---and it works on my phone!
+= Distributed Remote-Distributed-Compile from the n900 to native armel targets =
-This assumes familiarity with <code>qemu-system-arm</code>, <code>gcc</code> (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.
+For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel.  It, however, often useful to compile natively on the architecture you will be using.  Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!
+This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.
 Where possible I distinguish the different system root-prompts as:
@@ Line 8: / Line 10: @@
   guest-root#   The ARM VM's themselves.
-== Install Easy Debian ==
+== 1. Install Easy Debian ==
-Before continuing, you will need to install [[Easy Debian]] on your phone. This is the fastest way to get a working native gcc environment on the N900. Once it is installed, install <code>ccache</code>, <code>distcc</code>, and <code>build-essential</code>:
+Before continuing, you will need to install [http://wiki.maemo.org/Easy_Debian Easy Debian] on your phone.  This is the fastest way to get a working native gcc environment on the n900.  Once it is installed, install ccache, distcc, and build-essential:
   n900-chroot# apt-get install ccache distcc build-essential
-Then configure <code>/usr/bin/gcc-ccache</code> the same way as [[#Configure distcc and ccache|below]] in the easydebian chroot.
+Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.
-== Install a few QEMU VMs ==
+== 2. Install a few QEMU VMs ==
 On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:
@@ Line 22: / Line 25: @@
 [http://people.debian.org/~aurel32/qemu/armel/ Lenny armel qemu images]
-I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.
+I used the _small version so I can be as specific as possible with packages.  I assume you use the _small version from here.
+Install qemu-arm on your system (usually apt-get install [http://www.qemu.org qemu] and boot the VM:
+ host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
+    -initrd initrd.img-2.6.26-1-versatile \
+    -hda debian_lenny_armel_small.qcow2 \
+    -append "root=/dev/sda1"
-Install <code>qemu-arm</code> on your system, usually:
- apt-get install qemu
-and boot the VM:
-<pre>
-host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
-   -initrd initrd.img-2.6.26-1-versatile \
-   -hda debian_lenny_armel_small.qcow2 \
-   -append "root=/dev/sda1"
-</pre>
-== Install the required packages on the VM ==
+== 3. Install the required packages on the VM ==
-First update <code>/etc/apt/sources.list</code> for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:
+First update /etc/apt/sources.list for the country of your choice (defaults to .de).  I recommend commenting the volatile repo unless  you intend to use it.
+Then add the packages:
   guest-root# apt-get update
@@ Line 42: / Line 44: @@
   guest-root# apt-get install build-essential ccache distcc
-== Configure distcc and ccache ==
+== 4. Configure distcc and ccache ==
+a. Add the hosts to /etc/distcc/hosts
+b. Configure /etc/default/distcc with at least:
+ STARTDISTCC="true"
+ # configure your subnet
+ ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
+c. create a ccache/distcc wrapper
+ root# mv /usr/bin/gcc /usr/bin/gcc-orig
+ root# cat<<EOT > /usr/bin/gcc-ccache
+ #!/bin/sh
+ export CCACHE_PREFIX=distcc
+ # List the distcc hosts here; I had trouble with /etc/distcc/hosts working
+ export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
+ exec ccache gcc-orig "$@"
+ EOT
+ root# chmod 755 /usr/bin/gcc-ccache
+Be sure to do this on the n900 too!
+d. link the wrapper as the new gcc:
+ root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
+e. test the distcc-ccache-gcc wrapper:
+ root# gcc -version
+ gcc-orig: no input files
+ distcc[2486] ERROR: compile (null) on localhost failed
-<ol>
-<li>
-Add the hosts to <code>/etc/distcc/hosts</code>
-</li>
-<li>
-Configure <code>/etc/default/distcc</code> with at least:
-<pre>
-STARTDISTCC="true"
-# configure your subnet
-ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
-</pre>
-</li>
-<li>
-create a ccache/distcc wrapper
-<pre>
-root# mv /usr/bin/gcc /usr/bin/gcc-orig
-root# cat<<EOT > /usr/bin/gcc-ccache
-#!/bin/sh
-export CCACHE_PREFIX=distcc
-# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
-export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
-exec ccache gcc-orig "$@"
-EOT
-root# chmod 755 /usr/bin/gcc-ccache
-</pre>
-Be sure to do this on the N900 too!
-</li>
-<li>
-link the wrapper as the new gcc:
-<pre>
-root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
-</pre>
-</li>
-<li>
-test the distcc-ccache-gcc wrapper:
-<pre>
-root# gcc -version
-gcc-orig: no input files
-distcc[2486] ERROR: compile (null) on localhost failed
-</pre>
 The distcc error is expected
-</li>
-</ol>
-== Configure the network to DHCP ==
+== 5. Configure the network to DHCP ==
   guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
   guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces
-== Halt the vm and launch multiple instances ==
+== 6. Halt the vm and launch multiple instances ==
-<ol>
+a. Halt the guest and close the window:
-<li>
-Halt the guest and close the window:
   guest-root# shutdown -h now
-</li>
-<li>
+b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system.  Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
-Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the N900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
-<pre>
   root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
   #!/bin/sh
@@ Line 117: / Line 103: @@
   host-root# ifconfig br0 192.168.99.222
   host-root# ip route replace default via 192.168.99.1
-</pre>
-</li>
+c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03.  If you are using ISC's dhcpd, the config looks something like this:
-<li>
-Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:
-<pre>
         host arm-vm1 {
                 hardware ethernet 00:16:3e:00:00:01;
@@ Line 131: / Line 115: @@
                 hardware ethernet 00:16:3e:00:00:03;
                 fixed-address 192.168.99.103;        }
-</pre>
-</li>
-<li>
+d. Launch the vm's:
-Launch the vm's:
-<pre>
   host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
      -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
@@ Line 145: / Line 128: @@
      -daemonize \
      -snapshot ; done
-</pre>
-</li>
+e. The bridge listing should look something like this:
-<li>
-The bridge listing should look something like this:
-<pre>
   host-root# brctl show
   br0		8000.001cc0b1dadf	no		eth0
@@ Line 155: / Line 135: @@
   							tap2
   							tap3
-</pre>
-</li>
-</ol>
-== Run a distributed build on your N900 ==
+== Run a distributed build on your n900 ==
-<ol>
+a. Download a recent vanilla kernel:
-<li>
-Download a recent vanilla kernel:
-<pre>
   cd /usr/src
   wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
@@ Line 176: / Line 150: @@
   make -j4
      [...]
-</pre>
-</li>
+b. Open top and watch your ARM VM's spin at 100% cpu:
-<li>
-Open top and watch your ARM VM's spin at 100% CPU:
-<pre>
   USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND
   root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
@@ Line 186: / Line 157: @@
   root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
   root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
-</pre>
-</li></ol>
-As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.
-You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the N900's localhost to spin the native 600 MHz CPU too---and I leave this as an exercise for the reader.
+As you can see, I have 4 VM's on my system, one for each core on the i7.  Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.
+You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible).  This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.
-== References ==
+--[[User:ewheelerinc|ewheelerinc]] 03:27, 20 August 2010 (UTC)
-* [http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]
+=== Forum Thread ===
-* [[User:Ewheelerinc|Original author]]
+[http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]
-[[Category:N900]]
+=== About the Author ===
-[[Category:Software]]
+Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com].  He has 14 years of experience in Linux since 1996 with a great breadth of knowledge.  Eric is currently working on a PhD in Computer Science at Portland  State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].
-[[Category:Development]]