Distcc

m (6. Halt the vm and launch multiple instances)
m (References: better author link)
 
(2 intermediate revisions not shown)
Line 1: Line 1:
-
= Distributed Remote-Distributed-Compile from the n900 to native armel targets =
+
For those who have tried to compile on the [[Nokia N900|N900]], you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, is often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the N900 platform---and it works on my phone!
-
For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel.  It, however, often useful to compile natively on the architecture you will be using.  Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!
+
This assumes familiarity with <code>qemu-system-arm</code>, <code>gcc</code> (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.
-
 
+
-
This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.
+
Where possible I distinguish the different system root-prompts as:
Where possible I distinguish the different system root-prompts as:
Line 10: Line 8:
  guest-root#  The ARM VM's themselves.
  guest-root#  The ARM VM's themselves.
-
== 1. Install Easy Debian ==
+
== Install Easy Debian ==
-
Before continuing, you will need to install [http://wiki.maemo.org/Easy_Debian Easy Debian] on your phone. This is the fastest way to get a working native gcc environment on the n900. Once it is installed, install ccache, distcc, and build-essential:
+
Before continuing, you will need to install [[Easy Debian]] on your phone. This is the fastest way to get a working native gcc environment on the N900. Once it is installed, install <code>ccache</code>, <code>distcc</code>, and <code>build-essential</code>:
  n900-chroot# apt-get install ccache distcc build-essential
  n900-chroot# apt-get install ccache distcc build-essential
-
Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.
+
Then configure <code>/usr/bin/gcc-ccache</code> the same way as [[#Configure distcc and ccache|below]] in the easydebian chroot.
-
 
+
== Install a few QEMU VMs ==
-
== 2. Install a few QEMU VMs ==
+
On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:
On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:
Line 25: Line 22:
[http://people.debian.org/~aurel32/qemu/armel/ Lenny armel qemu images]
[http://people.debian.org/~aurel32/qemu/armel/ Lenny armel qemu images]
-
I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.
+
I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.
-
 
+
-
Install qemu-arm on your system (usually apt-get install [http://www.qemu.org qemu] and boot the VM:
+
-
 
+
-
host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
+
-
    -initrd initrd.img-2.6.26-1-versatile \
+
-
    -hda debian_lenny_armel_small.qcow2 \
+
-
    -append "root=/dev/sda1"
+
 +
Install <code>qemu-arm</code> on your system, usually:
 +
apt-get install qemu
 +
and boot the VM:
 +
<pre>
 +
host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
 +
  -initrd initrd.img-2.6.26-1-versatile \
 +
  -hda debian_lenny_armel_small.qcow2 \
 +
  -append "root=/dev/sda1"
 +
</pre>
-
== 3. Install the required packages on the VM ==
+
== Install the required packages on the VM ==
-
First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it.
+
First update <code>/etc/apt/sources.list</code> for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:
-
Then add the packages:
+
  guest-root# apt-get update
  guest-root# apt-get update
Line 44: Line 42:
  guest-root# apt-get install build-essential ccache distcc
  guest-root# apt-get install build-essential ccache distcc
-
== 4. Configure distcc and ccache ==
+
== Configure distcc and ccache ==
-
 
+
-
a. Add the hosts to /etc/distcc/hosts
+
-
 
+
-
b. Configure /etc/default/distcc with at least:
+
-
STARTDISTCC="true"
+
-
# configure your subnet
+
-
ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
+
-
 
+
-
c. create a ccache/distcc wrapper
+
-
 
+
-
root# mv /usr/bin/gcc /usr/bin/gcc-orig
+
-
root# cat<<EOT > /usr/bin/gcc-ccache
+
-
#!/bin/sh
+
-
export CCACHE_PREFIX=distcc
+
-
# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
+
-
export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
+
-
exec ccache gcc-orig "$@"
+
-
EOT
+
-
root# chmod 755 /usr/bin/gcc-ccache
+
-
 
+
-
Be sure to do this on the n900 too!
+
-
 
+
-
d. link the wrapper as the new gcc:
+
-
root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
+
-
 
+
-
e. test the distcc-ccache-gcc wrapper:
+
-
root# gcc -version
+
-
gcc-orig: no input files
+
-
distcc[2486] ERROR: compile (null) on localhost failed
+
 +
<ol>
 +
<li>
 +
Add the hosts to <code>/etc/distcc/hosts</code>
 +
</li>
 +
<li>
 +
Configure <code>/etc/default/distcc</code> with at least:
 +
<pre>
 +
STARTDISTCC="true"
 +
# configure your subnet
 +
ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
 +
</pre>
 +
</li>
 +
<li>
 +
create a ccache/distcc wrapper
 +
<pre>
 +
root# mv /usr/bin/gcc /usr/bin/gcc-orig
 +
root# cat<<EOT > /usr/bin/gcc-ccache
 +
#!/bin/sh
 +
export CCACHE_PREFIX=distcc
 +
# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
 +
export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
 +
exec ccache gcc-orig "$@"
 +
EOT
 +
root# chmod 755 /usr/bin/gcc-ccache
 +
</pre>
 +
Be sure to do this on the N900 too!
 +
</li>
 +
<li>
 +
link the wrapper as the new gcc:
 +
<pre>
 +
root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
 +
</pre>
 +
</li>
 +
<li>
 +
test the distcc-ccache-gcc wrapper:
 +
<pre>
 +
root# gcc -version
 +
gcc-orig: no input files
 +
distcc[2486] ERROR: compile (null) on localhost failed
 +
</pre>
The distcc error is expected
The distcc error is expected
 +
</li>
 +
</ol>
-
== 5. Configure the network to DHCP ==
+
== Configure the network to DHCP ==
  guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
  guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
  guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces
  guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces
-
== 6. Halt the vm and launch multiple instances ==
+
== Halt the vm and launch multiple instances ==
-
a. Halt the guest and close the window:
+
<ol>
 +
<li>
 +
Halt the guest and close the window:
  guest-root# shutdown -h now
  guest-root# shutdown -h now
-
 
+
</li>
-
b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
+
<li>
-
 
+
Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the N900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
 +
<pre>
  root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
  root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
  #!/bin/sh
  #!/bin/sh
Line 103: Line 117:
  host-root# ifconfig br0 192.168.99.222
  host-root# ifconfig br0 192.168.99.222
  host-root# ip route replace default via 192.168.99.1
  host-root# ip route replace default via 192.168.99.1
-
 
+
</pre>
-
c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:
+
</li>
-
 
+
<li>
 +
Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:
 +
<pre>
       host arm-vm1 {
       host arm-vm1 {
               hardware ethernet 00:16:3e:00:00:01;
               hardware ethernet 00:16:3e:00:00:01;
Line 115: Line 131:
               hardware ethernet 00:16:3e:00:00:03;
               hardware ethernet 00:16:3e:00:00:03;
               fixed-address 192.168.99.103;        }
               fixed-address 192.168.99.103;        }
-
 
+
</pre>
-
 
+
</li>
-
d. Launch the vm's:
+
<li>
-
 
+
Launch the vm's:
 +
<pre>
  host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
  host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
     -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
     -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
Line 128: Line 145:
     -daemonize \
     -daemonize \
     -snapshot ; done
     -snapshot ; done
-
 
+
</pre>
-
e. The bridge listing should look something like this:
+
</li>
 +
<li>
 +
The bridge listing should look something like this:
 +
<pre>
  host-root# brctl show
  host-root# brctl show
  br0 8000.001cc0b1dadf no eth0
  br0 8000.001cc0b1dadf no eth0
Line 135: Line 155:
  tap2
  tap2
  tap3
  tap3
 +
</pre>
 +
</li>
 +
</ol>
-
== Run a distributed build on your n900 ==
+
== Run a distributed build on your N900 ==
-
a. Download a recent vanilla kernel:
+
<ol>
 +
<li>
 +
Download a recent vanilla kernel:
 +
<pre>
  cd /usr/src
  cd /usr/src
  wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
  wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
Line 150: Line 176:
  make -j4
  make -j4
     [...]
     [...]
-
+
</pre>
-
b. Open top and watch your ARM VM's spin at 100% cpu:
+
</li>
 +
<li>
 +
Open top and watch your ARM VM's spin at 100% CPU:
 +
<pre>
  USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                           
  USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                           
  root      20  0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
  root      20  0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
Line 157: Line 186:
  root      20  0  530m 216m 1472 R  99  3.6 qemu-system-arm -name armel-vm3 ...
  root      20  0  530m 216m 1472 R  99  3.6 qemu-system-arm -name armel-vm3 ...
  root      20  0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
  root      20  0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
 +
</pre>
 +
</li></ol>
 +
As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.
-
As you can see, I have 4 VM's on my system, one for each core on the i7.  Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.
+
You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the N900's localhost to spin the native 600 MHz CPU too---and I leave this as an exercise for the reader.
-
 
+
-
You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.
+
-
--[[User:ewheelerinc|ewheelerinc]] 03:27, 20 August 2010 (UTC)
+
== References ==
-
=== Forum Thread ===
+
* [http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]
-
[http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]
+
* [[User:Ewheelerinc|Original author]]
-
=== About the Author ===
+
[[Category:N900]]
-
Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com].  He has 14 years of experience in Linux since 1996 with a great breadth of knowledge.  Eric is currently working on a PhD in Computer Science at Portland  State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].
+
[[Category:Software]]
 +
[[Category:Development]]

Latest revision as of 08:38, 20 August 2010

For those who have tried to compile on the N900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, is often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the N900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as:

n900-chroot#  The easydebian chroot prompt (you can ssh in too if you like)
host-root#    The high-power desktop system to run the ARM VM's under qemu
guest-root#   The ARM VM's themselves.

Contents

[edit] Install Easy Debian

Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the N900. Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as below in the easydebian chroot.

[edit] Install a few QEMU VMs

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

Lenny armel qemu images

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system, usually:

apt-get install qemu

and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
   -initrd initrd.img-2.6.26-1-versatile \
   -hda debian_lenny_armel_small.qcow2 \
   -append "root=/dev/sda1"

[edit] Install the required packages on the VM

First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:

guest-root# apt-get update
guest-root# apt-get upgrade
guest-root# apt-get install build-essential ccache distcc

[edit] Configure distcc and ccache

  1. Add the hosts to /etc/distcc/hosts
  2. Configure /etc/default/distcc with at least:
    STARTDISTCC="true"
    # configure your subnet
    ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
    
  3. create a ccache/distcc wrapper
    root# mv /usr/bin/gcc /usr/bin/gcc-orig
    root# cat<<EOT > /usr/bin/gcc-ccache
    #!/bin/sh
    export CCACHE_PREFIX=distcc
    # List the distcc hosts here; I had trouble with /etc/distcc/hosts working
    export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
    exec ccache gcc-orig "$@"
    EOT
    root# chmod 755 /usr/bin/gcc-ccache
    

    Be sure to do this on the N900 too!

  4. link the wrapper as the new gcc:
    root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
    
  5. test the distcc-ccache-gcc wrapper:
    root# gcc -version
    gcc-orig: no input files
    distcc[2486] ERROR: compile (null) on localhost failed
    

    The distcc error is expected

[edit] Configure the network to DHCP

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

[edit] Halt the vm and launch multiple instances

  1. Halt the guest and close the window: guest-root# shutdown -h now
  2. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the N900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
     root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
     #!/bin/sh
     iface="$1"
     bridgeName=`basename "$0" | cut -f3 -d-`
     echo "bridgeName=$bridgeName iface=$1"
     /sbin/ifconfig "$iface" 0.0.0.0 up
     /usr/sbin/brctl addif "$bridgeName" "$iface"
     EOT
     chmod 755 /usr/local/bin/qemu-ifup-br0
    
     host-root# brctl addbr br0
     host-root# brctl addif br0 eth0
     host-root# ifconfig br0 192.168.99.222
     host-root# ip route replace default via 192.168.99.1
    
  3. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:
           host arm-vm1 {
                   hardware ethernet 00:16:3e:00:00:01;
                   fixed-address 192.168.99.101;        }
           host arm-vm2 {
                   hardware ethernet 00:16:3e:00:00:02;
                   fixed-address 192.168.99.102;        }
           host arm-vm3 {
                   hardware ethernet 00:16:3e:00:00:03;
                   fixed-address 192.168.99.103;        }
    
  4. Launch the vm's:
     host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
        -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
        -initrd initrd.img-2.6.26-1-versatile \
        -hda debian_lenny_armel_small.qcow2 \
        -append "root=/dev/sda1" \
        -net nic,macaddr=00:16:3e:00:00:0$arm \
        -net tap,script=/usr/local/bin/qemu-ifup-br0 \
        -daemonize \
        -snapshot ; done
    
  5. The bridge listing should look something like this:
     host-root# brctl show
     br0		8000.001cc0b1dadf	no		eth0
     							tap1
     							tap2
     							tap3
    

[edit] Run a distributed build on your N900

  1. Download a recent vanilla kernel:
     cd /usr/src
     wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
     cd linux-2.6.34
        [...]
     make -j3 defconfig
     *** Default configuration is based on 'versatile_defconfig'
     #
     # configuration written to .config
     #
     make -j4
        [...]
    
  2. Open top and watch your ARM VM's spin at 100% CPU:
     USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                          
     root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
     root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ...
     root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
     root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
    

As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the N900's localhost to spin the native 600 MHz CPU too---and I leave this as an exercise for the reader.

[edit] References