Running Scratchbox in KVM

= Running Scratchbox in KVM =

Preface
This is currently (mostly) a personal notes page, but you're welcome to edit, comment and make corrections or additions to it. Hopefully (and after some editing) it might become a complete HOWTO guide at some point.

This is still a work in progress, meaning that I don't have a working installation yet. (I don't expect any problems in the setup though, I just have had limited time to put into this.)

I'm also a KVM newbie, so I could be doing some things wrong or in an unnecessarily complicated way. (Being new to this is also the reason why I decided to start writing down what I do; otherwise I'd have no way to backtrack and repeat my steps if I screw up somewhere along the way...)

Why run scratchbox in KVM?
There are several reasons why you might want to do this:
 * You might be running a 64-bit distribution, and be uncomfortable force-installing 32-bit packages on it (or at least unwilling to reboot in order to set the vdso32=0 parameter which is apparently required for the current Maemo 5 SDK)
 * You might not like installing software that hasn't gone through your distro's usual package management and QA process
 * You might want to be able to snapshot your environment when it's ready, to be able to reinstall easily and/or migrate it to another machine.

There are several alternatives which might also work for you:
 * The official VMware image, which is cross-platform and works out of the box. However, you might not like the audit rights clause of the license.
 * Other virtualization solutions such as Xen. Xen is open source but (unlike KVM) not part of the mainline linux kernel.
 * User-mode linux might work - I think there was some discussion about this on maemo-developers, but I'm not sure what's the status of that.
 * A simple chroot environment with ia32-libs, but then we're already somewhat drifting from the "why" part above.
 * Or using a separate, standalone, non-virtualized installation would of course work too, if you have a machine to spare for that or don't mind rebooting, but that's beside the point of this article.

Requirements
Firstly, your CPU must support virtualization. This can be checked with: $ egrep '(vmx|svm)' /proc/cpuinfo

Second, you need the modules kvm and either kvm-amd or kvm-intel available and loaded.

Third, and assuming a Debian system here, install the following: (YMMV; there are other ways to setup KVM, and other distros might package the tools differently, but this is what we're doing here.)
 * 1) apt-get install kvm virt-manager

Fourth, and finally, add yourself to the 'libvirt' group: and either log out of your X session and re-login, or (as I prefer to do), $ ssh -X localhost and do rest of your user-privileged setup in that terminal.
 * 1) adduser `whoami` libvirt

Installation
Also refer to the KVM documentation.

Network
As we're not setting up a virtual server farm here, you probably won't want to bridge your physical network interface; a virtual network will do just fine.

At least Debian kvm comes with a default virtual network predefined; start it with
 * 1) virsh net-start default

This will create a virtual virbr0 device and setup iptables forwarding rules for it. Those should be mostly fine, but you may want to check them.

Next, you'll probably want to run a DHCP server to provide an address for your virtual host(s). Assuming the server in the 'dhcp' package is used, add virbr0 to the served interfaces in /etc/default/dhcp, and add a subnet definition to /etc/dhcpd.conf - e.g. subnet 192.168.122.0 netmask 255.255.255.0 { range 192.168.122.2 192.168.122.254; option routers 192.168.122.1; option subnet-mask 255.255.255.0; option broadcast-address 192.168.122.255; option domain-name-servers 192.168.122.1; option netbios-name-servers 192.168.122.1; } (you can check the proper values with virsh net-dumpxml default). Oh, and yes I run my own DNS server too; you might try leaving out the domain-name-servers line if you don't. I guess probably the virbr0 forwarding rules would handle that with the outside world. The netbios-name-servers line is there just in case I add Win* virtual machines to the network later, and is not really needed here.

Now we're all set, proceed to:

Creating the virtual machine, option one
There are at least two ways to initialize the virtual machine. The first one I tried was - if my memory (er, shell history) serves me right: $ virt-install --connect qemu:///system -n sb -r 1024 --arch i686 --vcpus 1 --os-type linux --os-variant debianlenny -v --virt-type kvm --accelerate --disk path=/home/juha/kvm/sb.qcow2,size=20,sparse=true --disk path=/dev/cdrom,device=cdrom,bus=ide,perms=ro --network network=default --keymap fi --vnc --noautoconsole These options are somewhat superfluous (some of them would be the default anyway, but I was working through the man page and learning this stuff at the same time), but anyway this creates a 32-bit (i686) virtual machine named 'sb', with one virtual cpu, 1024 megs of ram, and 20 gigs of disk. I'm not sure what the os-type and os-variant options do exactly, but I assume they may enable some settings to be optimized for the particular OS flavor.

This also sets up vnc for the machine, with Finnish keymap (you may want to change that), but doesn't open the console automatically (I prefer to view it with virt-manager).

Note that we defined the cdrom with --disk instead of the usual --cdrom. This is to be able to give a bus=ide option instead of the usual virtio driver. For some reason kvm fails to boot from cdrom unless emulating the ide bus. Some quick googling revealed that it would - apparently - expect a 'boot=on' parameter which the current Debian virt-install fails to provide. You could try to patch the scripts to fix this, or you can just use 'ide' as above. Virtio is expected to be faster, but you probably won't be using the cdrom for much else than installation anyway.

Also note that you could as well give a path to a bootable ISO image, but I happened to have a Deian-Lenny install CD lying around, so I used that.

Creating the virtual machine, option two
I actually didn't finish the installation with the above command line, deciding instead to install newer versions of kvm and the tools from unstable - to check if the booting issue would be fixed. It wasn't, but with virt-manager 0.8.0-2 it's possible to create virtual machines through the GUI. (Actually I'm not sure if the version I had from 'testing' might have included that as well, but if it did, I didn't notice. Like said these are my installation notes, please comment or fix if what I'm saying is not correct.)

If you already created the virtual machine before and want to destroy and redefine it, this can be done with virsh destroy sb virsh undefine sb

So off to the virt-manager GUI. Most of the setup is pretty straightforward and won't be covered here in detail, but unfortunately this has a couple of gotchas too:
 * The GUI insists on creating your disk image in /var/lib/libvirt/images. This may be a problem if your /var is on a separate partition, and doesn't have enough space free.
 * To work around this, I bind-mounted $HOME/kvm there (a symlink didn't seem to do the trick, for some reason). Unfortunately the GUI still tries to be a bit too smart and limits the maximum image size to the space available on /var, so additionally you'll have to grep the python scripts included in the '*virt*' packages for the above path, and replace it with the real physical path to your homedir. (I could be wrong, but doing only this also didn't seem to work without the bind-mount.)
 * A better workaround could be selecting to use an existing image file, if you have one from a previous virt-install attempt or can figure out the right command to create it otherwise, but I didn't try that...
 * Unless the cdrom booting issue has been fixed by the time you're doing this, your virtual machine will start but you will be greeted with the "No bootable device" message on the vnc console.
 * To fix this, shut down (or force off) the virtual machine, and enter "virsh edit sb". Find the cdrom definition in the XML and change its bus attribute from 'virtio' to 'ide'. While you're at it, you might replace the path for the disk image with the real location, so you won't need to keep the bind-mount around. Save the file and off you go.

You should now have a working virtual machine installation. Insert the installation cdrom, check that the Boot Device in virt-manager "Details" page is set to CDROM (don't forget to Apply the change if it's not), and go.

Installing the OS
There shouldn't be anything special or noteworthy here. Just a plain normal Debian installation (you may need to change the virt-manager Boot Device back to Hard Disk after the installation is complete).

I realized afterwards that I probably allocated way too much disk space to the virtual machine. My intention is to keep all my source code on the host machine and do all editing there, just mounting the directories to the virtual machine using some network filesystem, so I won't be needing much space on the virtual side.

This time I also installed the full Debian system in the virtual machine, complete with the GNOME desktop, OpenOffice, GIMP, Iceweasel, etc. which won't be needed on the virtual development box. So if I do this again, I'll probably use a much smaller virtual disk, and only install a minimal Debian system (and use ssh to complete the setup, instead of using the full desktop environment over vnc).

Installing the SDK
With the OS in place, there shouldn't be anything unusual about the SDK installation either, but I didn't get that far yet. Up to now I had enough to do just learning to use KVM and working around the various bugs in the virtualization tools, and decided to stop here for today and write those steps down. If someone else has time to continue this before I do, please go ahead :-)

Setting up a network filesystem
To access sources stored on the host machine. TODO.

Setting up Xephyr
TODO.

OpenGL with the virtual setup
TODO. Possible?

Finetuning
Putting the necessary steps from above into scripts so that the setup is easy to restart after a reboot. Maybe other scripts or tools to make the initial setup easier, or even a ready-to-run SDK image, if some kind person at Nokia or in the community decides to create such things? TODO.