Setting up a NetBSD/XEN guest

Preliminary Note

The XEN host system here as well as the serving NFS are assumed to be a GNU/Linux, to be able to create sparse disks or partitions (dd seek option and underlying ext4 file-system). One could still imagine using a NetBSD dom0 as long as the guest virtual disk or partition are up and running. This setup could also be adapted to QCOW2 images instead of sparse RAW images or by means of using network block devices instead of vdisks on NFS.


Fetching the to-be-shared material,

mkdir -p /data/ISO-IMAGES/
cd /data/ISO-IMAGES/
wget -O - | grep amd64.iso > NetBSD-7.1.2-amd64.iso.SHA512
wget -O - | grep amd64.iso > NetBSD-8.0_RC1-amd64.iso.SHA512
cat *.SHA512
sha512sum *amd64.iso
ln -s ... netbsd7.iso
ln -s ... netbsd8.iso

mkdir -p /data/kernels/netbsd7/
cd /data/kernels/netbsd7/
grep netbsd-INSTALL_XEN3_DOMU.gz netbsd-XEN3_DOMU.gz SHA512
sha512sum netbsd-INSTALL_XEN3_DOMU.gz

mkdir -p /data/kernels/netbsd8/
cd /data/kernels/netbsd8/
grep _DOMU.gz SHA512
sha512sum *_DOMU.gz

Brutal FFS or EXT2FS on a BSD Partition (THE RIGHT WAY)

NetBSD/XEN on FFS vdisk image

Warning: this is READ-ONLY by default on Ubuntu and Slackware as the CONFIG_UFS_FS_WRITE kernel feature is not compiled in. Fortunately we use to have that additional kernel configuration enabled!.


mkdir -p /data/guests/$guest
cd /data/guests/$guest
dd if=/dev/zero of=$guest.ffs bs=1024k count=0 seek=4096
#10GB: seek=10240

grep ^proc /proc/cpuinfo | tail -1
cat > $guest <<-EOF
kernel = "/data/kernels/netbsd7/netbsd-INSTALL_XEN3_DOMU.gz"
#kernel = "/data/kernels/netbsd7/netbsd-XEN3_DOMU.gz"
#root = "xbd0a"
memory = 256
name = "$guest"
vcpus = 2
disk = ['tap:tapdisk:aio:/data/guests/$guest/$guest.ffs,xvda,w',
vif = [ 'vifname=$guest.0' ]

Note. providing the DOS partition xvda1 instead, does not make any difference in contrary to linux guest setups. NetBSD fdisk xbd0 still shows no DOS partition at all and its size remains identical. So I simply stick with xvda.

xl create $guest -c

choose language

dmesg | grep ^xbd
fdisk xbd0
disklabel xbd0
newfs /dev/xbd0a
mount /dev/xbd0a /mnt

mkdir /cdrom
mount_cd9660 /dev/xbd1a /cdrom
cd /cdrom/amd64/binary/sets
#all sets but x* and kern*
for set in `ls -1 *.tgz | sed -rn '/^[^xk]/p'`; do
    echo -n extracting $set to /mnt...
    tar xzphfe $set -C /mnt && echo done
done; unset set

chroot /mnt
cd dev/

cd /etc
ls -l localtime
ls -l ../usr/share/zoneinfo/Europe/Moscow
ln -sf ../usr/share/zoneinfo/Europe/Moscow localtime
#ln -sf ../usr/share/zoneinfo/Europe/Paris localtime
vi rc.conf


#xen guest

cat /etc/ttys #fine already

vi /etc/fstab

/dev/xbd0a      /       ffs  rw      1 1

echo netbsdffs > /etc/myname

echo inet x.x.x.x/xx up >> /etc/ifconfig.xennet0
echo x.x.x.x netbsdffs >> /etc/hosts

echo x.x.x.x >> /etc/mygate
echo x.x.x.x gw >> /etc/hosts

echo nameserver > /etc/resolv.conf

umount /mnt
cd ~/
umount /cdrom
halt -p

You will then be able to mount the thing from the dom0 for templating or maintenance (you might avoid chrooting into it, though),

mkdir lala
mount -t ufs -o loop,ufstype=44bsd $guest.ffs lala

If you get this error,

mount: /data/guests/ufs/lala: WARNING: device write-protected, mounted read-only.

==> you did not enable UFS writes in your Linux kernel, but you can still mount it as ro.

NetBSD/XEN on EXT2FS vdisk image

Same as with FFS but the formating may be done from the dom0, then BSD partition d should be used,

dd if=/dev/zero of=$guest.ext2 bs=1024k count=0 seek=4096
mkfs.ext2 $guest.ext2

disk = ['tap:tapdisk:aio:/data/guests/$guest/$guest.ext2,xvda,w']

mount -t ext2fs /dev/xbd0d /mnt

/dev/xbd0d      /       ext2fs  rw      1 1

root = "xbd0d"

The full disk flavor (THE WRONG WAY)

mkdir -p /data/guests/$guest
cd /data/guests/$guest
dd if=/dev/zero of=$guest.disk bs=1024k count=1 seek=4096
#10GB -- seek=10240
grep ^proc /proc/cpuinfo 
cat > $guest <<-EOF
kernel = "/data/kernels/netbsd7/netbsd-INSTALL_XEN3_DOMU.gz"
#kernel = "/data/kernels/netbsd7/netbsd-XEN3_DOMU.gz"
#root = "xbd0a"
memory = 256
name = "$guest"
vcpus = 2
disk = ['tap:tapdisk:aio:/data/guests/$guest/$guest.disk,xvda,w',
vif = [ 'vifname=$guest.0' ]

Launch the installation,

cd /data/guests/$guest/
xl create $guest -c

then install NetBSD on xbd0 using cdrom at xbd1a.

When finished,

x: Exit Install System

halt -p

Ready to go

Then switch to the ending production kernel and eventually remove the ISO,

vi $guest

kernel = "/data/kernels/netbsd7/netbsd-XEN3_DOMU.gz"
root = "xbd0a"
#(eventually remove the ISO)

xl create $guest -c

Once booted, check,

dmesg | grep ^xbd

Full disk flavor XEN specific post-installation (THE WRONG WAY)

cp -pi /etc/rc.conf /etc/rc.conf.dist
vi /etc/rc.conf


cp -i /etc/ttys /etc/ttys.dist
vi /etc/ttys

console "/usr/libexec/getty Pc"         vt100   on secure
ttyE1   "/usr/libexec/getty Pc"         wsvt25  off secure
ttyE2   "/usr/libexec/getty Pc"         wsvt25  off secure
ttyE3   "/usr/libexec/getty Pc"         wsvt25  off secure


Rescuing a full disk flavored netbsd/xen guest (THE WRONG WAY)

from another guest, say rescue,

disk = ['tap:tapdisk:aio:/data/guests/rescue/rescue.disk,xvda,w',
        'tap:tapdisk:aio:/data/guests/netbsd71/NetBSD-7.1.2-amd64.iso,xvdc,r' ]

fsck /dev/rxbd1a 
mkdir /lala
mount /dev/xbd1a /lala


For some special Failover IP usage (think of Dedibox/,

vif = [ 'vifname=netbsdsec.0, mac=00:16:3e:XX:XX:XX, bridge=xenbr0' ]

Note. no proxy_arp nor ip_forward are needed, this is xen bridge, not xen routing.

And in the netbsd guest,


ifconfig xennet0 inet $FAILOVER_IP/32 up
route add -net $XEN_HOST/32 -link -cloning -iface xennet0
ping -c1 $XEN_HOST

route add -net $ISP_HOSTGW/32 -link -cloning -iface xennet0
ping -c1 $ISP_HOSTGW

route add -net $ISP_GW/32 -link -cloning -iface xennet0
ping -c1 $ISP_GW

route add default $ISP_GW -ifa $FAILOVER_IP
ping -c1

netstat -rn -f inet

cat > /etc/ifconfig.xennet0 <<-EOF
inet $FAILOVER_IP/32 up
!/sbin/route add -net $XEN_HOST/32 -link -cloning -iface xennet0
!/sbin/route add -net $ISP_GW/32 -link -cloning -iface xennet0
!/sbin/route add default $ISP_GW -ifa $FAILOVER_IP

Note. eventually add a route to the whole subnet instead of XEN_HOST + ISP_GW.

And if you need a fake switch for the guests on this only to communicate with each other,

brctl addbr dummybr0
ifconfig dummybr0 ...

or in Debian/Ubuntu terms,

vi /etc/network/interfaces

auto dummybr0
iface dummybr0 inet static
        address x.x.x.x/xx
        bridge_ports none

then into the guest config,

vif = [ 'vifname=$guest.0, bridge=xenbr0',
    'vifname=$guest.1, bridge=dummybr0' ]



An alternative is to use LVM instead of a sparse file, which makes even more sense on a NetBSD dom0 as the dd seek option is not available. However I am not sure LVM is stable enough on NetBSD and the true value-added pvmove is missing anyway. Maybe another means of managing block devices would be appropriate.

lvm pvcreate /dev/rsd1d
lvm vgcreate guestsvg "/dev/rsd1d"
lvm lvcreate -n netbsdguest1 -L 5G guestsvg