A day of challenges and frustrations today

some as a result of my own carelessness, others because of the nature of a shared lab environment, and others still because technology hates us sometimes

A slow but steady start – sometimes the nested hypervisor matters

My aim was to get Contrail/OpenStack running for some testing today. Given that I don’t have a few baremetal servers lying around, I did what most people do, and decided to run my hosts on VMware (great tool). This can occasionally cause issues as I discovered later, but is great to get a few machines up and running in a hurry. I don’t really have issue with the idea of running Cloud Computing services on a virtualised environment, though a few did roll their eyes 🙂

I decided (having previously played with single-node installation) to go for a 3 machine setup, 2 compute nodes and 1 controller/openstack services. The reason for this was to have the compute hosts on physically separated machines to allow me to capture wire traffic between VMs and get a feel for the underlay networking and test some things there. Initial install went smoothly, as did installation of images onto Glance – word to the wise, make sure you have a large enough partition for the images! I ended up having to add a new partition to store a large image:

Stopping glance, copying the contents across to /mnt/glance-storage and editing /etc/glance/glance-api.conf to point to the new location, followed by restarting glance.

Large images enabled I then tried to fire up a tiny CIRROS image, to no avail. Fortunately, and surprisingly, /var/log/nova/conpute.log (on the compute node) offered a clue:

This rather helpful error indicates that the libvirt through which we control the OpenStack hypervisor is not available to run. Contrail aims to use KVM as the default hypervisor, though getting KVM to run when the base OS is hosted on ESXi requires a little more attention then I expected – next post. As a quick workaround, I simply told nova to use QEMU instead by modifying /etc/nova/nova.conf and specifying the libvirt type.

A quick restart of nova services and we were in business 🙂 or so I thought

The attack of the larger then m1.tiny instance

OpenStack has the concept of virtual machine flavors – these essentially define the capacity of a VM in terms of CPU, Memory, Disk Space, etc.

The default flavors are:

Imagine my surprise when it turned out that only a tiny instance could be booted. In the end the cause was obvious, in that the flavors causing issues were those allocated disk space.
The compute hosts require sufficient local storage to hold the virtual machine disk space (root and ephemeral – this is not the VM image in glance e.g. the OS) for all their running instances. This was a problem as I just allocated initial disks of ~24Gb to my compute hosts (datastore space can be limited in a shared lab, and wars have been fought over less).

Nova (compute service) actually stashes the running disk images in /var/lib/nova/instances/<instance-id>. The parameter which I think governs this path is set in /etc/nova/nova.conf  (I believe this to be state_path = /var/lib/nova though the documentation is fuzzy on that) though rumours reached me that very bad things happen to those who play with the paths in nova.conf 🙁

Apparently the preferred approach is to create a huge local disk partition (or just allocated a big disk) and mount it at /var/lib/nova – so that’s what I did

And added it into /etc/fstab so it survives the many reboots that a lab requires

And with that instances of all sizes could roam freely 🙂

More on getting KVM working as the hypervisor in the next post!

Click to share...

Facebook

Twitter

Linkedin

Reddit

Stumbleupon

Email

Print