Tag Archives: nova

OpenStack Nova Upgrade & RabbitMQ

Leave a reply

I had occasion to upgrade my Contrail cluster from 1.03 -> 1.04. While this was hampered by network issues in the lab (the witch hunt for the guilty party messing with cables is ongoing) I came out of it with a problem on one of the two compute hosts (the working one wasn’t upgraded due to the network issue).

/var/log/nova/compute.log provided this little gem:

2014-04-03 14:36:38.068 ERROR nova.openstack.common.rpc.impl_qpid [req-9becc8bb-e75a-4700-b289-a188a98a2645 None None] Unable to connect to AMQP server: client: 0-10, server: 9-1. Sleeping 60 seconds

As I understand it, RabbitMQ only supports 0-9-1 (running on the controller), but it looks like the client (vRouter side) is running 0-10-0 (QPID) – these are not compatible. Turns out on that the OpenStack side openstack-nova-compute was now configured to use QPID rather then RabbitMQ:

rpc_backend = nova.openstack.common.rpc.impl_qpid
...
qpid_hostname = x.y.x.230

Edit /etc/nova/nova.conf to have:

rpc_backend = nova.openstack.common.rpc.impl_kombu
...
rabbit_host = x.y.x.230

and restart the openstack-nova-compute service and we are in business.

Something else worth noting is that if you switch the RPC backend and start getting:

2014-04-04 13:26:28.659 ERROR nova.openstack.common.rpc.common [req-f0ea9acc-f6e0-46e2-81f6-1e4b23f0bd2a None None] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.

in your log (multi-node setup here with separate messaging server) then you have made the mistake, as I did, of assuming that the rabbit_host at the end of the file is parsed – it isn’t! Make sure your configuration is before the [keystone_authtoken] block e.g.

volume_api_class = nova.volume.cinder.API
enabled_apis=ec2,osapi_compute,metadata
auth_strategy=keystone
#qpid_hostname = x.y.z.230
glance_host = x.y.z.230
quantum_url_timeout = 300
novncproxy_base_url = http://10.3.8.230:5999/vnc_auto.html
vncserver_enabled = true
vncserver_listen = x.y.z.231
vncserver_proxyclient_address = x.y.z.231
security_group_api = quantum
heal_instance_info_cache_interval = 0
libvirt_cpu_mode = none
rabbit_host = x.y.z.230

[keystone_authtoken]
admin_tenant_name = service
admin_user = nova
admin_password = 154c7f845c59fe7b9a78
auth_host = x.y.z.230
auth_port = 35357
auth_protocol = http
signing_dir = /tmp/keystone-signing-nova

Checking the current version of Nova

Leave a reply

It seems there is nothing along the lines of /etc/release to tell us what version of nova-compute we are actually running, but you can extract the required information. On the compute host:

[root@compute-1 ~]# nova-manage shell python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from nova import version
>>> version.version_string()
'2013.1'
>>>

One Step Forward and Two Steps Sideways

Leave a reply

A day of challenges and frustrations today

some as a result of my own carelessness, others because of the nature of a shared lab environment, and others still because technology hates us sometimes

A slow but steady start – sometimes the nested hypervisor matters

My aim was to get Contrail/OpenStack running for some testing today. Given that I don’t have a few baremetal servers lying around, I did what most people do, and decided to run my hosts on VMware (great tool). This can occasionally cause issues as I discovered later, but is great to get a few machines up and running in a hurry. I don’t really have issue with the idea of running Cloud Computing services on a virtualised environment, though a few did roll their eyes 🙂

I decided (having previously played with single-node installation) to go for a 3 machine setup, 2 compute nodes and 1 controller/openstack services. The reason for this was to have the compute hosts on physically separated machines to allow me to capture wire traffic between VMs and get a feel for the underlay networking and test some things there. Initial install went smoothly, as did installation of images onto Glance – word to the wise, make sure you have a large enough partition for the images! I ended up having to add a new partition to store a large image:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
12G   5.5G 5.6G 50%   /
tmpfs     12G   0    12G   0%   /dev/shm
/dev/sda1 485M  37M  423M  9%   /boot
/dev/sdb1 30G   7.2G 21G  26%   /mnt/glance-storage

Stopping glance, copying the contents across to /mnt/glance-storage and editing /etc/glance/glance-api.conf to point to the new location, followed by restarting glance.

# ============ Filesystem Store Options ========================
#Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir = /mnt/glance-storage/

Large images enabled I then tried to fire up a tiny CIRROS image, to no avail. Fortunately, and surprisingly, /var/log/nova/conpute.log (on the compute node) offered a clue:

libvirtError: internal error no supported architecture for os type ‘hvm’

This rather helpful error indicates that the libvirt through which we control the OpenStack hypervisor is not available to run. Contrail aims to use KVM as the default hypervisor, though getting KVM to run when the base OS is hosted on ESXi requires a little more attention then I expected – next post. As a quick workaround, I simply told nova to use QEMU instead by modifying /etc/nova/nova.conf and specifying the libvirt type.

network_api_class = nova.network.quantumv2.api.API
libvirt_vif_driver = contrail_vif.contrailvif.VRouterVIFDriver
libvirt_type=qemu

A quick restart of nova services and we were in business 🙂 or so I thought

The attack of the larger then m1.tiny instance

OpenStack has the concept of virtual machine flavors – these essentially define the capacity of a VM in terms of CPU, Memory, Disk Space, etc.

The default flavors are:

  • m1.tiny (1 VCPU/0 GB Disk/512 MB RAM)
  • m1.small (1 VCPU/20 GB Disk/2048 MB RAM)
  • m1.medium (2 VCPU/40 GB Disk/4096 MB RAM)
  • m1.large (4 VCPU/80 GB Disk/8192 MB RAM)
  • m1.xlarge (8 VCPU/160 GB Disk/16384 MB RAM)

Imagine my surprise when it turned out that only a tiny instance could be booted. In the end the cause was obvious, in that the flavors causing issues were those allocated disk space.
The compute hosts require sufficient local storage to hold the virtual machine disk space (root and ephemeral – this is not the VM image in glance e.g. the OS) for all their running instances. This was a problem as I just allocated initial disks of ~24Gb to my compute hosts (datastore space can be limited in a shared lab, and wars have been fought over less).

Nova (compute service) actually stashes the running disk images in /var/lib/nova/instances/<instance-id>. The parameter which I think governs this path is set in /etc/nova/nova.conf  (I believe this to be state_path = /var/lib/nova though the documentation is fuzzy on that) though rumours reached me that very bad things happen to those who play with the paths in nova.conf 🙁

Apparently the preferred approach is to create a huge local disk partition (or just allocated a big disk) and mount it at /var/lib/nova – so that’s what I did

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
12G  3.5G 7.5G  32%  /
tmpfs      12G  0    12G   0%   /dev/shm
/dev/sda1  485M 37M  423M  9%   /boot
/dev/sdb1  99G  4.2G 90G   5%   /var/lib/nova

And added it into /etc/fstab so it survives the many reboots that a lab requires

/dev/sdb1 /var/lib/nova ext3 defaults 1 2

And with that instances of all sizes could roam freely 🙂

More on getting KVM working as the hypervisor in the next post!