I had occasion to upgrade my Contrail cluster from 1.03 -> 1.04. While this was hampered by network issues in the lab (the witch hunt for the guilty party messing with cables is ongoing) I came out of it with a problem on one of the two compute hosts (the working one wasn’t upgraded due to the network issue).
/var/log/nova/compute.log provided this little gem:
2014-04-03 14:36:38.068 ERROR nova.openstack.common.rpc.impl_qpid [req-9becc8bb-e75a-4700-b289-a188a98a2645 None None] Unable to connect to AMQP server: client: 0-10, server: 9-1. Sleeping 60 seconds
As I understand it, RabbitMQ only supports 0-9-1 (running on the controller), but it looks like the client (vRouter side) is running 0-10-0 (QPID) – these are not compatible. Turns out on that the OpenStack side openstack-nova-compute was now configured to use QPID rather then RabbitMQ:
rpc_backend = nova.openstack.common.rpc.impl_qpid
...
qpid_hostname = x.y.x.230
Edit /etc/nova/nova.conf to have:
rpc_backend = nova.openstack.common.rpc.impl_kombu
...
rabbit_host = x.y.x.230
and restart the openstack-nova-compute service and we are in business.
Something else worth noting is that if you switch the RPC backend and start getting:
2014-04-04 13:26:28.659 ERROR nova.openstack.common.rpc.common [req-f0ea9acc-f6e0-46e2-81f6-1e4b23f0bd2a None None] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
in your log (multi-node setup here with separate messaging server) then you have made the mistake, as I did, of assuming that the rabbit_host at the end of the file is parsed – it isn’t! Make sure your configuration is before the [keystone_authtoken] block e.g.
volume_api_class = nova.volume.cinder.API
enabled_apis=ec2,osapi_compute,metadata
auth_strategy=keystone
#qpid_hostname = x.y.z.230
glance_host = x.y.z.230
quantum_url_timeout = 300
novncproxy_base_url = http://10.3.8.230:5999/vnc_auto.html
vncserver_enabled = true
vncserver_listen = x.y.z.231
vncserver_proxyclient_address = x.y.z.231
security_group_api = quantum
heal_instance_info_cache_interval = 0
libvirt_cpu_mode = none
rabbit_host = x.y.z.230
[keystone_authtoken]
admin_tenant_name = service
admin_user = nova
admin_password = 154c7f845c59fe7b9a78
auth_host = x.y.z.230
auth_port = 35357
auth_protocol = http
signing_dir = /tmp/keystone-signing-nova







