The importance of basic system configuration
As there is a tendency to happen, my contrail lab got completely hosed by someone (shared accounts are going to be disabled going forward!) and I decided to take advantage and rebuild it using separate admin and control/traffic interfaces – closer to production type environment. This also had the benefit that I could connect to the lab MX router and therefore have ‘public’ routable addresses.
Imagine my surprise when a completely clean install failed 🙁 This was especially surprising since the procedure hand worked before. Having spend a day pounding my head against the wall and blaming everything from VMware, to typos in the testbed.py file, to Gremlins, I found the problem.
Target Setup
The desired setup is as follows:
Controller node – running config management, analytics, control, etc.
Admin IP: a.b.c.230
Control/Dataplane: x.y.z.230
2 x Compute Hosts – running nova-compute and vRouter
Admin IP: a.b.c.231 – 232
Control/Dataplane: x.y.z.231 – 232
Problem – vRouter Provisioning Stalls/Fails
During the fab setup_all the setup process was stalling at the point when it was adding the vRouter (compute host). Even left overnight, this failed to progress.
[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep mtu | awk '{ print $NF }'
[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep MTU | sed 's/.*MTU.\([0-9]\+\).*/\1/g'
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-vhost0 /etc/sysconfig/network-scripts/ifcfg-vhost0
[[email protected]] out: [localhost] local: sudo cp /etc/sysconfig/network-scripts/ifcfg-p2p2p0 /etc/sysconfig/network-scripts/ifcfg-p2p2p0.rpmsave
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-p2p2p0 /etc/contrail/
[[email protected]] out: [localhost] local: sudo chkconfig network on
[[email protected]] out: [localhost] local: sudo chkconfig supervisor-vrouter on
[[email protected]] out: [[email protected]] run: source /opt/contrail/api-venv/bin/activate && python /opt/contrail/utils/provision_vrouter.py --host_name compute-1.lab.flat-planet.net --host_ip x.y.z.231 --api_server_ip x.y.z.230 --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:
Looking through the contents of /var/log/contrail/ I saw that the Query Engine was reporting issues:
[root@contrail-controller contrail]# cat /var/log/contrail/qe.log
Pre exec cmd:/usr/bin/contrail_qe_pre
log4cplus:ERROR No appenders could be found for logger (root).
log4cplus:ERROR Please initialize the log4cplus system properly.
2014-03-13 Thu 16:55:22:109.910 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: http-server-port 8091
2014-03-13 Thu 16:55:22:110.018 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Endpoint 0.0.0.0:0
2014-03-13 Thu 16:55:22:110.034 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Collectors 1
2014-03-13 Thu 16:55:22:110.051 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: ROLE : Generator
2014-03-13 Thu 16:55:22:110.063 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: MODULE : QueryEngine
...
2014-03-13 Thu 16:55:22:113.246 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:22 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused
2014-03-13 Thu 16:55:22:113.353 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Sandesh: Send: FAILED: 2014-Mar-13 16:55:22.113282 QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/query_engine/query.cc 1048
2014-03-13 Thu 16:55:27:113.648 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:27 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused
Port 9160 is where our contrail-database (cassandra) lives – so the problem is that the provisioning script cannot connect to the system database in order to complete the provisioning.
[root@contrail-controller utils]# service contrail-database status
contrail-database FATAL Exited too quickly (process log may have details)
[root@contrail-controller utils]# service contrail-database start
contrail-database: ERROR (abnormal termination)
[root@contrail-controller utils]#
The problem here is that the process dies before logging any errors so at this point I have no clue as to why this is happening. Fortunately there was an already running system elsewhere (can also be found if you retained your console logs for the install) so I could get the exact command used to fire up the DB and see where it is going wrong.
[root@contrail-controller cassandra]# java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -Xloggc:/var/log/cassandra/gc-1393935278.log -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes -cp /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-1.2.11.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar org.apache.cassandra.service.CassandraDaemon
Error: Exception thrown by the agent : java.net.MalformedURLException: Local host name unknown: java.net.UnknownHostException: contrail-controller.lab.flat-planet.net: contrail-controller.lab.flat-planet.net
Turns out that when cassandra is being fired up, it uses the FQDN rather then an IP address when binding to the network socket, and of course I didn’t have the FQDN in /etc/hosts or in the nameserver. Quick edit later:
[root@contrail-controller cassandra]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
x.y.z.230 contrail-controller.lab.flat-planet.net
And we are in business
[root@contrail-controller cassandra]# service contrail-database start
contrail-database: started
[root@contrail-controller cassandra]# contrail-status
vRouter is NOT PRESENT
vRouter agent is NOT PRESENT
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
supervisor-dns: active
contrail-dns active
contrail-named active
== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-nodemgr active
contrail-collector active
contrail-opserver active
contrail-qe active
redis-query active
redis-sentinel active
redis-uve active
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
contrail-zookeeper active
ifmap active
redis-config active
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active
redis-webui active
== Contrail Database ==
supervisord-contrail-database:active
contrail-database active
Run the fab setup_all command again and everything provisions as it should and we have a working contrail cluster. My testbed.py file is available here as a reference.
