Tag Archives: FQDN

/etc/hosts and Signalware – DFdaemon issues

3 Replies

While doing an install of Signalware’s SIGTRAN stack (SignalWare 9.0 SP6) I had big problems getting it to actually work. The menu install process ran as expected, but the node wasn’t commissioned and therefore the go.omni process wouldn’t start, as the DFconvert step was failing.

It turns out the root cause was a problem with DFdaemon (this is a service that has something to do with maintaining the file system used by Ulticom). The service was running, but when I tried DFdaemon_status it was timing out.

Eventually killed off the service (init kept restarting it) by commenting out the contents of the init file and kill -15 1 a few times, and ran it manually in debug mode:

root@server init]# DFdaemon -d
16:49:11  0 DFdaemon::main           : using port 10600
16:49:11  0 Platform                 : ************* CE_c *************
Name:   server
mCeIndex:       0
mLanCount:      0
Current LAN index:      0
Current status: DOWN
Synchronization status: SYNC_REQD
Pending status request list:
EMPTY

oluCEList:      8
OluCe 0:
OluCe 0:
OluCe 0:
OluCe 0:
OluCe 0:
OluCe 0:
OluCe 0:
OluCe 0:
-------------

16:49:11  0 MessageQueue::MessageQue :
16:49:11  0 MessageQueue::MessageQue :
16:49:11  0 UDPmsgQueue_c::UDPmsgQue : Recv buffer : 524288
16:49:11  0 UDPmsgQueue_c::UDPmsgQue : Send buffer : 524288
16:49:11  0 UDPmsgQueue_c::UDPmsgQue : Send buffer : 524288
16:49:11  0 DFfileList_c::initEntrie : reach the end of /opt/apps/ulcm/sls0479/dffile

16:49:11  0 UDPmsgQueue_c::startComm :
16:49:11  0 UDPmsgQueue_c::MessageQu : begin thread
16:49:11  0 global                   : DFdaemon started
16:49:11  0 DFtaskQueue_c::main      : begin thread
16:49:11  0 UDPmsgQueue_c::MessageQu : begin thread
16:49:11  0 SDFS::main               : begin thread
16:49:11  0 DFfileList_c::main       : begin thread
16:49:11  0 HealthCheck::main        : begin thread
16:49:11  0 HealthCheck::main        : wakeup before sleep
16:49:11  0 SDFS::main               : wakeup before sleep(1000)
16:49:11  0 HealthCheck::main        : wakeup after sleep
16:49:11  0 HealthCheck::main        : wakeup before sleep
16:49:12  0 SDFS::main               : wakeup after sleep
16:49:12  0 HealthCheck::main        : wakeup after sleep
16:49:12  0 HealthCheck::main        : wakeup before sleep
16:49:12  0 SDFS::main               : wakeup before sleep(1000)
16:49:12  0 HealthCheck::main        : wakeup after sleep
16:49:12  0 HealthCheck::main        : wakeup before sleep
16:49:13  0 HealthCheck::main        : wakeup after sleep
16:49:13  0 HealthCheck::main        : wakeup before sleep
16:49:13  0 SDFS::main               : wakeup after sleep
16:49:13  0 SDFS::main               : wakeup before sleep(1000)
16:49:13  0 HealthCheck::main        : wakeup after sleep
...
16:49:40  0 HealthCheck::main        : wakeup after sleep
16:49:40  0 HealthCheck::main        : wakeup before sleep
16:49:40  0 SDFS::main               : wakeup after sleep
16:49:40  0 SDFS::main               : wakeup before sleep(1000)
16:49:40  0 UDPmsgQueue_c::MessageQu : Received message from unknown sender, IP address: ::ffff:127.0.0.1    <<---- DFdaemon_status
16:49:40  0 HealthCheck::main        : wakeup after sleep
16:49:40  0 HealthCheck::main        : wakeup before sleep
16:49:41  0 HealthCheck::main        : wakeup after sleep
16:49:41  0 HealthCheck::main        : wakeup before sleep
16:49:41  0 SDFS::main               : wakeup after sleep
16:49:41  0 SDFS::main               : wakeup before sleep(1000)
16:49:41  0 HealthCheck::main        : wakeup after sleep
16:49:41  0 HealthCheck::main        : wakeup before sleep

It seems that for some reason the localhost entry for IPv6 was causing a problem. The entry that was part of the server build looked like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost

It should be like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

With that DFconvert ran and the rest of the process was fine.

Contrail Setup – vRouter Provisioning Failure on new install

4 Replies

The importance of basic system configuration

As there is a tendency to happen, my contrail lab got completely hosed by someone (shared accounts are going to be disabled going forward!) and I decided to take advantage and rebuild it using separate admin and control/traffic interfaces – closer to production type environment. This also had the benefit that I could connect to the lab MX router and therefore have ‘public’ routable addresses.

Imagine my surprise when a completely clean install failed 🙁 This was especially surprising since the procedure hand worked before. Having spend a day pounding my head against the wall and blaming everything from VMware, to typos in the testbed.py file, to Gremlins, I found the problem.

Target Setup

The desired setup is as follows:
Controller node – running config management, analytics, control, etc.
Admin IP: a.b.c.230
Control/Dataplane: x.y.z.230

2 x Compute Hosts – running nova-compute and vRouter
Admin IP: a.b.c.231 – 232
Control/Dataplane: x.y.z.231 – 232

Problem – vRouter Provisioning Stalls/Fails

During the fab setup_all the setup process was stalling at the point when it was adding the vRouter (compute host). Even left overnight, this failed to progress.

[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep mtu | awk '{ print $NF }'
[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep MTU | sed 's/.*MTU.\([0-9]\+\).*/\1/g'
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-vhost0 /etc/sysconfig/network-scripts/ifcfg-vhost0
[[email protected]] out: [localhost] local: sudo cp /etc/sysconfig/network-scripts/ifcfg-p2p2p0 /etc/sysconfig/network-scripts/ifcfg-p2p2p0.rpmsave
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-p2p2p0 /etc/contrail/
[[email protected]] out: [localhost] local: sudo chkconfig network on
[[email protected]] out: [localhost] local: sudo chkconfig supervisor-vrouter on
[[email protected]] out: [[email protected]] run: source /opt/contrail/api-venv/bin/activate && python /opt/contrail/utils/provision_vrouter.py --host_name compute-1.lab.flat-planet.net --host_ip x.y.z.231 --api_server_ip x.y.z.230 --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:

Looking through the contents of /var/log/contrail/ I saw that the Query Engine was reporting issues:

[root@contrail-controller contrail]# cat /var/log/contrail/qe.log
Pre exec cmd:/usr/bin/contrail_qe_pre
log4cplus:ERROR No appenders could be found for logger (root).
log4cplus:ERROR Please initialize the log4cplus system properly.
2014-03-13 Thu 16:55:22:109.910 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: http-server-port 8091
2014-03-13 Thu 16:55:22:110.018 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Endpoint 0.0.0.0:0
2014-03-13 Thu 16:55:22:110.034 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Collectors 1
2014-03-13 Thu 16:55:22:110.051 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: ROLE   : Generator
2014-03-13 Thu 16:55:22:110.063 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: MODULE : QueryEngine
...
2014-03-13 Thu 16:55:22:113.246 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:22 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused
2014-03-13 Thu 16:55:22:113.353 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Sandesh: Send: FAILED: 2014-Mar-13 16:55:22.113282 QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/query_engine/query.cc 1048
2014-03-13 Thu 16:55:27:113.648 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:27 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused

Port 9160 is where our contrail-database (cassandra) lives – so the problem is that the provisioning script cannot connect to the system database in order to complete the provisioning.

[root@contrail-controller utils]# service contrail-database status
contrail-database                FATAL      Exited too quickly (process log may have details)
[root@contrail-controller utils]# service contrail-database start
contrail-database: ERROR (abnormal termination)
[root@contrail-controller utils]#

The problem here is that the process dies before logging any errors so at this point I have no clue as to why this is happening. Fortunately there was an already running system elsewhere (can also be found if you retained your console logs for the install) so I could get the exact command used to fire up the DB and see where it is going wrong.

[root@contrail-controller cassandra]# java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -Xloggc:/var/log/cassandra/gc-1393935278.log -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes -cp /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-1.2.11.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar org.apache.cassandra.service.CassandraDaemon
Error: Exception thrown by the agent : java.net.MalformedURLException: Local host name unknown: java.net.UnknownHostException: contrail-controller.lab.flat-planet.net: contrail-controller.lab.flat-planet.net

Turns out that when cassandra is being fired up, it uses the FQDN rather then an IP address when binding to the network socket, and of course I didn’t have the FQDN in /etc/hosts or in the nameserver. Quick edit later:

[root@contrail-controller cassandra]# vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
x.y.z.230 contrail-controller.lab.flat-planet.net

And we are in business

[root@contrail-controller cassandra]# service contrail-database start
contrail-database: started

[root@contrail-controller cassandra]# contrail-status
vRouter is NOT PRESENT
vRouter agent is NOT PRESENT
== Contrail Control ==
supervisor-control:           active
contrail-control              active
contrail-control-nodemgr      active
supervisor-dns:               active
contrail-dns                  active
contrail-named                active
== Contrail Analytics ==
supervisor-analytics:         active
contrail-analytics-nodemgr    active
contrail-collector            active
contrail-opserver             active
contrail-qe                   active
redis-query                   active
redis-sentinel                active
redis-uve                     active
== Contrail Config ==
supervisor-config:            active
contrail-api:0                active
contrail-config-nodemgr       active
contrail-discovery:0          active
contrail-schema               active
contrail-svc-monitor          active
contrail-zookeeper            active
ifmap                         active
redis-config                  active
== Contrail Web UI ==
supervisor-webui:             active
contrail-webui                active
contrail-webui-middleware     active
redis-webui                   active
== Contrail Database ==
supervisord-contrail-database:active
contrail-database             active

Run the fab setup_all command again and everything provisions as it should and we have a working contrail cluster. My testbed.py file is available here as a reference.