Finally got around to putting up a description of a project I’ve been poking along for a while to put Internet access and a remotely accessible security camera in the parents holiday home (ld building, thick walls and no wireline connectivity) – all the hardware used and details on the build are available here.
An EMC systems engineer has written a blog post about the VMAX FAST VP array. While it’s full of good advice, he unintentionally sums up everything that’s wrong with many applications in IT today: the knobs.
This is a problem I’ve seem more then once – the box has many nerd-knobs and therefore people feel they have to fiddle with them. Once they start to do that, inevitably something goes wrong and the performance drops. Then they start to fiddle some more to try and ‘fix’ it, until there are so many knobs turned it is impossible to maintain/troubleshoot.
Configuration nodes also provide a discovery service that the clients can use to locate the service providers (i.e. other nodes providing a particular service). It is used as a directory service. e.g. the control nodes are designed to scale horizontally: the vrouter agent can use the discovery service to determine which control-nodes to connect to (as that list may change as the cluster grows).
For example, when the vRouter agent in a compute node wants to connect to a control node (to be more precise: to an active-active pair of Control VM’s) it uses service discovery to discover the IP address of the control node. The clients use local configuration, DHCP or DNS to locate the service discovery server.
Service providers e.g. DNS, IF-MAP, etc. register with the discovery service as they come online, and anything requiring the use of the services subscribes. Usefully for debugging node issues you can take a look at the discovery service, by connecting to the server at port 5998 e.g. http://x.y.z.230:5998/services and see what services are registered:
And who are using them:
So you can see, for example, that the the vRouters are subscribed to the XMPP service, and the dns-server.
As more services/nodes are brought online they are registered with the discovery service and will be used. Likewise, if a service stops responding to heartbeats, the discovery will remove that service from the active list, and the clients will use the alternatives.
As there is a tendency to happen, my contrail lab got completely hosed by someone (shared accounts are going to be disabled going forward!) and I decided to take advantage and rebuild it using separate admin and control/traffic interfaces – closer to production type environment. This also had the benefit that I could connect to the lab MX router and therefore have ‘public’ routable addresses.
Imagine my surprise when a completely clean install failed 🙁 This was especially surprising since the procedure hand worked before. Having spend a day pounding my head against the wall and blaming everything from VMware, to typos in the testbed.py file, to Gremlins, I found the problem.
The desired setup is as follows:
Controller node – running config management, analytics, control, etc.
Admin IP: a.b.c.230
Control/Dataplane: x.y.z.230
2 x Compute Hosts – running nova-compute and vRouter
Admin IP: a.b.c.231 – 232
Control/Dataplane: x.y.z.231 – 232
During the fab setup_all the setup process was stalling at the point when it was adding the vRouter (compute host). Even left overnight, this failed to progress.
[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep mtu | awk '{ print $NF }'
[[email protected]] out: [localhost] local: ifconfig p2p2p0 | grep MTU | sed 's/.*MTU.\([0-9]\+\).*/\1/g'
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-vhost0 /etc/sysconfig/network-scripts/ifcfg-vhost0
[[email protected]] out: [localhost] local: sudo cp /etc/sysconfig/network-scripts/ifcfg-p2p2p0 /etc/sysconfig/network-scripts/ifcfg-p2p2p0.rpmsave
[[email protected]] out: [localhost] local: sudo mv /tmp/tmpG4fUw3/ifcfg-p2p2p0 /etc/contrail/
[[email protected]] out: [localhost] local: sudo chkconfig network on
[[email protected]] out: [localhost] local: sudo chkconfig supervisor-vrouter on
[[email protected]] out: [[email protected]] run: source /opt/contrail/api-venv/bin/activate && python /opt/contrail/utils/provision_vrouter.py --host_name compute-1.lab.flat-planet.net --host_ip x.y.z.231 --api_server_ip x.y.z.230 --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:
[[email protected]] out: [[email protected]] out:
Looking through the contents of /var/log/contrail/ I saw that the Query Engine was reporting issues:
[root@contrail-controller contrail]# cat /var/log/contrail/qe.log
Pre exec cmd:/usr/bin/contrail_qe_pre
log4cplus:ERROR No appenders could be found for logger (root).
log4cplus:ERROR Please initialize the log4cplus system properly.
2014-03-13 Thu 16:55:22:109.910 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: http-server-port 8091
2014-03-13 Thu 16:55:22:110.018 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Endpoint 0.0.0.0:0
2014-03-13 Thu 16:55:22:110.034 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Collectors 1
2014-03-13 Thu 16:55:22:110.051 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: ROLE : Generator
2014-03-13 Thu 16:55:22:110.063 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: SANDESH: MODULE : QueryEngine
...
2014-03-13 Thu 16:55:22:113.246 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:22 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused
2014-03-13 Thu 16:55:22:113.353 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Sandesh: Send: FAILED: 2014-Mar-13 16:55:22.113282 QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/query_engine/query.cc 1048
2014-03-13 Thu 16:55:27:113.648 GMT contrail-controller.lab.flat-planet.net [Thread 47816409982496, Pid 9400]: Db_Init: TTransportException what: connect() failed: Connection refused
Thrift: Thu Mar 13 16:55:27 2014 TSocket::open() connect() <Host: x.y.z.230 Port: 9160>Connection refused
Port 9160 is where our contrail-database (cassandra) lives – so the problem is that the provisioning script cannot connect to the system database in order to complete the provisioning.
[root@contrail-controller utils]# service contrail-database status
contrail-database FATAL Exited too quickly (process log may have details)
[root@contrail-controller utils]# service contrail-database start
contrail-database: ERROR (abnormal termination)
[root@contrail-controller utils]#
The problem here is that the process dies before logging any errors so at this point I have no clue as to why this is happening. Fortunately there was an already running system elsewhere (can also be found if you retained your console logs for the install) so I could get the exact command used to fire up the DB and see where it is going wrong.
[root@contrail-controller cassandra]# java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -Xloggc:/var/log/cassandra/gc-1393935278.log -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes -cp /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-1.2.11.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-1.2.11.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar org.apache.cassandra.service.CassandraDaemon
Error: Exception thrown by the agent : java.net.MalformedURLException: Local host name unknown: java.net.UnknownHostException: contrail-controller.lab.flat-planet.net: contrail-controller.lab.flat-planet.net
Turns out that when cassandra is being fired up, it uses the FQDN rather then an IP address when binding to the network socket, and of course I didn’t have the FQDN in /etc/hosts or in the nameserver. Quick edit later:
[root@contrail-controller cassandra]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
x.y.z.230 contrail-controller.lab.flat-planet.net
And we are in business
[root@contrail-controller cassandra]# service contrail-database start
contrail-database: started
[root@contrail-controller cassandra]# contrail-status
vRouter is NOT PRESENT
vRouter agent is NOT PRESENT
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
supervisor-dns: active
contrail-dns active
contrail-named active
== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-nodemgr active
contrail-collector active
contrail-opserver active
contrail-qe active
redis-query active
redis-sentinel active
redis-uve active
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
contrail-zookeeper active
ifmap active
redis-config active
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active
redis-webui active
== Contrail Database ==
supervisord-contrail-database:active
contrail-database active
Run the fab setup_all command again and everything provisions as it should and we have a working contrail cluster. My testbed.py file is available here as a reference.
As I finally got my Arduino reading the temperature sensor, and sending the values to this page consistently I noticed that the values were quite ‘jittery’ – the issue being that I was basically grabbing a single read from the sensor every 60 seconds, and then firing it on. Since this was a single read any issues affecting the sensor at the instant of the read could throw off the value significantly.
As you can see in the above chart we can have massive changes in the read temperature without any major changes (that I could tell) in the the actual physical environment – anything between 5 and 20 degrees between samples.
I decided that a better approach would be to average the sensor values over a time and use that as the value sent to the server. The code below shows how I’ve done that:
float readTempAvg(){ static float buff[12]; static int index = 0; static float sum = 0; static int count = 0; sum -= buff[index]; buff[index] = (analogRead(A0) / 9.31); sum += buff[index]; index = ( (index + 1) % 12 ); if( count < 12 ){ count++; } return( sum / count); }
The sum-=buff[index] has the value that is about to be overwritten in the buffer removed from the sum. This means that sum is always kept up to date. Small improvement to this code would be to define a buffer size at the start and replace all the ‘magic’ const 12 that is in the code.
Following this change I’m getting much smoother temperature readings – generally +/-1 degree or less every 60 seconds.
Next step is to try and sort out the Ethernet Shield crashes.
As part of an ongoing side project involving my newly acquired Arduino, I’ve been thinking about datalogging, and specifically network based data logging – the goal being to have all the sensor data available for analysis, graphing, etc. I’ve decided to do this on my own server to get familiar with RRD, and PHP again rather then using a Web-based service like Xively.
Also given that Google values a thermostat company, Nest, at $3.2bn the temperature and other sensor data is clearly valuable in someone’s mind 🙂
As I sketched up a very simple GET submission of the data to a PHP page, it occurred to me that I didn’t want just anybody to be able to submit sensor data. So the question is ‘How can I allow a very simple datalogger to send in data, but block all others?’. For me one possible approach is HTTP Header Enrichment.
HTTP Header Enrichment is most commonly used by mobile operators to add interesting custom fields to HTTP Header requests, usually at the GGSN or just after in the Services Gateway. The typical application is to add the MSISDN or IMSI to HTTP requests going to internal self-service portals so that sign-in is not required from a mobile device.
Typical format used for this is X-something. This is not a formal requirement, as RFC2616 basically allows most things to be used in the headers, but was borrowed from RFC2047 Section 5 (since deprecated). This is a convention, not a requirement but it is one that I will use here.
Basically the approach is to have the server side code check for the custom header, and its value and if it matches what the client sends, then log the data. If not, then return HTTP 404 – Page Not Found.
<?php
$auth_key="my_shared_secret";
if($auth_key == $_SERVER['HTTP_X_AUTH_KEY']){
// Code to handle sensor data
}
else {
error_log("Auth Key Incorrect. Received value was $_SERVER['HTTP_X_AUTH_KEY']\n");
header('HTTP/1.0 404 Not Found');
exit;
}
>
$_SERVER is an array containing information such as headers, paths, and script locations. The entries in this array are created by the web server. There is no guarantee that every web server will provide any of these; servers may omit some, or provide others not listed here. HTTP_ is also added automatically as a prefix, and the hyphens (-) are replaced with underscores (_).
So on the client (arduino side) the code segment looks like:
client.print("GET /path/file.php?t="); client.print(t1); client.print("&v="); client.print(v1); client.println(" HTTP/1.1"); client.println("User-agent: Arduino Uno"); client.println("Host: flat-planet.net"); client.println("X-AUTH-KEY: my_shared_secret"); client.println("Accept: text/html"); client.println();
so we are sending X-AUTH-KEY and that is available to the server as HTTP_X_AUTH_KEY. As long as they match, all will work fine, and remember that X-AUTH-KEY can be anything, just ensure that it is meaningful for you.
I had occasion to shrink my Contrail compute side today, and it occurred to me that while I could just pull the plug, this would result in a persistent alarm in the dashboard, mourning the loss of the old node. Since this is not an uncommon occurrence, I found a procedure to do this:
fab detach_vrouter[root@hostname ~]# openstack-status
openstack-nova-compute: active
[root@hostname ~]# service openstack-nova-compute stop
Stopping openstack-nova-compute: [ OK ]nova-manage service disable --host=<host ID> --service=nova-computeprovision_vrouter.py command to delete this nodesource /opt/contrail/api-venv/bin/activate && python/opt/contrail/utils/provision_vrouter.py --host_name <host ID> --host_ip a.b.c.d --api_server_ip a.b.c.x --oper del --admin_user <ADMIN USER> --admin_password <ADMIN PASSWORD> --admin_tenant_name admin