WordPress XML-RPC Attack

This week, one of my sites, sptr.net, has been under a co-ordinated and sustained attack from what appears to be a botnet – a collective of several hundred virus-infected computers running Microsoft Windows. The attack comprises attempts to use the remote procedure call methods built into WordPress to post unauthorised content.

Detection

I was notified by one of my independent monitoring services that the site was having trouble some time after the attack began. It appears that once triggered by the attacker, it takes a while for the command to spread to a significant number of infected machines – this is reasonable if you assume the greatest number of infected PCs is in the USA. The attack peaked around the middle of the day in Scotland, coincident with the switching on of computers as the sun moved East to West across the continental US. Although the server remained operational, it was struggling to continue to respond to requests in a reasonable time as the CPU usage soared way above 1000% of nominal maximum. A look at the top processes on the server showed that it was trying to keep things together:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
12345 xxxxxxx    20   0 55992 36080  7128 R 49.0  6.9  0:15.73 [see below]
12346 xxxxxxx    20   0 56036 36124  7188 R 46.0  6.9  0:04.96 [see below]
12347 xxxxxxx    20   0 55940 36076  7128 R 46.0  6.9  0:03.82 [see below]
12349 xxxxxxx    20   0 55912 35908  6984 R 46.0  6.8  0:07.88 [see below]
12340 xxxxxxx    20   0 55976 36116  7180 R 46.0  6.9  0:03.59 [see below]
12342 xxxxxxx    20   0 55940 36064  7128 R 44.0  6.9  0:07.21 [see below]
12341 xxxxxxx    20   0 55948 36140  7196 R 44.0  6.9  0:34.79 [see below]
12343 xxxxxxx    20   0 55972 36248  7276 R 44.0  6.9  2:20.11 [see below]

The command attempted showed that it was an attack on a php script:

/usr/bin/php-cgi -c /var/www/vhosts/sptr.net/etc/php.ini

Further investigation

Looking at the server access logs identified the specific script targeted by the attacker, the machines and methodology involved. The range of IP addresses showed that the infected PCs were world-wide (in the sample below, India, Poland, Egypt, Thailand, Algeria, Brazil and Pakistan).

106.76.44.110 - - [10/Jul/2014:14:03:19 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
194.50.157.187 - - [10/Jul/2014:14:03:34 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.235.83.103 - - [10/Jul/2014:14:03:43 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
171.6.204.105 - - [10/Jul/2014:14:03:54 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.107.87.186 - - [10/Jul/2014:14:04:04 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
179.186.51.47 - - [10/Jul/2014:14:04:06 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
39.44.61.247 - - [10/Jul/2014:14:04:14 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

Mitigation

Restarting the VPS container made no difference. CPU usage remained very high. Installing a plugin to disable XML-RPC in WordPress seemed to make things better, probably because of the response time improvement but as the day progressed, the attack seemed to abate and the server was coping better with CPU usage falling below 100% nominal maximum. The log sample above is from today, when the attacks have fallen to a few per minute instead of the hundreds per second on Tuesday. It looks like the botnet is learning that there are robust passwords on the system that will take too long to guess and is giving up.

Brute force solution

I’m not happy with this constant knocking at my door, however, so have decided that I don’t need a door there at all. Removing the target script doesn’t directly affect the rate of attack, it changes the 200 response to a 404 (page not found), which is quickly delivered.

94.55.132.13 - - [10/Jul/2014:14:09:13 +0000] "POST /xmlrpc.php HTTP/1.1" 404 430 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

Redmine on CentOS

If you listened to that last audioboo, you’ll maybe recognise that I like the idea that being in control of your destiny is connected to how much you know about your life. The podcast was talking about organisations but my life at the moment is not unlike an organisation, with projects, finance and time management all being features. I have been using a number of tools to track all of this activity and frankly they’re not good enough, so I thought I’d give Redmine a try, after a couple of strong recommendations. Here’s I how I set it up on my CentOS VPS (Virtual Private Server). Continue reading “Redmine on CentOS”

Pro Git and more MX DNS

Continuing the summer of code into the early autumn, I have been developing, enhancing and debugging the new server. New and migrated sites are stable and responding well within the resource limits I’ve chosen of 10GB disk, 50GB traffic (although we’re close to whacking this one) and 256/512 MB RAM/Swap space. Uptime has been 100% for over 60 days now.

Within the suite of services running on the server are database, web server, CGI, mail, stats and monitoring. What is not, is the DNS service, which I have learned to keep in a different place, with the registrar. Setting up reverse DNS for the mail service to work correctly is important: I discovered that one client had been having difficulties receiving mail from just one of his friends. This was because the MX DNS entry for his domain pointed to an IP address which some service providers will reject as it doesn’t comply with the RFC. Changing it to the host domain of the server’s IP, however, stopped all mail getting through to the client. This was finally resolved by pointing the MX record for the domain to the domain itself:

example.com. A     192.0.2.1
@            MX 10 example.com.

If you want to know how the Internet works, by the way, a really good place to start is the Internet Engineering Task Force (IETF). They have a good introduction here. Many internet standards are defined in RFC documents.

Other services on the server operate as database-driven php suites such as the WordPress CMS, Moodle, LimeSurvey or phpBB. All of these are subject to modifications, code hacks and tweaks to make them work to the needs of the site owner. Whilst the Parallels Plesk Panel allows install-at-a-click for many application suites, I prefer to manage the installation and customisation of these myself. Until now, I had used the download-unzip-upload over FTP method but I’m going to try using the more elegant command-line facility offered by Git. I’m getting started by using their excellent online documentation. This should allow me a much faster update route and potentially a way to be a better contributor to open source than the consumer I have been.

A summer of code

anarchyThe summer has had me getting to grips with the nitty-gritty of internet web hosting, caused by a consolidation and move of all of the websites and services that I host to a new server. I had been using HostPapa in a shared environment for several years but the traffic and resource usage of these sites had been on the increase for about 18 months, to the point that HostPapa invited me to pack up and leave.

After a detailed survey of requirements and possible alternatives, I elected to move to the affordable but much more powerful next-step-up of a virtual private server (VPS) solution from HostingUK. I’ve known these guys since they set up business in the late 90’s and felt comfortable that I would get good support from the people behind the business. I haven’t been disappointed.

The new server runs CentOS 6.4, a version of the Red Hat Linux operating system and has the usual LAMP features of Apache Web server, mySQL and PHP, with the Parallels Plex 11 management panel.

My development has been firstly in the area of learning how to set it all up using the Plex panel: it’s a very powerful tool but it’s not quite plug-and-play. The DNS for each of the domains on the site is best managed at the registration server using their nameservers: they have redundancy built in and although the VPS can be its own NS, if it goes down for any reason, this can lead to problems with mail transport and SEO indexing. Within the DNS records for each domain, minimum configuration requires appropriate A, MX and CNAME  entries as well as TXT or SPF records to stop your mail from being forever consigned to the spam folder.

Further learning has included getting down and dirty with the *nix command line, from basic file operations to examining logs, setting up CRON and managing and installing further packages. I’ve installed Munin to help identify what normal operation looks like. One of the things that my new insight has given me is an appreciation of just how much sustained attack is endured by even the smallest of websites by the likes of Turkish, Chinese, North Korean and other interests. The importance of having decent passwords is underlined when you see 20,000 (yes, twenty thousand) attempts to guess the root password in a single day.

The summer of code has reminded me of what I’m best at, and what I enjoy doing.