WordPress XML-RPC Attack

This week, one of my sites, sptr.net, has been under a co-ordinated and sustained attack from what appears to be a botnet – a collective of several hundred virus-infected computers running Microsoft Windows. The attack comprises attempts to use the remote procedure call methods built into WordPress to post unauthorised content.

Detection

I was notified by one of my independent monitoring services that the site was having trouble some time after the attack began. It appears that once triggered by the attacker, it takes a while for the command to spread to a significant number of infected machines – this is reasonable if you assume the greatest number of infected PCs is in the USA. The attack peaked around the middle of the day in Scotland, coincident with the switching on of computers as the sun moved East to West across the continental US. Although the server remained operational, it was struggling to continue to respond to requests in a reasonable time as the CPU usage soared way above 1000% of nominal maximum. A look at the top processes on the server showed that it was trying to keep things together:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
12345 xxxxxxx    20   0 55992 36080  7128 R 49.0  6.9  0:15.73 [see below]
12346 xxxxxxx    20   0 56036 36124  7188 R 46.0  6.9  0:04.96 [see below]
12347 xxxxxxx    20   0 55940 36076  7128 R 46.0  6.9  0:03.82 [see below]
12349 xxxxxxx    20   0 55912 35908  6984 R 46.0  6.8  0:07.88 [see below]
12340 xxxxxxx    20   0 55976 36116  7180 R 46.0  6.9  0:03.59 [see below]
12342 xxxxxxx    20   0 55940 36064  7128 R 44.0  6.9  0:07.21 [see below]
12341 xxxxxxx    20   0 55948 36140  7196 R 44.0  6.9  0:34.79 [see below]
12343 xxxxxxx    20   0 55972 36248  7276 R 44.0  6.9  2:20.11 [see below]

The command attempted showed that it was an attack on a php script:

/usr/bin/php-cgi -c /var/www/vhosts/sptr.net/etc/php.ini

Further investigation

Looking at the server access logs identified the specific script targeted by the attacker, the machines and methodology involved. The range of IP addresses showed that the infected PCs were world-wide (in the sample below, India, Poland, Egypt, Thailand, Algeria, Brazil and Pakistan).

106.76.44.110 - - [10/Jul/2014:14:03:19 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
194.50.157.187 - - [10/Jul/2014:14:03:34 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.235.83.103 - - [10/Jul/2014:14:03:43 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
171.6.204.105 - - [10/Jul/2014:14:03:54 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.107.87.186 - - [10/Jul/2014:14:04:04 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
179.186.51.47 - - [10/Jul/2014:14:04:06 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
39.44.61.247 - - [10/Jul/2014:14:04:14 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

Mitigation

Restarting the VPS container made no difference. CPU usage remained very high. Installing a plugin to disable XML-RPC in WordPress seemed to make things better, probably because of the response time improvement but as the day progressed, the attack seemed to abate and the server was coping better with CPU usage falling below 100% nominal maximum. The log sample above is from today, when the attacks have fallen to a few per minute instead of the hundreds per second on Tuesday. It looks like the botnet is learning that there are robust passwords on the system that will take too long to guess and is giving up.

Brute force solution

I’m not happy with this constant knocking at my door, however, so have decided that I don’t need a door there at all. Removing the target script doesn’t directly affect the rate of attack, it changes the 200 response to a 404 (page not found), which is quickly delivered.

94.55.132.13 - - [10/Jul/2014:14:09:13 +0000] "POST /xmlrpc.php HTTP/1.1" 404 430 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

WordPress Redirect Loop

WordPress is a brilliant tool, probably the best of the CMSs – Google says so – but every now and then it can stop you in your tracks. It did this today as I was setting up a new site for Marc Walker, the British Biathlon veteran and team manager who is retiring from Her Majesty’s service in August to set up a very special personal trainer business in Knutsford.

Marc Walker (image copyright Marcel Laponder CC-BY-3.0)
Marc Walker (image copyright Marcel Laponder CC-BY-3.0)

I hit a wee problem with an unexpected redirect loop when trying to access the back end. There are plenty of articles and “fixes” available on the web, none of which were relevant to my installation and most of which relate to permalinks and .htaccess. Because my installation is a long-standing derivative of WPMU or multi-site, it could not have been that.

For others in the same position, here’s what my install looks like:

  • LAMP hosted (on a VPS)
  • Version 3.9.1
  • Multiple WP sites, domain-mapped

I had a while ago, for some reason I have now forgotten, network disabled the default WordPress themes. When I added this new site, created the new admin user and mapped the domain, I found that the admin or login pages simply got stuck in a redirect loop.

The fix was easy enough – I simply had to enable Twenty-Fourteen (the default WP theme) for the new site via the network admin panel.

If you want to visit Marc’s new site, it’s at AvantgardePT.com. His new business will start up in August and will have a strong European baseline from his track record in Biathlon, military fitness, Iron Man, and an impressive bunch of competitive sports.

Efficient, searchable logging

I’ve been trying to find out what is the best way – for me – to keep a record of readings, meetings, seminars and the other stuff of studying to become a researcher. I begin a part time PhD in September and as I will have severe demands on my time for the day job, need to be sure that I work smart.

I thought about all kinds of tools for this. The first thing to realise is that I will probably be making use of pen and paper as the ultimate portable and immediate way to organise my thinking. I’ve done this since 1976 and have a wall full of diaries and notes back to that time. Despite being a technophile, I have tried and failed to like any of the web or tablet based services like Evernote. I want to capture images, probably like this photo of hand-written notes. I want what I record to be searchable.

So, what am I going to try? A combination of email and blogging. I already have the blog you’re reading and this post is made using the JetPack “post by email” feature. I wonder if it will work?

WordPress network domain mapping fix

I’ve just been on an interesting little journey that started last September when I discovered that some of the sites on one of my WordPress networks had stopped working.

You might enjoy a little schadenfreude if I admit now that it was because I had a brilliant idea and did something stupid. I’ve posted details here in the hope that (a) if I do it again, I can find out how to fix it, and (b) if you’ve done it too, you’re closer to the solution than I have been for the past six months. I’ve ‘genericed’ the details to help you map it to your own setup. Continue reading “WordPress network domain mapping fix”

Code Hacks: WordPress WP-Filebase Pro

I updated my WordPress installations to 3.7 “Basie” this weekend and found that the file download stopped working for all users except admins. The files affected are those for which permissions are required (Subscriber and above).

I made a workaround to get it working for my physics teachers resource site at http://sptr.net. In classes/Item.php, within function CurUserCanAccess($for_tpl=false, $user = null), above the loop that checks user roles, you have to populate the roles array by calling get_role_caps(), thus:

...
if(empty($frs)) return true; // item is for everyone!
$user->get_role_caps();
foreach($user->roles as $ur) { // check user roles against item roles
 if(in_array($ur, $frs))
 return true;
 }
...

It looks like it’s working for now. I’ve posted the fix to the Plugin support forum.

Code hacks: Internet Explorer Compatibility mode

guess-browserI’ve been developing a site for a national competition to be launched in October and ran into a little difficulty with Internet Explorer. Despite this browser losing ground amongst even the unthinking default user community, it is still sufficiently popular that I needed to look at the problem.

The site is running WordPress on a LAMP server using Konstantin Kovshenin‘s Expound theme. One of the testers noticed the letters “Ski” next to the Home menu item. Mousing over it produces a fleeting grey box to appear top left of the browser window. This turned out to be only visible in IE8, IE9 and IE 10 in “compatibility mode”, a feature of IE that allows the browser to render sites that are broken by IE’s shockingly poor implementation of standards, using a model from an earlier version. It’s what a code monkey might call a Kludge.

SkiThe “Ski” is in fact, the first few letters of “Skip to content”. It is one of several features of the site’s theme implementation which are broken in IE’s compatibility mode.

There are several suggestions in the forums designed to force IE into non-compatibility mode and render the site properly. Most rely on delivering a <Doctype> tag on the very first line, followed immediately by a X-UA-Compatible meta tag. Unless this tag is placed on the line immediately after the Doctype tag, IE ignores it.

I considered trying to knock up a plugin to make this work in some kind of customisable way. Editing the theme’s header.php file seemed doomed to be overwritten on the next update, and branching a child theme felt like too much hard work for such a small fix to accommodate a browser that I personally would like everyone to stop using. Part of me wants all sites to look broken when viewed with IE so as to encourage the masses to make an intelligent choice for once. Let’s not start talking about democracy.

Anyway, a little more digging found a really elegant solution which suited my particular needs from Reza Qorbani, which is to use the .htaccess file to have the Apache server sniff the browser and send the metatag. This is what I finished up with:

BrowserMatch MSIE best-standards-support
Header set X-UA-Compatible IE=edge env=best-standards-support

It works a treat. Thanks, Reza!

A summer of code

anarchyThe summer has had me getting to grips with the nitty-gritty of internet web hosting, caused by a consolidation and move of all of the websites and services that I host to a new server. I had been using HostPapa in a shared environment for several years but the traffic and resource usage of these sites had been on the increase for about 18 months, to the point that HostPapa invited me to pack up and leave.

After a detailed survey of requirements and possible alternatives, I elected to move to the affordable but much more powerful next-step-up of a virtual private server (VPS) solution from HostingUK. I’ve known these guys since they set up business in the late 90’s and felt comfortable that I would get good support from the people behind the business. I haven’t been disappointed.

The new server runs CentOS 6.4, a version of the Red Hat Linux operating system and has the usual LAMP features of Apache Web server, mySQL and PHP, with the Parallels Plex 11 management panel.

My development has been firstly in the area of learning how to set it all up using the Plex panel: it’s a very powerful tool but it’s not quite plug-and-play. The DNS for each of the domains on the site is best managed at the registration server using their nameservers: they have redundancy built in and although the VPS can be its own NS, if it goes down for any reason, this can lead to problems with mail transport and SEO indexing. Within the DNS records for each domain, minimum configuration requires appropriate A, MX and CNAME  entries as well as TXT or SPF records to stop your mail from being forever consigned to the spam folder.

Further learning has included getting down and dirty with the *nix command line, from basic file operations to examining logs, setting up CRON and managing and installing further packages. I’ve installed Munin to help identify what normal operation looks like. One of the things that my new insight has given me is an appreciation of just how much sustained attack is endured by even the smallest of websites by the likes of Turkish, Chinese, North Korean and other interests. The importance of having decent passwords is underlined when you see 20,000 (yes, twenty thousand) attempts to guess the root password in a single day.

The summer of code has reminded me of what I’m best at, and what I enjoy doing.