Last.fm scrobbler v2 doesn’t work

lastfmThis morning, I finally gave up trying to sort out the scrobbling problem I’ve been having since December. The current Last.fm scrobbler, version 2, is just not functioning, so I’ve reverted to version 1.5, losing 4 months’ scrobbles in the process. Not impressed. Why can’t anybody write software that works any more?

The problem has been that although the Last.fm app on my OSX device seems to work, reporting scrobbles normally, these seem to get stuck in cache. In the app, these tracks show as “cached” and do not appear on my last.fm profile.

Long story short, if you’re a Mac user having trouble with last.fm not scrobbling your tracks, delete the last.fm scrobbler, empty the trash and download the older version 1.5 here (dmg).

And(roid) another one bites the dust

I don’t have good experience with Android devices, not that I’ve ever actually bought one. My first device was a Google Nexus I won in a prize draw which was just a flaky piece of junk, even when they eventually sent me a new one after returning the original, with its faulty display, four times.

Luckily enough, I won an Amazon Kindle Fire in the University’s Digital Footprint launch competition, which was handy – my reading list consists mostly of papers and articles that I manage with Mendeley. I can send pdfs automatically using the brilliant Kinsync service, straight to my Kindle.

This morning, alas, my Kindle Fire woke up dead. No response, even to the 40-second reset. I eventually found myself talking to Amazon support, who, because the device is still in warranty, have next-day shipped me a new replacement on the promise (and credit card collateral) that I send them back the dud. They’re even paying the return postage. More than this I can’t ask: it’s certainly better than the “all our customers are morons” experience I had with Google.

So far, my experience with Android equipment has been worth every penny I’ve spent.

Productivity of a new researcher

I’ve spent part of the summer preparing to begin a six-year research project alongside my day job in initial teacher education at the University of Edinburgh. Time is possibly the scarcest resource I have and that preparation has involved assessment and selection of systems that will enable me to be efficient, effective and productive. Here’s what I have in current use.

To-do

logo2Keeping a handle on things I have to do, prioritising and postponing according to progress, is essential to getting things done.

rememberthemilk.com provides this functionality through a web interface which includes a calendar feed and the ability to add new tasks by sending an email to a private address. A Chrome extension shows the RTM current list within the Google calendar web view and allows task completion or postponing.

Calendar

Google Calendar – or rather, several google calendars – allow me to manage the various demands on my time and keep an eye on events of interest that I’m following. Synching the calendars to the Calendar app on my mac and mobile devices means I know where I’m supposed to be at any time, and what gaps exist for new opportunities. New events (such as seminars booked through services like eventbrite) can quickly be added to the calendars by downloading an ics file. The RTM list and timed events appear within the calendar. On the mac, dates within emails can be directly viewed in your calendar and optionally added, allowing fast and selective adding of new opportunities such as seminars.

Workspace

WikiMy study, reading and research diary needs to be quick, easy and searchable. I have set up a MediaWiki installation on my server at http://cullaloe.net/w and given my supervisors write access to allow public commentary and guidance that is similarly searchable. I like the wiki markup which is just a small step from plain text – it provides very rapid content-focused editing and light touch formatting.

I have used a couple of extensions for in-page references (Cite) and to make it easy to insert citations (Bibtex) to papers and books I am reading, by copying references from Mendeley and pasting directly into the page.

Citation Manager

logo-mendeleyThe tool of choice here is Mendeley, which is a cloud-based bibliography manager with easy import from many formats (including books on Amazon, Google Scholar and the academic libraries). It has a “Save to Mendeley” bookmark for rapid extraction from webpages and a desktop application that synchs automatically to the web database. What I really like about this software is that it allows groups of references to be created which are automatically saved in BibTex files, one per group, which makes compilation against LaTeX seamless.

Paper/thesis creation

200px-LaTeX_logo.svgWhat else? LaTeX – I use the TexShop environment on my mac – produces beautiful documents (output to pdf) in a few keystrokes without any worries about formatting, compatibility or platform, and the almost transparent inclusion and rendering of bibliographies, tables of contents, margin notes, tables, figures and images.

Clippings

Evernote-logo-e1362251497276The handiest tool I have to quickly grab things I want to refer to later is Evernote. It has the quick post facility within my browser and the ability to forward emails out of my inbox to a less in-your-face place for later review and action. There’s also a nice desktop app to complement the easy web interface. Notebooks can be organised any way to suit you and can be bundled together to manage the important separation between different workflows.

Storage

Dropbox-LogoDropbox is one of the services I use for cloud storage. All source files and working documents are kept here. I’ve been using Dropbox long enough to have earned additional storage free of charge but most of that is taken up in the backup of files for my teachers’ site at sptr.net.

In addition to DropBox, I also make use of I also make use of copy.com which works in a similar way. Significantly, I do not use Google’s GDrive because I dislike how it works, as much as I dislike Google docs. Having been stung by Google’s sudden removal of services I’ve relied on in the past, such as bookmarks, I am reluctant to rely too heavily on them.

Cost

All of these tools, services and software are free. There are paid services but I am a light enough user not to incur the need to pay the subscription for any of the services mentioned here. That’s not to say I’m not willing to pay for these services because they are worth it, but the price points are disproportionate for most of them so I don’t volunteer cash I don’t have to spend. Service providers, take note: less is more. Cut your fees and more will pay. I do have a Premium Evernote account but only because it’s on promotion with O2 at the moment. You will not find Microsoft products on any technology I own.

Workflow

I always take pens and good-quality plain paper notebooks with me wherever I go. Email is ever present on a mobile device or laptop, as is my calendar, dropbox and browser. Also mobile but less central to hour-by-hour workflow are Evernote and Mendeley. I manage RTM only via a browser, and editing the workspace wiki is easily done there also. Chrome is my browser of choice on all of my devices – all the bookmarks synch automatically. It is likely that I will try other tools from time to time but I do not have the luxury of time to trial alternatives: my focus has to be on being effective if I am to meet current aspirations and obligations.

I hope this entry has been of interest – please get in touch if you have a suggestion to make, especially one that might make my life easier.

Hacking the Canon Powershot SX20 IS

I’ve had my Canon Powershot SX20 IS camera for a few years now and have always regarded it as a stepping-stone to a better, “proper” camera. The problem is I have never quite got to the point where I can justify shelling out the considerable wonga to take the next step.

What I’d like is a modern digital equivalent to my brilliant old Nikon FM that served me well for a number of years, with up to date features as well as the best of the old. Two things in particular have annoyed me about the SX20 – the maximum exposure time of 15 seconds and the digital compression which irrationally leaves me with FOMO – something is missing from my photographs.

Having resolved not to spend a grand on a new camera, instead I lobbed a hundred quid into the Physics Pixies UNICEF appeal and set about altering the camera I have to deal with the two “problems”. The alterations amount to a firmware update using the CHDK (Canon Hack Development Kit) firmware addon. This is now an open-source project built on the work of programmer VitalyB’s RAW enabler and Andrei Gratchev’s development kit. The firmware update now includes a number of other really nice features including time-lapse, motion detection and bracketing of exposure and focus.

Finding out the camera’s firmware

The EXIF data in a digital photograph tells you quite a lot about the camera that took it and the settings used – see, for example, this picture on Flickr. Click “show EXIF”. This tells me almost but not quite enough about the firmware Revision – 1.02 rev 2.00. Your camera will tell you, though. First, create an empty file called ver.req in the root of the SD card. I did this on a MacBook Pro with the SD card in a slot on the laptop by issuing these commands:

$ cd Volumes/CANON_DC/
$ touch ver.req

Put the card in your camera and start it up in playback mode. From the main screen (should be displaying NO IMAGE for no images on the card), press FUNC SET and DISP. buttons and the camera will display a screen like this for about 5 seconds:

IMG_4842

So my firmware version is GM1.02B. Other information is available – read the CHDK wiki for more.

Getting the firmware update

There are lots of different versions of the CHDK available and it seems to be important that you get the right one. Visit the download page and click the link to the stable build – this takes you the list of available versions. Obviously, pick the right one for your camera – the SX20 files are near the end of the page. I went for this one:

sx20-102b-1.2.0-3537-full.zip

I downloaded and unzipped the archive locally, then removed the quarantine tag from the binary (something the OSX archive utility does to protect you from yourself):

$ xattr -d com.apple.quarantine DISKBOOT.BIN

Choosing the load method

There are two possible methods to set up your camera with this new software, neither of which alters the camera’s installed firmware. In the first and simplest, the SD card contains files that are loaded by the camera using the normal “firmware update” menu function. It doesn’t actually update the firmware: the code is loaded into RAM which means that the camera reverts to standard operation when it is switched off.

The second method requires a “bootable” SD card containing the CHDK and partitioned in the right way – a slightly more complex procedure being required to set this up. I wanted to go with the first method initially, principally because I am impatient, but discovered (because the required PS.FIR file was missing from the download archive) that the SX20 CHDK does not support the firmware update method. All the details for both methods are available on the wiki.

Preparing the SD card

First step in preparing for the “bootable” method is to partition and format the SD card. I used the OSX disk utility to do this on an 8GB SD card, setting up a 500MB MBR partition and the rest in a second partition, both formatted as FAT. The disk utility seemed to throw an error after partitioning and didn’t mount the first partition at this stage.

The next step requires the first partition to be unmounted anyway, as we convert it to a FAT16 partition by issuing this command using the appropriate disk identifier (disk1s1 in my case):

$ sudo newfs_msdos -F 16 -v Canon_DC -b 4096 -c 128 /dev/disk1s1

Ejecting and re-inserting the SD card shows the new partition arrangement is OK and both partitions mounted. The next step is to make the card bootable – first, by invoking the fdisk utility (you type the bold bits):

$ sudo fdisk -e /dev/disk1
fdisk: could not open MBR file [] No such file or directory <== IGNORE THIS
fdisk: 1> setpid 1
Partition id ('0' to disable) [0 - FF]: [B] (? for help) 1
fdisk:*1> write
Device could not be accessed exclusively.
A reboot will be needed for changes to take effect. OK? [n] y
Writing MBR at offset 0.
fdisk: 1> exit

Next, we have to edit the SD card’s Master Boot Record. Get a copy of it locally by issuing this:

$ sudo dd if=/dev/disk1s1 of=BootSector.bin bs=512 count=1

Remember to use the correct disk identifier (disk1s1 in my case). If you get “Resource busy”, it’s because the first partition is mounted – unmount (do not eject) it and try the dd command again. Next, the BootSector.bin file needs to be edited – I used HexEdit.app – to overwrite from position 0x40 the word BOOTDISK:

bs

You should finish up with a file that’s still exactly 512 bytes that you can dd back to the SD card boot partition:

$ sudo dd if=BootSector.bin of=/dev/disk1s1 bs=512 count=1

Remounting the partition (using disk utility), the final step in preparing the SD card is to copy the CHDK files over. The file DISKBOOT.BIN (and PS.FI?, if you have it) goes in the first partition, everything else from the archive goes in the second, larger partition.

Finally

Eject the card and move the lock switch to the LOCK position (this is required to make CHDK operate – in the UNLOCK position, it’s just a normal Powershot but limited to the first partition). Put the SD card in the camera and start it up – you’ll notice a new splash screen:

IMG_1865

You’ll also see some new items, like a battery monitor, but most of the CHDK functions are accessed through their own menus – you (and I) will have to spend a little time with the user manual, but look out for results on BlipFoto, Flickr or maybe even 500px.

WordPress XML-RPC Attack

This week, one of my sites, sptr.net, has been under a co-ordinated and sustained attack from what appears to be a botnet – a collective of several hundred virus-infected computers running Microsoft Windows. The attack comprises attempts to use the remote procedure call methods built into WordPress to post unauthorised content.

Detection

I was notified by one of my independent monitoring services that the site was having trouble some time after the attack began. It appears that once triggered by the attacker, it takes a while for the command to spread to a significant number of infected machines – this is reasonable if you assume the greatest number of infected PCs is in the USA. The attack peaked around the middle of the day in Scotland, coincident with the switching on of computers as the sun moved East to West across the continental US. Although the server remained operational, it was struggling to continue to respond to requests in a reasonable time as the CPU usage soared way above 1000% of nominal maximum. A look at the top processes on the server showed that it was trying to keep things together:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
12345 xxxxxxx    20   0 55992 36080  7128 R 49.0  6.9  0:15.73 [see below]
12346 xxxxxxx    20   0 56036 36124  7188 R 46.0  6.9  0:04.96 [see below]
12347 xxxxxxx    20   0 55940 36076  7128 R 46.0  6.9  0:03.82 [see below]
12349 xxxxxxx    20   0 55912 35908  6984 R 46.0  6.8  0:07.88 [see below]
12340 xxxxxxx    20   0 55976 36116  7180 R 46.0  6.9  0:03.59 [see below]
12342 xxxxxxx    20   0 55940 36064  7128 R 44.0  6.9  0:07.21 [see below]
12341 xxxxxxx    20   0 55948 36140  7196 R 44.0  6.9  0:34.79 [see below]
12343 xxxxxxx    20   0 55972 36248  7276 R 44.0  6.9  2:20.11 [see below]

The command attempted showed that it was an attack on a php script:

/usr/bin/php-cgi -c /var/www/vhosts/sptr.net/etc/php.ini

Further investigation

Looking at the server access logs identified the specific script targeted by the attacker, the machines and methodology involved. The range of IP addresses showed that the infected PCs were world-wide (in the sample below, India, Poland, Egypt, Thailand, Algeria, Brazil and Pakistan).

106.76.44.110 - - [10/Jul/2014:14:03:19 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
194.50.157.187 - - [10/Jul/2014:14:03:34 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.235.83.103 - - [10/Jul/2014:14:03:43 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
171.6.204.105 - - [10/Jul/2014:14:03:54 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.107.87.186 - - [10/Jul/2014:14:04:04 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
179.186.51.47 - - [10/Jul/2014:14:04:06 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
39.44.61.247 - - [10/Jul/2014:14:04:14 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

Mitigation

Restarting the VPS container made no difference. CPU usage remained very high. Installing a plugin to disable XML-RPC in WordPress seemed to make things better, probably because of the response time improvement but as the day progressed, the attack seemed to abate and the server was coping better with CPU usage falling below 100% nominal maximum. The log sample above is from today, when the attacks have fallen to a few per minute instead of the hundreds per second on Tuesday. It looks like the botnet is learning that there are robust passwords on the system that will take too long to guess and is giving up.

Brute force solution

I’m not happy with this constant knocking at my door, however, so have decided that I don’t need a door there at all. Removing the target script doesn’t directly affect the rate of attack, it changes the 200 response to a 404 (page not found), which is quickly delivered.

94.55.132.13 - - [10/Jul/2014:14:09:13 +0000] "POST /xmlrpc.php HTTP/1.1" 404 430 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

MacBook Memory Storage upgrade

Noticing the performance of my 5-year-old MacBook Pro was becoming – well, noticeable, I thought it might be time to replace it. The trouble with that is cost. That, and the inconvenience of it. I asked around some of my Mac buddies, did a bit of googling (other search engines are available, but nobody uses them), and sought advice from the Apple Store. The hive mind seemed to favour and recommend an increase in RAM and a replacement of the hard drive with a solid state drive (SSD). Here’s how it went.

1. Backup/clone

The first thing I had to do was create a copy of all the files on my 256GB internal hard drive. The drive was running with about 75GB unused spaceIMG_2699 but I had a 256GB external USB hard drive available for this purpose. You can buy USB enclosures for about a fiver which would allow you to clone your old hard drive straight to your new SSD but I took the intermediate step because I wanted to tread carefully and also create a backup of the drive and settings, something I hadn’t properly done before. I am a regular backer-upper of my files, sometimes keeping several redundant copies of important things. The backup tool of choice is Bombich’s Carbon Copy Cloner. I like this software because (a) it works, (b) it anticipates the user’s possible lack of caution or experience and (c) helps the user do the right thing. There’s a 30-day trial available but if you’re spending money on an upgrade, why not include the just-under-thirty-quid full licence as part of your budget and get a decent backup solution while you’re at it?

2. Buy the right bits

The Apple Store people were quite clear and unhesitant in telling me that Crucial Memory is the place to buy your upgraded memory chips and drives. There is a config checker tool on their site that will tell you what you need and guarantee it’s right if you buy from them. If you’re certain that it was 2009 when you bought your Mac, then you might not need this. I was certain but used the tool anyway. I was impressed with the speed of service and delivery from Crucial. Ordering late on Wednesday night and not paying extra for faster postage, the new drive and memory dropped through my letterbox on Friday morning.

IMG_2700IMG_2701

3. Replace

The replacement is pretty straightforward. The back of the MBP is retained by 10 cross-head screws (you’ll need instrument screwdIMG_2702rivers), 3 of which are longer than the rest. Remember where they go. I recommend that you read about and follow good static discharge protection procedures, before you fry anything electronic and sensitive. The big yellow sticker is trying to tell you that. I used a bonding wrist strap that I connected to the case of the laptop and generally tried hard not to touch anything metal at all.

The memory cards are held in their slots by two plastic clips, one on each side, that you pull aside to allow the card to pop up for easy removal. The second card slot on my MBP is under the first.

IMG_2704

Make sure you align the new memory cards correctly before trying to press them into the holder. There is a notch in the connector side which requires you to put it in the correct way round. On mine the notch is to the left of centre. If you refer to the top image of the new memory, you’ll see this meant that I had to put the memory in with that “Crucial” label facing down (i.e. on the keyboard side).

IMG_2705

The HDD came out easily enough. There are two retaining brackets held in by a total of four cross-head screws. You can only see one clearly in the picture, at the bottom left. Remove these and disconnect the HDD carefully. For some reason there was a rubber blind washer in the case by this connector (see picture). I have no idea why. I have put it back where it was.

The HDD is fitted with four spigots which are needed for the new SSD. They are Torx screws, of a size I do not have the tool for, so I removed them with pliers and installed them in the SSD the same way. Reassembly is the reverse of what you’ve done so far.

4. Restore

Once your MBP is back in one piece, so to speak, you will need to boot it from the external clone if you made one, or straight from the new SSD if you didn’t take that step. If like me, you used the external clone, then you have another couple of hours to wait while you clone onto the new SSD.

5. Outcomes

What I have now is the same MBP as before only a lot faster and quieter. The fan did run fast for the first hour or two of operation but this is partly due to Google Drive getting its knickers in a twist. GDrive cannot cope with moving your GDrive folder to a different place – it has to start from scratch, synching from the web. Yet another good reason not to use GDrive. Dropbox needs you to log in again. Copy.com just copes.

Some of your file associations will have reverted to default settings in the process of setting up partitions on the clone. Apart from that, it looks exactly the same as it did before but somehow feels new. Browsing, email and compiling are all noticeably faster.

Thanks to those who steered me in the right direction; also to CarbonCopy and to Crucial for great products and customer support.

6. Next steps

Next? Probably a new battery. Even with the low-energy SSD, I’m probably down to about 3 or 4 hours maximum use from my five-year old battery which has been asking for service for months.

WordPress Redirect Loop

WordPress is a brilliant tool, probably the best of the CMSs – Google says so – but every now and then it can stop you in your tracks. It did this today as I was setting up a new site for Marc Walker, the British Biathlon veteran and team manager who is retiring from Her Majesty’s service in August to set up a very special personal trainer business in Knutsford.

Marc Walker (image copyright Marcel Laponder CC-BY-3.0)
Marc Walker (image copyright Marcel Laponder CC-BY-3.0)

I hit a wee problem with an unexpected redirect loop when trying to access the back end. There are plenty of articles and “fixes” available on the web, none of which were relevant to my installation and most of which relate to permalinks and .htaccess. Because my installation is a long-standing derivative of WPMU or multi-site, it could not have been that.

For others in the same position, here’s what my install looks like:

  • LAMP hosted (on a VPS)
  • Version 3.9.1
  • Multiple WP sites, domain-mapped

I had a while ago, for some reason I have now forgotten, network disabled the default WordPress themes. When I added this new site, created the new admin user and mapped the domain, I found that the admin or login pages simply got stuck in a redirect loop.

The fix was easy enough – I simply had to enable Twenty-Fourteen (the default WP theme) for the new site via the network admin panel.

If you want to visit Marc’s new site, it’s at AvantgardePT.com. His new business will start up in August and will have a strong European baseline from his track record in Biathlon, military fitness, Iron Man, and an impressive bunch of competitive sports.

Surgeons Tweeting

I was at a talk given by George Veletsianos today, under the title of “Acts of defiance and personal sharing when academics use social media”. This was a thought-provoking session and not the first of his I’ll be attending this week, hopefully. I’ll not report the details of the talk here, but I will point you towards George’s Networked Scholars open course and this little scribble I made as I thought about how pervasive the use of social media has become.

SurgeonsTweeting

CentOS 6.5 on MSWind

Rather than make a pig’s ear out of my live VPS by testing out new Ruby code I’m playing with, I thought it would be prudent to have a machine that I can break without upsetting users. I have an Atom-based Advent netbook which only ever gets played with occasionally and this afternoon, seems quite willing to volunteer for a rebuild as a CentOS server. The world loves a volunteer. Continue reading “CentOS 6.5 on MSWind”