Photographer portfolios – Koken

I set up a test site for a photography journal over at http://dev.cullaloe.net/koken/. I’ve been trying a number of alternatives and hosting options: koken is php software that runs on a Linux server over a mySql database and Apache. I happen to have one of those at dev.cullaloe.net.

So far, it looks like it has really nice features, including a tight integration with Adobe Lightroom that allows you to set up a direct publishing link. Most of the images on the site are reduced-size versions of some of my “good” photos.

I have found some bugs and irritations: the admin back-end fails completely from time to time, requiring clearing of api file cache over FTP. Themes are limited but they are quite pretty, I think, with development quite straightforward.

The original developer of this programme sold out to a new owner last year, I believe, but there seems to be some investment in bug fixing and development.

So far I don’t think it’s stable enough for a main online portfolio: you should probably just buy yourself a 500px Awesome membership for that and use the portfolio feature of that site.

Learn code

codeOver fifty years ago, my father was a US Air Force signals operator: he, like any other professional in communication, had to learn the languages of communication, command and control. I still have the LP (long-playing record, what the kids call “vinyl” now, although these weren’t vinyl) record set that he listened to as he learned Morse Code.

Today’s young people live in a world of communication and it is increasingly important for them – and all users – to at least have an appreciation of the languages used by the systems that pervade our modern lives. Learning to code – and the computational thinking that goes with it – is fun and interesting as well as being intellectually good for you. It’s also potentially lucrative: coding skills are at a premium, wherever you are in the world. While there is still a need for certain people to know Morse Code, there are many other languages to know about: from the languages of data to the logic of a sick (sic) 3D immersive games experience.

I have carried a link to CodeCademy on this site for some time because they offer some excellent resources and courses for people to learn how to code. I have used some of them myself and recommend them highly. If you’re not sure where to start, there is  a visual overview of the main programming languages and possible benefits of learning each one to help you make an informed decision. You can find it here: http://wiht.link/learncodeguide.

DISCLAIMER: I am not connected with Codecademy and have received no financial or other incentive to write this post. The infographic is not Codecademy’s and includes links to other free online places where you can learn. It’s just a good idea and a good place to get started. Get on with it!

GNU PSPP on OSX Yosemite

I have a project I’m working on that requires the use of a data analysis tool like IBM’s SPSS but at about six thousand dollars per year, it’s a little out of reach. There is an open source project, fortunately, that provides all the functionality I need for a lot less.

PSPP is, according to the project website:

“…designed as a Free replacement for SPSS. That is to say, it behaves as experienced SPSS users would expect, and their system files and syntax files can be used in PSPP with little or no modification, and will produce similar results (the actual numbers should be identical). The number of variables and cases is limited only by the computer architecture.”

There are a number of ways of getting PSPP depending on your operating system: I am a Mac OSX user running 10.10.5 Yosemite so installed it using MacPorts. As this is a brand new machine I’m installing it on, I needed to install MacPorts first: download and run the install package from the download page, update and then run the install (you need super user privilege):

$ sudo port selfupdate
$ sudo port install pspp

This will give you a working PSPP from the command line. If you want to use the graphical user interface over PSPP, known as PSPPIRE, you’ll need to update your X11 DISPLAY driver by downloading and installing XQuartz which is a community produced X-window server assisted but not supported by Apple. Once you’ve installed Quartz, you’ll need to log out and in again to update the DISPLAY environment. Once this is done you can launch the GUI version of PSPP from the command line:

$ sudo psppire

This allows you to work with your SPSS data sets and command files almost without modification.

Hello, World! Nice HAT.

Hello World!
Hello World!
Hello World!

For those of you trying to get to grips with the Raspberry Pi’s Astro-Pi Sense HAT… wait, what?

The Raspberry Pi is the amazing, powerful and compact computer-on-a-board that has got children of all ages around the world coding and investigating computational thinking. For less than fifty bucks, this machine includes a fast processor, a decent amount of RAM and USB, Ethernet and HDMI interfaces that let you connect it up to a TV and keyboard and do almost anything you can do on machines twenty times the price (like write this post, for example). If, like me, you like things tidy, you can add a box to put it in and if, like me, you’re a physics teacher, you can add on a sense HAT (Hardware Attached on Top) that is exactly the same as the kit to be used by Astronaut Tim Peake on the International Space Station to conduct experiments in space using the many sensors on board the HAT.

The whole kit cost me £75 including power supply and SD card with operating system (Raspbian – a version of Debian Linux) software pre-installed.

The setting up is simple and step-by-step, I got it working as a stand-alone machine before installing the Sense HAT. I had to take a knife to the official Raspberry Pi box once the HAT was added to the Pi board – it almost fits but just needs a little adjustment near the corner of the lid to make it snap into place. There are plenty of resources on the web to help you get started but development has taken place at such a pace that some of the guides don’t quite match the installed software. The Getting Started with the Sense Hat page at raspberrypi.org is no exception. There is a simple “Hello World!” program:

from sense_hat import SenseHat
sense=SenseHat()
sense.show_message("Hello, World!")

On my Pi 3B, I got an error at this point:

Traceback (most recent call last):
 File "/home/pi/hw.py", line 1, in <module>
 from sense_hat import SenseHat
 File "/usr/lib/python3/dist-packages/sense_hat/__init__.py", line 2, in <module>
 from .sense_hat import SenseHat, SenseHat as AstroPi
 File "/usr/lib/python3/dist-packages/sense_hat/sense_hat.py", line 14, in <module>
 from PIL import Image # pillow
ImportError: No module named PIL

This was because there was a step missing from the sense-HAT installation instructions which should have read:

sudo apt-get install sense-hat
sudo pip-3.2 install pillow

The second line was omitted, leading to the above error. Once the pillow module was installed OK, running the test python script above produced the results I was looking for (see picture). There is a lot of decent documentation at pythonhosted.org that I hope to take a look at in order to get some ideas for physics teaching using the sensors in my new HAT. I’m loving the sense of really playing (and learning) with computers: those of you old enough will remember the same joy of getting a BASIC program to run properly on your BBC or ZX Spectrum. Suddenly, computers are fun again.

Last.fm scrobbler v2 doesn’t work

lastfmThis morning, I finally gave up trying to sort out the scrobbling problem I’ve been having since December. The current Last.fm scrobbler, version 2, is just not functioning, so I’ve reverted to version 1.5, losing 4 months’ scrobbles in the process. Not impressed. Why can’t anybody write software that works any more?

The problem has been that although the Last.fm app on my OSX device seems to work, reporting scrobbles normally, these seem to get stuck in cache. In the app, these tracks show as “cached” and do not appear on my last.fm profile.

Long story short, if you’re a Mac user having trouble with last.fm not scrobbling your tracks, delete the last.fm scrobbler, empty the trash and download the older version 1.5 here (dmg).

PHP Mail and stripping of lines in Microsoft Outlook

A client recently contacted me about problems with the formatting of messages he was getting from a php contact form on his site. He asked if I could insert a couple of CRLFs to make it easier to read and to stop it breaking the email links in the message.

The client’s site is one of those creaking anachronistic beasts, from the days of hand-hacked HTML, which is full of things that work just well enough to enable him to concentrate on his business. I’ve been trying to get him to move to a CMS like WordPress for several years now, but he’s not quite able to let go.

The contact form had not been a problem, as far as I knew, but all this while he has been putting up with messages from the site that look a bit like this:

Name: FredEmail: fred@bloggs.comTel: 09999899988Hi I was
wondering blah blah blah blah?RegardsFred

On my machines, they look like this:

Name: Fred
Email: fred@bloggs.com
Tel: 09999899988
Hi I was wondering blah blah blah blah?
Regards
Fred

It seems that there is a “feature” that has existed in Microsoft Outlook since 2002, at least. What it does, often without letting the user know, is strip out any formatting of lines in the original message and replaces it with what it thinks you’d prefer. In text-only messages, this results in what you see in the first example above.

There’s a lot written about this, much of it along the lines of altering the user’s practice to include workarounds that are only necessary because Microsoft can’t write good code. See here, for example, or here for one of the empirical solutions that suggests changing code to accommodate Outlook’s perverse behaviour. Many others remain baffled. However, thanks to a bit of forensic inquiry by Matthew Truesdell, there are some rules that can be interpreted in such a way that allows the php script to work for all users. Matthew posted the rules he found in Outlook 2007, over on Stack Overflow: I’ve adapted from those here, slightly, using the term “mode” to mean the behaviour of Outlook that strips out line breaks from plain text messages. Lines are assessed one at a time:

  • Every message starts with the mode OFF.
  • Lines 40 characters or longer switch the mode ON.
  • Lines that end with a full stop (.), question mark (?), exclamation (!) or colon (:) switch the mode OFF.
  • Lines that turn the mode off will start with a line break, but will turn it back on if they are longer than 40 characters.
  • Lines that start or end with a tab turn the mode off.
  • Lines that start with 2 or more spaces turn the mode off.
  • Lines that end with 3 or more spaces turn the mode off.

So it seems that one way to trick Outlook is to add 3 spaces at the end of each line, which in the code is just before the CRLF. I tried this, but be careful if you rely on it: different versions of Outlook do different things. Outlook 2013 is still stripping out the line breaks on the client machine, so we have this:

Name: Fred   Email: fred@bloggs.com   Tel: 09999899988
Hi I was wondering blah blah blah blah?   Regards   Fred

Which is still not satisfactory but at least allows him to click on the email address for a quicker response.

On my own machine (OSX Yosemite), Outlook 10 seems to be working as you’d expect, without interfering with the line breaks. Gmail works fine also. I think that’s as far as I’m going to take it.

Adobe Lightroom 5.7 Crash

Adobe has recently released an update for its Lightroom 5 photo management system which on OSX Yosemite does not work. The application crashes. I’ve gone through all the usual precious time-wasting permutations including completely uninstalling, clearing trash and reinstalling, even re-downloading from Amazon (where I bought it from a couple of months ago). The problem seems to be Adobe, like everyone else, is developing code for the majority market, i.e. the Microsoft Slaves.

A lot of Mac users, me included, operate with the flexibility of case-sensitive drives and here lies the problem. Adobe’s sloppy coders have assumed that all systems are case insensitive. The error log gives a clue:

Library not loaded: @executable_path/../Frameworks/asneu.framework/versions/a/asneu

This library is actually located in the application folder in:

/Content/Frameworks/asneu.framework/Versions/A/asneu

Changing the path to match that expected by the application (V becomes v, A becomes a) allows it to run OK. I’m not aware of any other case-sensitivity issues with LR5.7 – it seems to work just fine.

Tip: if you’re a LR user, the 500px plugin makes publishing to your favourite photo showcase easy.

 

Hacking the Canon Powershot SX20 IS

I’ve had my Canon Powershot SX20 IS camera for a few years now and have always regarded it as a stepping-stone to a better, “proper” camera. The problem is I have never quite got to the point where I can justify shelling out the considerable wonga to take the next step.

What I’d like is a modern digital equivalent to my brilliant old Nikon FM that served me well for a number of years, with up to date features as well as the best of the old. Two things in particular have annoyed me about the SX20 – the maximum exposure time of 15 seconds and the digital compression which irrationally leaves me with FOMO – something is missing from my photographs.

Having resolved not to spend a grand on a new camera, instead I lobbed a hundred quid into the Physics Pixies UNICEF appeal and set about altering the camera I have to deal with the two “problems”. The alterations amount to a firmware update using the CHDK (Canon Hack Development Kit) firmware addon. This is now an open-source project built on the work of programmer VitalyB’s RAW enabler and Andrei Gratchev’s development kit. The firmware update now includes a number of other really nice features including time-lapse, motion detection and bracketing of exposure and focus.

Finding out the camera’s firmware

The EXIF data in a digital photograph tells you quite a lot about the camera that took it and the settings used – see, for example, this picture on Flickr. Click “show EXIF”. This tells me almost but not quite enough about the firmware Revision – 1.02 rev 2.00. Your camera will tell you, though. First, create an empty file called ver.req in the root of the SD card. I did this on a MacBook Pro with the SD card in a slot on the laptop by issuing these commands:

$ cd Volumes/CANON_DC/
$ touch ver.req

Put the card in your camera and start it up in playback mode. From the main screen (should be displaying NO IMAGE for no images on the card), press FUNC SET and DISP. buttons and the camera will display a screen like this for about 5 seconds:

IMG_4842

So my firmware version is GM1.02B. Other information is available – read the CHDK wiki for more.

Getting the firmware update

There are lots of different versions of the CHDK available and it seems to be important that you get the right one. Visit the download page and click the link to the stable build – this takes you the list of available versions. Obviously, pick the right one for your camera – the SX20 files are near the end of the page. I went for this one:

sx20-102b-1.2.0-3537-full.zip

I downloaded and unzipped the archive locally, then removed the quarantine tag from the binary (something the OSX archive utility does to protect you from yourself):

$ xattr -d com.apple.quarantine DISKBOOT.BIN

Choosing the load method

There are two possible methods to set up your camera with this new software, neither of which alters the camera’s installed firmware. In the first and simplest, the SD card contains files that are loaded by the camera using the normal “firmware update” menu function. It doesn’t actually update the firmware: the code is loaded into RAM which means that the camera reverts to standard operation when it is switched off.

The second method requires a “bootable” SD card containing the CHDK and partitioned in the right way – a slightly more complex procedure being required to set this up. I wanted to go with the first method initially, principally because I am impatient, but discovered (because the required PS.FIR file was missing from the download archive) that the SX20 CHDK does not support the firmware update method. All the details for both methods are available on the wiki.

Preparing the SD card

First step in preparing for the “bootable” method is to partition and format the SD card. I used the OSX disk utility to do this on an 8GB SD card, setting up a 500MB MBR partition and the rest in a second partition, both formatted as FAT. The disk utility seemed to throw an error after partitioning and didn’t mount the first partition at this stage.

The next step requires the first partition to be unmounted anyway, as we convert it to a FAT16 partition by issuing this command using the appropriate disk identifier (disk1s1 in my case):

$ sudo newfs_msdos -F 16 -v Canon_DC -b 4096 -c 128 /dev/disk1s1

Ejecting and re-inserting the SD card shows the new partition arrangement is OK and both partitions mounted. The next step is to make the card bootable – first, by invoking the fdisk utility (you type the bold bits):

$ sudo fdisk -e /dev/disk1
fdisk: could not open MBR file [] No such file or directory <== IGNORE THIS
fdisk: 1> setpid 1
Partition id ('0' to disable) [0 - FF]: [B] (? for help) 1
fdisk:*1> write
Device could not be accessed exclusively.
A reboot will be needed for changes to take effect. OK? [n] y
Writing MBR at offset 0.
fdisk: 1> exit

Next, we have to edit the SD card’s Master Boot Record. Get a copy of it locally by issuing this:

$ sudo dd if=/dev/disk1s1 of=BootSector.bin bs=512 count=1

Remember to use the correct disk identifier (disk1s1 in my case). If you get “Resource busy”, it’s because the first partition is mounted – unmount (do not eject) it and try the dd command again. Next, the BootSector.bin file needs to be edited – I used HexEdit.app – to overwrite from position 0x40 the word BOOTDISK:

bs

You should finish up with a file that’s still exactly 512 bytes that you can dd back to the SD card boot partition:

$ sudo dd if=BootSector.bin of=/dev/disk1s1 bs=512 count=1

Remounting the partition (using disk utility), the final step in preparing the SD card is to copy the CHDK files over. The file DISKBOOT.BIN (and PS.FI?, if you have it) goes in the first partition, everything else from the archive goes in the second, larger partition.

Finally

Eject the card and move the lock switch to the LOCK position (this is required to make CHDK operate – in the UNLOCK position, it’s just a normal Powershot but limited to the first partition). Put the SD card in the camera and start it up – you’ll notice a new splash screen:

IMG_1865

You’ll also see some new items, like a battery monitor, but most of the CHDK functions are accessed through their own menus – you (and I) will have to spend a little time with the user manual, but look out for results on BlipFoto, Flickr or maybe even 500px.

WordPress XML-RPC Attack

This week, one of my sites, sptr.net, has been under a co-ordinated and sustained attack from what appears to be a botnet – a collective of several hundred virus-infected computers running Microsoft Windows. The attack comprises attempts to use the remote procedure call methods built into WordPress to post unauthorised content.

Detection

I was notified by one of my independent monitoring services that the site was having trouble some time after the attack began. It appears that once triggered by the attacker, it takes a while for the command to spread to a significant number of infected machines – this is reasonable if you assume the greatest number of infected PCs is in the USA. The attack peaked around the middle of the day in Scotland, coincident with the switching on of computers as the sun moved East to West across the continental US. Although the server remained operational, it was struggling to continue to respond to requests in a reasonable time as the CPU usage soared way above 1000% of nominal maximum. A look at the top processes on the server showed that it was trying to keep things together:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
12345 xxxxxxx    20   0 55992 36080  7128 R 49.0  6.9  0:15.73 [see below]
12346 xxxxxxx    20   0 56036 36124  7188 R 46.0  6.9  0:04.96 [see below]
12347 xxxxxxx    20   0 55940 36076  7128 R 46.0  6.9  0:03.82 [see below]
12349 xxxxxxx    20   0 55912 35908  6984 R 46.0  6.8  0:07.88 [see below]
12340 xxxxxxx    20   0 55976 36116  7180 R 46.0  6.9  0:03.59 [see below]
12342 xxxxxxx    20   0 55940 36064  7128 R 44.0  6.9  0:07.21 [see below]
12341 xxxxxxx    20   0 55948 36140  7196 R 44.0  6.9  0:34.79 [see below]
12343 xxxxxxx    20   0 55972 36248  7276 R 44.0  6.9  2:20.11 [see below]

The command attempted showed that it was an attack on a php script:

/usr/bin/php-cgi -c /var/www/vhosts/sptr.net/etc/php.ini

Further investigation

Looking at the server access logs identified the specific script targeted by the attacker, the machines and methodology involved. The range of IP addresses showed that the infected PCs were world-wide (in the sample below, India, Poland, Egypt, Thailand, Algeria, Brazil and Pakistan).

106.76.44.110 - - [10/Jul/2014:14:03:19 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
194.50.157.187 - - [10/Jul/2014:14:03:34 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.235.83.103 - - [10/Jul/2014:14:03:43 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
171.6.204.105 - - [10/Jul/2014:14:03:54 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
41.107.87.186 - - [10/Jul/2014:14:04:04 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
179.186.51.47 - - [10/Jul/2014:14:04:06 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"
39.44.61.247 - - [10/Jul/2014:14:04:14 +0000] "POST /xmlrpc.php HTTP/1.1" 200 159 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"

Mitigation

Restarting the VPS container made no difference. CPU usage remained very high. Installing a plugin to disable XML-RPC in WordPress seemed to make things better, probably because of the response time improvement but as the day progressed, the attack seemed to abate and the server was coping better with CPU usage falling below 100% nominal maximum. The log sample above is from today, when the attacks have fallen to a few per minute instead of the hundreds per second on Tuesday. It looks like the botnet is learning that there are robust passwords on the system that will take too long to guess and is giving up.

Brute force solution

I’m not happy with this constant knocking at my door, however, so have decided that I don’t need a door there at all. Removing the target script doesn’t directly affect the rate of attack, it changes the 200 response to a 404 (page not found), which is quickly delivered.

94.55.132.13 - - [10/Jul/2014:14:09:13 +0000] "POST /xmlrpc.php HTTP/1.1" 404 430 "-" "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)"