Rename Image Files in Folders

This is a Python script to walk a directory tree, renaming .jpg and .JPG files. It was written for a client who had a directory containing multiple directories, each containing zero or more (up to 100) image files. Most of the images were named <abitrary name 1>.jpg or <arbitrary name 2>.JPG.
The client wanted each folder’s images to be numbered 001.jpg to nnn.jpg for some further process.

import os
import imghdr

def renamefiles(path):
rootstructure=os.path.abspath(path)
n = 0
for root, dirs, files in os.walk(rootstructure):
for f in files:
if os.getcwd() != root:
n = 0 # Reset the counter
os.chdir(root)

type = imghdr.what(f)
if type == 'jpeg':
n +=1
os.rename(f, str(n).zfill(3) + ".jpg")

# Specify the full path to the directory here.
renamefiles("/Users/myuser/Desktop/TheImageFolder")

Grav Error 403 in Admin Panel

I’m trying out Grav, a flat file CMS, on my server. It looks promising, having just installed it with the admin plugin. It seems that there is a fairly well-known problem with false positives from apache’s mod security web application firewall software, which manifests as an annoying 403 error banner in the admin pages (and the notifications panel not loading).

This is easily fixed by whitelisting mod security rule 350147. In Plesk, mod security rules can be whitelisted in the Server Management | Tools and Settings | Security | Web Application Firewall (ModSecurity) page. Enter the security rule ID (in this case 350147) in the box at the end of the page and click apply.

There is a security risk associated with this and users should assess the risk to their servers before implementing this workaround.

Upgrading solr server from 6.6.0 to 7.3.1

I use Apache’s Solr to provide a global search facility on Moodle. Now that my courses have ended for the summer, it’s time to bite the bullet and upgrade the Solr server software from version 6.6.0 which I installed last year, to the current 7.3.1. This turned out to be more straightforward than I feared, and did not require me to touch the PHP solr module that I had to compile from source when I installed it the first time. Here are the steps:

# cd /opt
# wget http://apache.mirrors.nublue.co.uk/lucene/solr/7.3.1/solr-7.3.1.tgz
# tar zxvf solr-7.3.1.tgz
# cp solr-7.3.1/bin/install_solr_service.sh .
# rm -rf solr-7.3.1
# ./install_solr_service.sh solr-7.3.1.tgz -f

Notice the -f flag which tells the script to upgrade an existing installation. The script stops the currently running instance, extracts the new code and starts the instance. A quick check of the admin interface on port 8983 shows the new code running OK, the cores intact, and the client service on Moodle nominal.

EDIT: At present (June 2018) Solr 7 is not supported on Moodle 35. The latest version of the Solr server that works with Moodle 35 is 6.6.4: the instructions above will install Solr 7.

If you want global search to work with Moodle 35, replace “7.3.1” with “6.6.4” and fetch the code using wget from the Apache archive at http://archive.apache.org/dist/lucene/solr/6.6.4/solr-6.6.4.tgz.

Google Research releases exoplanet discovery ML code

This week, Google Research released the AI code used in the discovery of two new exoplanets. Their article includes an accessible intro to planet hunting and machine learning for those interested in either topic.

The associated paper is mostly about the AI but includes a good sense of just what real planetary research looks like and how far it’s come: Shallue, C.J. & Vanderburg, A., 2017. Identifying Exoplanets with Deep Learning: A Five Planet Resonant Chain around Kepler-80 and an Eighth Planet around Kepler-90. The Astronomical Journal, 155(2), p.94.

There’s an interview with Chris Shallue, the lead author on the paper on the TWIMLAI podcast which makes interesting listening for anyone interested in machine learning, AI or exoplanetary research.

tryGit: it’s the teach, not the tech.

I picked up a link to a free online course from a recent Linux Voice podcast. I tried the course and wasn’t really impressed with it. The interface is nice, with an embedded virtual terminal to let you practice typing in the commands but the pedagogy is pretty weak. There’s nothing in the presentation that indicates that the designers understand how to construct understanding: all this lovely bit of code is doing is rehearsing a list of commands (and there’s even a click-once shortcut if you can’t be bothered actually typing). It’s more of a checklist than a course. It would be easy to turn it into a really effective bit of online learning with the addition of some better structure and graphics, and maybe a little assessment for learning. A shame, really, as it is clearly a loss leader to sell the Code School itself: I am in the market for some good quality online learning in their area, but I’m not likely to look any further at their catalogue.

ownCloud installation on Centos 7

For some time, I’ve wanted to have a calendaring tool independent of Google Calendar, which has become a central tool to my productivity and a source of concern as to how much data profiling results from it.

This afternoon, I installed the open source ownCloud file storage, calendar and contacts suite on my Centos VPS. It was a straightforward exercise:

  • Create a subdomain on the server and switch it to use PHP 5.6. Add /dev/urandom to open_basedir in php settings.
  • Make a data folder behind the web root, chowned to the web user.
  • Create a MySQL database for the ownCloud service.
  • In the web root folder, get the software:

curl -O https://download.owncloud.org/community/owncloud-10.0.3.tar.bz2

  • Check the MD5 hash, chown and extract. Copy the extracted files into the root folder (be careful to include dotfiles, e.g. cp owncloud/* . and cp owncloud/.* .)
  • Visit the domain to configure the installation.

What this server now provides is an independent calendar service, contacts, and secure file storage, at no additional cost and under my own secure control.

Migrating from AudioBoom

I’ve been podcasting using Audioboo (now Audioboom) since 2011 and supported the service by paying for a podcaster subscription for much of that time. Audioboom introduced a new pricing structure that I can’t sustain so I need to migrate to a new solution.
Using my LAMP server, which has bandwidth capacity for my podcasts, I set up a new subdomain on http://podcast.cullaloe.net and installed podcastgenerator 2.6. I uploaded a selection of episodes over FTP and used the import function in the admin interface (I had to change the timestamp using touch -t yymmddhhmm.ss filename.mp3 for each episode to get them into the right time order) finally editing in the description for each show.
For the most part, my episodes had embedded images in the mp3 file, but where they did not, I thought about having to edit the ID3 tags. I read about a method for avoiding this which includes posting an image (say, 300×300) for each track within the long description field in the admin interface of the new software, but this doesn’t seem to work. There’s a command line way to pull an image into an .mp3 file which sometimes works as an alternative:
ffmpeg -i input.mp3 -i cover.png -map 0:0 -map 1:0 -c copy -id3v2_version 3 -metadata:s:v title="Album cover" -metadata:s:v comment="Cover (Front)" out.mp3
If that doesn’t work, you can just drop an appropriate jpg/png in the images folder of the same name as the episode audio file.
The iTunes podcast feed was easily updated by visiting https://podcastsconnect.apple.com/  and pointing to the new feed. Note that podcast artwork must now be between 1400 x 1400 and 3000 x 3000 pixels, JPG or PNG, in RGB color space, and hosted on a server that allows HTTP head requests. The iTunes podcast updated overnight without any difficulty.
Finally, I emailed audioboom to request they apply a 301 or iTunes <itunes:new-feed-url/> redirect to the old feed (pointing to the new one).

Global Search for Moodle on Centos

My students are using a Moodle VLE to access resources and teaching materials and it became evident that some kind of global search function would help them find things quickly, especially later in the programme when they come to write their assignments.

I’m running Moodle on a CentOS 7.3 virtual private server with Plesk Onyx. The server hosts several other sites running WordPress, bespoke PHP and some other bits and pieces including the usual mail services. Some of the containers require the OS-standard PHP5.4 but a recent upgrade to Moodle 3.3 required me to switch the container to PHP 7.0.

Installing Global Search was a little tricky because of the multiple PHP versions running on the server, but I eventually figured it out to these key steps:

Install the Solr Server

$ cd /opt
$ wget http://apache.mirrors.nublue.co.uk/lucene/solr/6.6.0/solr-6.6.0.tgz
$ tar zxvf solr-6.6.0.tgz
$ cp solr-6.6.0/bin/install_solr_service.sh .
$ rm -rf solr-6.6.0
$ ./install_solr_service.sh solr-6.6.0.tgz
$ chkconfig solr on
$ su - solr -c "/opt/solr/bin/solr create_core -c moodle"

You should be able to visit http://your-domain.tld:8983 to verify the Solr server is running OK.

Secure the Solr Server

By default, Solr is open to the world. You might want to secure it by adding this at the end of /opt/solr/server/etc/webdefault.xml:

  <security-constraint>
   <web-resource-collection>
       <web-resource-name>Solr Administration</web-resource-name>
       <url-pattern>/*</url-pattern>
   </web-resource-collection>
   <auth-constraint>
       <role-name>solr-admin</role-name>
   </auth-constraint>
  </security-constraint>

  <login-config>
   <auth-method>BASIC</auth-method>
   <realm-name>Solr Administration</realm-name>
  </login-config>

Create a file in the same directory called realm.properties containing your chosen authentication details (matching the role above) in a single line:

admin: password, solr-admin

Finally, add this just before the last line in jetty.xml in the same directory:

<Call name="addBean">
 <Arg>
  <New class="org.eclipse.jetty.security.HashLoginService">
    <Set name="name">Solr Administration</Set>
    <Set name="config"><SystemProperty name="jetty.home" default="."/>/etc/realm.properties</Set>
    <Set name="refreshInterval">0</Set>
  </New>
 </Arg>
</Call>

Install the PHP Solr Extension

$ rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
$ yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear

You’ll need to build the extension using the right versions of phpize and php-config for your version of PHP, in my case, 7.0:

$ cd /opt
$ curl -O https://pecl.php.net/get/solr-2.4.0.tgz
$ tar zxvf solr-2.4.0.tgz
$ cd solr-2.4.0/
$ ../plesk/php/7.0/bin/phpize
$ ./configure --with-php-config=/opt/plesk/php/7.0/bin/php-config
$ make
$ make install
$ cp /opt/solr-2.4.0/modules/solr.so /opt/plesk/php/7.0/lib64/php/modules/
$ sudo service httpd restart

Visit the Site administration / ▶︎ Plugins / ▶︎ Search / ▶︎ Manage global search page in your Moodle installation to configure, index and enable the Solr Search Engine.

I am impressed with how quickly this has been used and appreciated by the students.

get_iplayer broken, version 3.00 available

For those of us who make use of the amazing get_iplayer program to obtain clips and other resources for classroom and other conveniences, it comes as a bit of a blow to find that in the past week or two, it has stopped working. Fortunately, there is a new version of the program available that with a little effort, gets the facility working again.

From the release notes:

The BBC removed all the XML-based data sources used by get_iplayer on 2017-04-26, breaking a lot of get_iplayer functionality. That functionality has been restored, but there are changes to be aware of – get_iplayer has not survived unscathed.

Phil Lewis and the team have (once again) done a fantastic job of quickly responding to changes in the way the BBC delivers its content. Many, many thanks to all the devs and hacks involved in this release.

Finally, my advice to users is to read the release notes carefully. You may also hit issues installing the new dependencies including Mojolicious and Perl as well as the cpan perl repository. Persevere, there is lots of useful advice out there. Finally, finally, the cache updates are much slower than before, although they are now only updated weekly.