Links
These are a few of my favourite links.

Flavours
There's more than one way to view this weblog; try these flavours on for size.

Subscribe
Subscribe(RSS) to a syndicated feed of my weblog, brought to you by the wonders of RSS.
or by Atom..

Some nifty command line stuff
So after reading in the Halloween countdown pages at the Redecorating Middle-Earth in Early Lovecraft blog that Mary Shelley's book 'The Last Man' was available as a free audiobook, I thought I'd download it and listen to it as a series of podcasts (thanks to podcastamatic) but when I got to the page, the low quality mp3's were mixed in with the high quality ones. I've addressed the multiple mp3 download issue before, but this time I thought I could come up with something that got what I wanted. One last night command line sesson, and lo;

for i in `~/perl/mp3grep.pl thelastman.html | grep -v _64k` ; do filename=`echo $i | cut -d '/' -f 6 `;curl -C - -L -o $filename $i; done

Posted at: 21:53:34 19 Oct 2008 [/code] permanent link

GrabRSSEnclosures3.pl
As I blogged before, I've been having problems with RSS feeds where the enclosure line is spread of multiple lines in the file. The BBC has recently moved to doing this for all thier podcasts, which neatly broke every download I have from them. So tonight I rewrote the script - properly this time, as it now treats the RSS as a stream it needs to cherry pick the url's out of..
So, without further ado;

useage: GrabRSSEnclosures3.pl <RSS URL>

Takes a RSS feed url, downloads it and searches for enclosure tags, extracts the url, and attempts to download each. If it does sucessfully download the file, it adds it to a MLDB hash (saved with the same name as the feed's name), so that it doesn't try to download it again.

Written by john@kript.net. Licensed under the GPL.

Check for updates at http://cgi.kript.net/blosxom.cgi/code/

Download the latest version; http://www.kript.net/perl/GrabRSSEnclosures3.pl

Posted at: 23:47:09 9 Mar 2007 [/code] permanent link

TrackDownload.pl
I've blogged about some of the perl scripts I use to pull down mp3/aac files from RSS feeds in the 'unmetered' time on my ISP, before. I've run into some problems with GrabRSSEnclosures.pl, where poorly done (cough *windows* cough) feeds contain multiple newlines, the links are spread accross multiple lines etc. I spent some time fiddling with regular expressions to try and match (and then fix) this, then gave up. I now preprocess the file, and send the list of links to this script which downloads them, and tracks what has been downloaded in the same way as GrabRSSEnclosures.pl did, so you can automate as much of this as possible. For example, to process the Chinese Pod's Saturday show XML, I do the following;

#!/bin/bash
#
cd /mnt/mp3/mp3/Podcasts/ChinesePodSaturdayShow/
wget -O satshow.xml http://www.chinesepod.com/podcast.php?cat=13
for i in `~/perl/mp3grep.pl satshow.xml`
do
~/perl/TrackDownload.pl ChinesePodSaturdayShow $i
done

Some information on the script, courtesy of pod2html;

useage: TrackDownload.pl <statefile> <URL>

Takes a url, and attempts to download each. If it does sucessfully download the file, it adds it to a MLDB hash (saved with the same name as the statefile name), so that it doesn't try to download it again.

Written by john@kript.net. Licensed under the GPL.

Check for updates at http://cgi.kript.net/blosxom.cgi/code/

Download the latest version; http://www.kript.net/perl/TrackDownload.pl

Posted at: 17:58:30 27 Oct 2006 [/code] permanent link

My DVD Collection
Now, I'm sure you check my DVD list religiously, to see anything new I might have bought that you might like to borrow, and I know, like me, you're thinking "there's got to be a better way! Why can't he make the list available as RSS, so I'm notified by my reader every time he adds a new one?"
Well, now you can with DVDList.xml

Posted at: 11:28:15 17 Sep 2006 [/code] permanent link

Yay! Podcastamatic does video!
My RSS feed creator of choice, Podcastamatic has a Beta version out that handles video files. Now I can have a feed of my locally held ReFederator stream of cartoons ready formatted for the iPod.
and there was much rejoicing..

Posted at: 21:28:12 27 Apr 2006 [/code] permanent link

Convert::Addressbook::Mozilla2Blackberry
A follow-up to Mozilla::Mork

Convert::Addressbook::Mozilla2Blackberry - Perl extension for converting a CSV comma delimited addressbook from Mozilla to Blackbery csv import format

               use Mozilla::Mork;
               use Convert::Addressbook::Mozilla2Blackberry;

               #get the file from the command line or quit with an error
               $file = $ARGV[0];
               unless ($file) { die "Useage: $0 \n"; }

               #access the address book and setup the memory structure
               my $MorkDetails = Mozilla::Mork->new($file);

               #get a reference to an array of hash's
               my $results = $MorkDetails->ReturnReferenceStructure();
               #create an instance of the converter code
               my $converter = Convert::Addressbook::Mozilla2Blackberry->new();
               #print the Blackberry File headers
               $converter->PrintBlackberryHeaders();

               #process those results
               # for each line in the database
               # each line in the database corresponds to an address book record
               # pass this reference to the StreamConvert routine which will
               # convert and print it

               for my $record_array ( @{$results} )
               {
                       $converter->StreamConvert($record_array);
               }

Posted at: 13:32:32 10 Jan 2006 [/code] permanent link

Podcastamatic
I've been using Podcastamatic for a few weeks to organise all the podcasts I listen to - it generates RSS feeds for mp3 and m4a files, as well as a summary web page, including iTunes specific feeds, if thats your thing. It (at the moment) can't select the encoding method, so if your podcast (like the BBC, grumble) use windows-1252 instead of UTF-8 for the charachter set, then you have to ask him nicely for one that generates the appropriate XML tags. I did do some work on a in-line converter before he sent me the fix, and I keep meaning to finish it and send it in. Encoding is a nightmare..

Posted at: 21:23:44 9 Jan 2006 [/code] permanent link

The Perl Advent Calender is back!
I was a great fan of the original Perl Advent Calender, and I was disappointed that he's not choosing to pursue it this year (I was so fond of it I would have been happy to stump up a few dollars, too!). I was even considering rolling my own, and then some one decided to pick up the challenge. Hurrah!

Posted at: 09:10:52 7 Dec 2005 [/code] permanent link

My First CPAN Module!
Mozilla::Mork

Mozilla::Mork - Perl extension for reading Mork hash database file such as are used in the Mozilla Address Book and History files.

        use Mozilla::Mork;
        $file = $ARGV[0];
        unless ($file) { die "Useage: $0 <filename>\n"; }
        #get a reference to an array of hash's
        my $MorkDetails = Mozilla::Mork->new($file);
        my $results = $MorkDetails->ReturnReferenceStructure();
        #process those results
        # for each line in the database
        my %array = %{ $results->[0] };
        my @field_names = sort(keys(%array));
        #my @field_names = $MorkDetails->ListHeaders();
        map { print "Field Names: $_\n"; } @field_names;
        print "\ndone!\n";

Posted at: 14:19:31 11 Nov 2005 [/code] permanent link

Geek Tool
I've been using Konfabulator on both my iBook and laptop at work for a while. Its not the greatest user of memory, and I've finally ditched it on the mac for the Tiger compatible version of GeekTool, a utility that allows you to have a tail of log files (or any files), display images, or most powerfully, have the output of a unix command or shell/perl script, all display overlaid on whatever background image you have selected.
Its best explained by an image, so you can see my current desktop.

I've made use of several of my past scripts (check the code section of the blog), and taken a couple from assorted Macosxhints.com articles on GeekTool, such as;

geektool-iTunes.sh

geektool-cpuload.pl

Posted at: 12:03:52 8 Oct 2005 [/code] permanent link

Travelsync
I certainly didn't develop this, but I did do some help testing and point Ian at backstage.bbc.co.uk. Anyway, if you want traffic info on your phone, and your phone can run java and has GPRS, you might like to take a look at the low bandwidth/simple travelsync..
If you do/don't like it, please tell Ian..

Posted at: 14:06:48 2 Oct 2005 [/code] permanent link

Software Release: Pktfilter Dynamic Connection Manager
I've run into the need to manage the pktfilter open source windows firewall with dynamic connections a few times, and a post on the pktfilter-users mailing list on friday prompted me to put a release of my management software and implementation together;
Using The pktfilter software with dynamic interfaces.
It takes Pktfilter, an open source, free windows firewall to that little bit more useability. You do have to know how to write firewall rules, though. Having said that, if you're looking into pktfilter, then frankly, it will be enough for you to read the PDF that comes with the release.

Posted at: 19:20:51 3 Jul 2005 [/code] permanent link

Podcasting Tools
I've been listening to a fair number of podcasts recently - mostly from the BBC trial, but also from a few other places ( MacCast, IT Conversations). I use shrook as my RSS reader, but I have to download each mp3 individually, and I don't like the way Shrook stores then in the ~/Library folder. Also, being mp3's mean I can't take advantage of the AAC format's bookmarking feature, where you can pause the track, go and listen to something else and come back where you left off - very useful for the longer features such as "In Our Time" (45 mins). So, I've been converting each mp3 to AAC via iTunes. The problem is that this then stores the converted file locally.
What I really wanted, was a way to download each track from a RSS feed, and then convert them to AAC, ready to import into iTunes as just a pointer to the network store.
So, I decided to write one.
The only command line based AAC encoder I could find was the excellent FFmpeg (I found that most distributions only packaged the decoder. I had to download the FAAC encoder from its home here.) some experimenting with invoking this from perl gave me the converter I wanted. Unfortunately, it seems the iPod has a bug when playing non-iTunes encoded AAC mono files. It took me a week of googling to find this, cursing Apple for (as I thought) introducing proprietary incompatibilities into their products (even though they'd created and documented the standard). I finally found other people with the problem in the forums at ipodlounge - fortunately, they'd found that converting the AAC file from mono to stereo would allow it to play, and sure enough, when I made the appropriate changes to the script, the resulting files played just fine. The threads for reference;
Guide 6. Convert to Stereo to avoid 4G & Mini Lockup on NEW or EXISTING Mono m4b file
.m4b files cause iPod to lock up after I listen to them.
and this was the thread that put me on the right track.

It seems that AAC files dont have the same rich metadata that can exist in MP3's, so I wasn't able to transfer all the information that well-put together mp3's (such as the BBC's) contain - the script keeps as much information as it can (title, author, copyright and description), but I ended up putting together an Automator plugin to add the extra tags I use to manage the playlists in iTunes (genre, and year, usually). Talking of which, I have to say that Automator hasn't proved very useful for me, despite wanting to script several processes. I've been able to use it for very simple GUI scripting stuff, though, and it means I can put off learning Applescript for a while longer, so I guess I can't complain too much.

Once I'd had the conversion script working, I then put together a script to parse a RSS feed. So far, only RSS 2.0 has support for included files (enclosures), so I didn't worry about coding something for multiple variants of RSS and Atom. Unfortunately, as Ben Hammersley wrote in his excellent "Developing Feeds with RSS and Atom" (which is the source I got all my understanding of all of this, plus some skeleton for the start of the script, from), Perl support for RSS is extremely variable. It can generate superb RSS, but parsing it is another matter. Most of the RSS parsers in perl don't deal well at all with RSS 2.0 feeds, and I had to resort to using the plain XML parser instead, which worked well, but took me a day of playing with different modules to arrive at the solution. Once I had the feed, I had to extract the enclosure tag and its contents manually, and write a simple database that kept track of feeds it had seen and downloaded - not too complex a task, but it took me the best part of a day, with assorted blind alleys.
So, I present here for your use and entertainment;
mp3ToAAC.pl
GrabRSSEnclosures.pl

I'm going to create an individual directory for each feed and run the parser into it via cron. Once a day is needed for some, but twice a week should be more than enough for the weekly based ones.
Do let me know if you find them useful!

Posted at: 21:27:03 12 Jun 2005 [/code] permanent link

m3ugrep.pl
I've modified my mp3grep.pl script so that it extracts the .m3u files from a webpage. Some websites use these to point to individual sites, so you need to grab these before extracting the mp3 link with mp3grep.pl.
You can download the script, should you fancy, at; http://www.kript.net/perl/m3ugrep.pl

Posted at: 08:39:18 5 Jun 2005 [/code] permanent link

mp3grep.pl version 2
I ran into a couple of problems using this during the week, so I fixed the bugs and even provided end-user documentation (type perldoc ./mp3grep.pl)!
Download the latest version..

Posted at: 10:09:59 20 Mar 2005 [/code] permanent link

HTML link parser
Boing Boing had a recent link to a hundred covers of The Animals "House of The Rising Sun". The maintainer has put it up as a CGI script, presumably to prevent people wget'ing the entire collection. Not to be outdone (I really like "House Of The Rising Sun"), I whipped up a quick Perl script to parse the HTML for links, which I then used in a bash command line to grab all the files with wget;
for i in `~/perl/mp3grep.pl index-house`; do; wget $i; done;
You can download the script, if you feel so inclined; mp3grep.pl. Its only version 1 at the moment. I may add url checking for mp3's, and a progress meter in the future, but this only took me 15 mins, thanks to CPAN.
Update Excuse me, thats two hundred and fifty covers.. Oh my..

Posted at: 12:42:20 16 Jan 2005 [/code] permanent link

Announcing: Weather 0.2
Inspired by a section "Spidering Hacks", I created a small Perl script to get the weather for Cambridge from The Weather Underground.
This version just outputs basic text about the weather, and only about Cambridge, although it would be a trivial hack to add a command line argument to accept the city to check.
Download Weather 0.2.
If you have any suggestions, mail me on the address in the sidebar..

Posted at: 19:25:35 10 Apr 2004 [/code] permanent link