Category Archives: PHP

Piping email to a script with cPanel

Note: Found a better solution to some of the annoyances below detailed in a later blog-post.

I recently found myself wanting to read email from an address, parse it and act upon it giving me an entrypoint to a very simple API from my Nokia e71’s native email facilities. My host uses cPanel for administration, and it helpfully has the ability to pipe mail destined for an email address to an arbitrary script file.

Given it took a little effort, a few different web resources and some annoyances I’ve documented the steps here.

Setup your processing script

Your processing script can be anything that can be run by the shell on your host. My application required a PHP end-point so these instructions are for a PHP script.

First off, create a new PHP file. Unlike PHP files you’ve created to be accessed from the web, we’ll need a shebang line at the top pointing to the executable to be used to process the script. On my host, that makes the very initial script the following:

#!/usr/bin/php -q
<?php
?>

Notice that -q? By default, invoking the PHP binary will render HTTP headers to the output stream. As we’re going to see shortly, any output generated by your processing script will be seen as a delivery failure by the mailer so we want to avoid shoving those headers into the output. The -q switch turns off those headers.

The mailer will be supplying the full text of the email (replete with headers) to STDIN – we can read in the email by reading from php://stdin using fopen and the standard PHP file access functions. Handily a chap called webignition already produced a function to process STDIN and yield us a string in a forum post.

However, that string’s pretty much just raw protocol – it’d be easier if we could access the email’s contents in a more structured way. Taking my cue from a Stack Overflow post I decided upon using the Plancake email parser class available in this Git repo. Using the class is straightforward – we construct a PlancakeEmailParser object passing in the string yielded by mailRead. This exposes the contents of the email through accessor methods on the object.

So our recipe so far:

  • Create a PHP script with a shebang line that points to the PHP binary and contains the -q switch to suppress HTTP header output
  • Read the contents of STDIN and pass it as a string to the constructor of PlancakeEmailParser giving us structured email data
  • <Do something with the structured email data>

That last bit’s on you.

Setup forwarding on your desired address

Let’s say that the address you want to use is api@example.com.

  • Log into cPanel and fire up the Forwarders page in the Mail group
  • Hit ‘Add Forwarder’ – you’ll be taken to the Add New Forwarder page
  • Enter the email account name in the ‘Address to forward’ box (and pick the correct domain if your cPanel’s set up to manage multiple)
  • Click the Advanced Options link to expand the form out
  • Pick the ‘Pipe to a program’ radio button and, in the text field next to it, enter the path (relative to your home directory) of the processing script created from the above steps
  • Click ‘Add forwarder’ – your new forwarder should be created

Now any email sent to the address you specified will be pumped to the script just created. We’re then free to do any processing we want.

Gotchas and things to check when it doesn’t work

There’re a lot of things that can go wrong with this – some of the ones I encountered are below.

All script output is considered to be an error
Anything your script outputs is considered by the mailer to be a problem and will cause the email to bounce back to the sender. While it might seem like you’re not outputting anything, the following non-exhaustive list are some examples:

  • Anything you output using PHP’s echo method (or equivalent)
  • Any ‘die’ statements that are called
  • Any HTTP headers (hence our using the -q switch)
  • Any syntax errors output by the PHP binary when it’s fired up
  • Any stray whitespace before the opening <?php tag or after the closing ?> tag

That last one caused some significant issues for me. By any whitespace I mean any whitespace – and not just in the PHP file that’s the target of your pipe but in anything it subsequently includes through require_once or similar.

One way to test this is to run your script on the command-line. If it outputs anything at all you’ll see it in the console – perhaps as stray whitespace, perhaps as more obvious parser errors.

Failing to read all of STDIN may be interpreted as an error too
This one I wasn’t expecting but failing to read all of STDIN may also be interpreted as an error state. The easiest way to handle this is to simply read the whole damned thing.

Debugging

Given that pretty much anything that your script does wrong will cause a bounce, and that the bounce may or may not contain useful information, it’s useful to split out the bulk of your processing logic into a separate file and have your endpoint responsible only for parsing STDIN into a structured email and calling your actual processing logic. Then you can have a test harness around your actual processing logic (perhaps a simple webpage that lets you paste in an email) and test it offline, without having to try to diagnose issues solely from having emails bounce.

It can also be handy to store all incoming emails in a file or database table for a short period of time to help you diagnose issues in conjunction with your test harness. If you get a bounce during development, you can dig up the email that caused the problem and fire it into your test harness.

An homage to Pages from Ceefax

I always loved Ceefax as a kid, and in particular (for reasons that probably indicate some manner of mental deficiency) Pages From Ceefax, the small non-navigable subset that was broadcast after-hours accompanied by a variety of big-band and easy listening music. And with a spare bank holiday weekend at my disposal and a girlfriend who is more accepting of my eccentricities than is necessarily prudent, I put together an HTML5 canvas + Javascript version that pulls in RSS feeds from the BBC news website and renders them in a hand-made sprite font:


Click image for project page – requires HTML5 canvas support

Two sprite fonts (one for character data and one for the graphics blocks from the header section) are combined with feeds from the BBC periodically downloaded by a cron job. Three feeds are in use in the above sample – the Top Stories, Sport and Business feeds. Each of these map to a single page (signified by the Pxxx number at the top of the view).

These are then split down into subpages so that both headline and byline are rendered together. For a given page there will be one or more subpages that contain the actual content – these are cycled between on a 15 second interval, and when all of the subpages of a given page have been shown the next page is slotted into place.

Graphing the Whisky Fringe with Visiblox

The Whisky Fringe Tasting Tracker generated a couple of hundred datapoints over the course of two days (after Neil gave it a crack on the Sunday, and a few of us had played with it on the Saturday). The data we have available is pretty simple, and associates opinions about whiskies against the times that those opinions were recorded.

This lets us look at a couple of things from the off:

  • What was the sampling rate like over the course of the day? Did it get quicker near the end when time was constrained, did we start off quickly and wind down early?
  • Did our sentiment towards the drams being tasted change over the course of the afternoon?

I charted some of the data using our company Scott Logic’s Visiblox charting component in a pretty rough-and-ready fashion. By way of a quick summary:

  • A PHP file generates a JSON object containing the various metrics
  • A custom Silverlight control hosting a Visiblox chart is coded to expose some ScriptableMember methods that directly manipulate the axes and data-series of the hosted chart, allowing Javascript on the HTML page hosting the Silverlight control to programmatically add data and configure axes
    • Note that this isn’t the way the Visiblox component’s intended to be used – I did it in this way as a fun experiment, to be written up in a subsequent post

For now here’s some static images:

Sampling rate

Sampling rate over the Saturday

First off we see the aggregate sampling rate. Each bar represents the total number of drams sampled (across all people using the system) in a given 15 minute time period. While the event started at 2pm, we didn’t actually get started on sampling until about 2.20pm (with the rest of the time spent actually getting into the venue and waiting for a few stragglers). Further, the tables started getting cleared at 5.45pm so there’s a corresponding lack of data there.

  • Around 4pm there’s a sharp drop-off as everyone camped out at their chosen ‘half-time orange’ stand – these are the rarer whiskies of which you can try one. We also seemed to use the subsequent wee while taking a breather.
  • Once the half-time orange period had ended things pick up substantially, possibly as people realised that time was no longer on our side.
  • There was another minor panic right near the end of the day as the time when stands packed up approached, though it’s clear that by about 5.30 everyone was pretty much done.

Likes and dislikes

Likes and dislikes over the Saturday

Here we’re charting the same data as above, but split out into those drams that were marked as ‘liked’ vs those marked as disliked. Dislikes are given a negative score to push them below the axis. It seems like there was a period about 45 minutes after the half-time orange where we were more critical of what we were trying, perhaps as a result of having recalibrated as a result of really liking the half-time whiskies. However, it’s a little difficult to tell (and in fairness, we don’t have enough data points with so few people using the system).

Sentiment

Sentiment over the Saturday

Sentiment’s a tricky thing to gauge. My initial attempt subtracts the number of whiskies marked as ‘disliked’ from the number marked as ‘liked’ in each 15 minute bucket, giving a net number of liked whiskies in the period. This is then divided by the total number of whiskies sampled during that 15 minute window, giving a normalised value between 0 and 1:

  • Values close to 0 suggest that more whiskies were disliked than liked in the period (that is to say that sentiment was negative overall). Zero values also occur when no tasting took place (such as the 4pm half-time orange lull).
  • Values close to 1 suggest that more whiskies were liked than disliked in the period (or that sentiment was positive overall).

With some of the buckets having so few samplings within them (for example, the 5.30ish bucket has only three samplings), the measure is very sensitive to noise but it’s an interesting diversion nonetheless.

Whisky Fringe Tasting Tracker

Each year the Royal Mile Whiskies Whisky Fringe showcases the wares of the whisky industry in the picturesque settings of Mansfield Traquair. For £20, you have four hours in which to wander round 30 or so stands sampling any of the ~250 whiskies and ~30 rums on offer.

Upon entry you get three things – a printed programme booklet, a biro and a tasting glass. I’ve tended in the past to try to take notes in the booklet, but it requires an unusual manoeuvre of trying to hold the tasting glass and booklet in one hand while scrawling (with less success the later in the day you are) with the other.

So on Thursday night I threw together a quick mobile-friendly website for myself and a few friends to use to track which whiskies we taste on the day and which we particularly like or dislike.

Screenshot of the Whisky Fringe Tasting Tracker UI

The interface is as simple as possible to allow one-handed operation on a phone – phone in one hand, sampling glass in the other. The list of whiskies is presented without much fanfare with a alphabetical jump-list at the top for navigation. Users can say that they either liked, disliked or simply tasted a given whisky which is recorded in a MySql database along with a timestamp such that we can piece together the events of the day after the fact.

It remains to be seen whether it’ll hold up for the day (having effectively had to teach myself PHP in the space of a night for this one, it’s probably not the most robust implementation I’ve ever been involved in)…

Graphing the “Keep F1 On The BBC Petition”

After the BBC announced that they were going to enter into a timeshare of sorts for the live TV rights to Formula 1 from 2012 onwards there’s been a fairly vocal crowd on Twitter, Facebook, the BBC message boards and petition sites making their views known. And those views are, understandably, almost universally negative. At time of writing, the ‘Keep F1 Coverage On The BBC For All Races From 2012‘ petition on PetitionBuzz has over 28,000 signatures and nearly 20,000 Facebook likes.

Since I’d intended this to be a technical blog, I figured that there wasn’t much better a way of starting off than hammering out some code over a single malt to track the progress of the petition as a chart:

F1 petition chart showing number of signatures over time, starting 03/08/2011 where over 28,000 signatures were already registered

A cron job running every 15 minutes first scrapes the petition site for the number of signatories, which is stored in a MySQL database against a timestamp. The above graph is then refreshed and saved to disk, such that it’s always in line with the latest data but not forever being generated on-the-fly. The graph is generated using the jpGraph library (the free version) with the actual incantations effectively copied-and-pasted from a sample on the website – I was going for quick and dirty with this one, and good Lord is it that.