Category Archives: Coding

nrconfig – Automatically generating custom instrumentation configuration files for NewRelic with .NET

New Relic’s a pretty good monitoring service and its support for .NET is pretty good – both integrated instrumentation and support for custom runtime metrics.

However, configuring custom instrumentation gets frustrating if you have to instrument more than a handful of methods:

  • You need to specify each method to be instrumented down to the parameter list (if you have method overloads) – this is something of a time vampire
  • There’s no way of saying ‘just instrument everything in this class for me’
  • There’s no link in Visual Studio between the instrumentation config file and your code – refactoring breaks your instrumentation silently

To help, I’ve written a build-time tool that post-processes .NET assemblies and generates custom instrumentation files for NewRelic very quickly. It’s available on NuGet as two packages:

It’s got a GitHub project page that contains documentation and use-cases.

It’s got two modes:

Coarse-grained filtering on the command-line – no code changes required

Here we’re assuming that we don’t want to (or can’t) change the source of the projects we want to appear in the configuration file. The tool has a mode that does a more freestyle reflection over assemblies and generates instrumentation entries for every method (or constructor or property) that matches some very coarse criteria – it’s not perfect, but it’s intended to quickly bootstrap the instrumentation process with a file that can be manually tailored.

The workflow’s very simple:

  • Run the tool, specifying which code elements should be instrumented
    • One or more of {all, constructors, properties, methods}
    • Append + or – to the end of each class of code element to specify inclusion of public (+) or non-public (-) members
    • Use them in combination – for example, methods+- properties+ constructors+ will generate a configuration file to instrument all methods (public or otherwise), and public constructors and public properties
  • Either use the configuration file straight-off, or adjust it manually

Marking up your code with attributes

Need more control? The most fine-grained way to use the tool is to mark up assemblies, classes or methods (with properties and constructors also supported) with an [Instrument] attribute.

  • If you mark up an assembly with the attribute, it’s assumed you want to instrument every method in every class in that assembly – heavy handed but brutally straightforward
  • If you mark up a class with the attribute, it’s assumed you want to instrument every method in that class, and every method in any nested classes
  • You can limit the scope of the attribute to constructors, methods or properties (or combinations thereof), and to public and non-public members in each case

The workflow is still pretty straightforward:

  • Mark up your assemblies, classes or methods with [Instrument] attributes as required
  • Run the tool as a post-build step – it reflects over the assemblies and generates an appropriate custom instrumentation file

Download and Documentation

Binaries are available on NuGet – first, the tool itself (which can then be referred to as ‘nrconfig.exe’:

PM> Install-Package NRConfig.Tool

Then, optionally, the library that contains the [Instrument] attribute that can mark up assemblies, classes and methods:

PM> Install-Package NRConfig.Library

You can also download the tool and optional the library directly.

Documentation is available on the nrconfig GitHub page. Source is available at https://github.com/Pablissimo/nrconfig.

Limitations

New Relic’s instrumenting profiler has some limitations with respect to generics and the tool can’t do much about them:

  • It doesn’t appear possible to specify that a particular method overload should be instrumented if that method overload takes an instance of a generic class as a parameter and the generic class is itself parameterised by more than one type parameter. For example, IEnumerable<string> is fine, but IDictionary<string, string> isn’t as it has two type parameters.
    • The tool handles this by generating an instrumentation entry for all methods with the same name in the same class when a complex (i.e. >= 2 generic type parameters) generic type is detected – it means some methods get instrumented when you don’t want them to, but appears at present to be the only way to get instrumentation to work
  • It’s not possible to instrument generic classes at all.

Disclaimer

This project has nothing to do with New Relic the company – it’s neither supported nor sanctioned by them, so if it breaks come crying to me and not the nice folks on their support desk.

CodeKicker.BBCode

We recently used CodeKicker.BBCode in a work project, and part of the licence agreement is a post on an employee’s personal blog – voila!

Except that’s not much fun, so let’s detail some of the modifications we made to it and released to GitHub.

First things first

There’re loads of BBCode parsers out there that suffer from a variety of problems, and the CodeKicker implementation seemed to be the closest to a ‘working out-the-box’ solution, but we still needed to fix a few bugs and behaviours we weren’t happy with.

[code] tags keep parsing their contents

Say you want to post a code snippet of a for-loop to a forum:

[code]
for (int i = 0; i < arr.Length; i++)
{
arr_copy[i] = arr[i];
}
[/code]

We’d expect the for-loop to render maybe in pre tags:

for (int i = 0; i < arr.Length; i++)
{
   arr_copy[i] = arr[i];
}

Instead the parser sees those two array accesses and interprets them as italic tags:

for (int i = 0; i < arr.Length; i++)
{
   arr_copy = arr[i]; }

So – we want [code] tags to cause the parser to stop processing until it finds a [/code] tag.

Can’t have spaces in tag attributes

If you’re quoting a member of a forum where everyone has a username, the following syntax would do the trick:

[quote=pablissimo]Hi there![/quote]

If however your username has a space in it, or your forum uses real names then you’re in trouble:

[quote=Paul O’Neill]Hi there![/quote]

In this instance, the CodeKicker parser sees the quote tag as having a single default attribute with value ‘Paul’, and disregards the ‘O’Neill’ part. Bummer.

Whitespace handling requires the user to know too many implementation details to format a post

When you’re mixing tags in a longer forum post, you might expect whitespace to be largely ignored if it’s just for the purposes of laying out your BBCode in an understandable fashion. For example, given the following:

[list]
[*]Here's the first item
[*]Here's the second item
[/list]
And here's the code snippet:
[code]
var frob = MakeFrob();
[/code]
It's that easy!

We might expect that:

  • The first list item isn’t preceded by a newline just because it’s on a different line to the opening [list] declaration
  • There’s no extra new-line after the last item in the list
  • There’s no extra new-line after the code tag

Out of the box this isn’t the case, and it can mean that getting a sensible-formatted output means you have to write some pretty horrible-looking BBCode.

The fixes

Stopping [code] tags from parsing BBCode contained within

First we added a property to the BBTag class that lets us specify ‘StopProcessing’ behaviour, defaulting to off – we can turn this on for [code] tags and it gives us the flexibility to introduce a [noparse] tag if we desire.

Now the parser’s stack-based, so to implement the change all we need to do is never match starting tags if the node on the top of the stack is marked ‘StopProcessing’. This’ll make us parse everything between the [code] and [/code] tags as plain text, ignoring anything that happens to look like an [i] or a [u] or anything else.

Allowing spaces within tag attributes

We added another new property to BBTag, ‘GreedyAttributeProcessing’, that again defaults to false. When true, we’ll have the parser assume that there’ll only ever be zero or one attributes against the tag, and that the entire text up until the closing square bracket of the opening tag is the value of the attribute. In our quote example above, we go from the old behaviour (underlined means ‘parsed as attribute value’):

[quote=Paul O’Neill]Hi there![/quote]

to something more sensible in an opt-in fashion:

[quote=Paul O’Neill]Hi there![/quote]

We extend the signature of the BBCodeParser.ParseAttributeValue method to accept a boolean parameter signifying that it should consume all other text in the opening tag as the value of the attribute – we’ll give it a default value of false. Then in ParseTagStart we pass in the value of the current tag’s GreedyAttributeProcessing value into ParseAttributeValue.

Finally, we modify ParseAttributeValue to change the set of characters that it considers ends an attribute value, from {<space>, opening square bracket, closing square bracket} to just {opening square bracket, closing square bracket}. Spaces are now up for grabs as attribute values!

Fixing whitespace handling

We’ll tackle this in two parts. First, we want to suppress the first newline that follows an opening tag – like when the [list] tag and its first [*] item appear on different lines, we probably don’t want an extra newline in there.

We’ll add an extra call to ParseWhitespace near the end of ParseTagStart – this will consume all whitespace characters before we return to the parsing loop so that they’re not interpreted as text nodes.

Next, we tackle the problem that some tags naturally demand a newline after them when we’re composing our message (again, [/list] but also [/code]) to keep it readable while editing, but that don’t need that newline to show in rendered output.

We’ll add another new property to BBTag, ‘SuppressFirstNewlineAfter’ that does what it says on the tin. We’ll set this to false by default, and true on tags that are ‘block-type’ – [list] and [code] being two clear examples.

We’ll then modify the parser again – in ParseTagEnd, we’ll see if the tag we’re closing has the SuppressFirstNewlineAfter attribute set and then consume all whitespace up to and including the first newline after the tag closes so that it doesn’t end up in the output. We’ll do that by adding a new method similar to ParseWhitespace called ParseLimitedWhitespace that takes a parameter for the maximum number of newlines to consume:

        static bool ParseLimitedWhitespace(string input, ref int pos, int maxNewlinesToConsume)
        {
            int end = pos;
            int consumedNewlines = 0;

            while (end < input.Length && consumedNewlines < maxNewlinesToConsume)
            {
                char thisChar = input[end];
                if (thisChar == '\r')
                {
                    end++;
                    consumedNewlines++;

                    if (end < input.Length && input[end] == '\n')
                    {
                        // Windows newline - just consume it
                        end++;
                    }
                }
                else if (thisChar == '\n')
                {
                    // Unix newline
                    end++;
                    consumedNewlines++;
                }
                else if (char.IsWhiteSpace(thisChar))
                {
                    // Consume the whitespace
                    end++;
                }
                else
                {
                    break;
                }
            }

            var found = pos != end;
            pos = end;
            return found;
        }

Perfect. One final thing to do – make sure that all newlines that escape the above treatment get converted into <br /> tags as appropriate. We can do this in TextNode’s ToHtml method by tacking a simple Replace(“\n”, “<br />”) onto the end.

Ninject + NewRelic + Windows Azure Worker Role = horrific crash?

We recently encountered a weird issue – using Ninject for our DI needs, in a Windows Azure Worker Role with NewRelic instrumentation. All good-practice stuff, using proven technology.

And it didn’t work.

The role wouldn’t start up, RDPing in to look at the Event Log showed horrific errors in the .NET runtime itself:

Application: WaWorkerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in 
the .NET Runtime at IP 000007FEF6076B81 (000007FEF6070000) with exit 
code 80131506.

Application popup: WaWorkerHost.exe - System Error : Exception 
Processing Message 0xc0000005 Parameters 0x000007FEFD08718C 
0x000007FEFD08718C 0x000007FEFD08718C 0x000007FEFD08718C

Faulting application name: WaHostBootstrapper.exe, version: 
6.0.6002.18488, time stamp: 0x4fcaabe9
Faulting module name: ntdll.dll, version: 6.1.7601.17696, 
time stamp: 0x4e8147f0
Exception code: 0xc0000374
Fault offset: 0x00000000000a0d6f
Faulting process id: 0x4a8
Faulting application start time: 0x01cd9fb18d252932
Faulting application path: E:\base\x64\WaHostBootstrapper.exe
Faulting module path: D:\Windows\SYSTEM32\ntdll.dll

Using the Ninject source code and a bit of detective work the likes of which House would be proud, we found that creating a new AppDomain on an Azure Worker Role with NewRelic installed causes a pretty horrible crash. In fact, the issue had nothing to do with Ninject at all – simply having a Worker Role whose code created a new AppDomain caused the crash.

The solution we found was to:

  1. Construct Ninject kernels using either a list of INinjectModule instances or a list of assemblies where the modules live – importantly do not use the Load method with a set of string filenames as Ninject will create a new AppDomain to host the assemblies while it reflects over them which causes the above crash
  2. In the INinjectSettings object passed to the kernel constructor, turn off LoadExtensions as this causes the same code path as above to be run through, irrespective of whether any extension assemblies exist on disk

Both solutions aim to avoid the creation of a new AppDomain and thus avoid the crashing behaviour.

While we hope it’s an issue NewRelic will resolve in due course, hopefully the above’ll keep you going in the meantime.

Piping email to a script with cPanel

Note: Found a better solution to some of the annoyances below detailed in a later blog-post.

I recently found myself wanting to read email from an address, parse it and act upon it giving me an entrypoint to a very simple API from my Nokia e71’s native email facilities. My host uses cPanel for administration, and it helpfully has the ability to pipe mail destined for an email address to an arbitrary script file.

Given it took a little effort, a few different web resources and some annoyances I’ve documented the steps here.

Setup your processing script

Your processing script can be anything that can be run by the shell on your host. My application required a PHP end-point so these instructions are for a PHP script.

First off, create a new PHP file. Unlike PHP files you’ve created to be accessed from the web, we’ll need a shebang line at the top pointing to the executable to be used to process the script. On my host, that makes the very initial script the following:

#!/usr/bin/php -q
<?php
?>

Notice that -q? By default, invoking the PHP binary will render HTTP headers to the output stream. As we’re going to see shortly, any output generated by your processing script will be seen as a delivery failure by the mailer so we want to avoid shoving those headers into the output. The -q switch turns off those headers.

The mailer will be supplying the full text of the email (replete with headers) to STDIN – we can read in the email by reading from php://stdin using fopen and the standard PHP file access functions. Handily a chap called webignition already produced a function to process STDIN and yield us a string in a forum post.

However, that string’s pretty much just raw protocol – it’d be easier if we could access the email’s contents in a more structured way. Taking my cue from a Stack Overflow post I decided upon using the Plancake email parser class available in this Git repo. Using the class is straightforward – we construct a PlancakeEmailParser object passing in the string yielded by mailRead. This exposes the contents of the email through accessor methods on the object.

So our recipe so far:

  • Create a PHP script with a shebang line that points to the PHP binary and contains the -q switch to suppress HTTP header output
  • Read the contents of STDIN and pass it as a string to the constructor of PlancakeEmailParser giving us structured email data
  • <Do something with the structured email data>

That last bit’s on you.

Setup forwarding on your desired address

Let’s say that the address you want to use is api@example.com.

  • Log into cPanel and fire up the Forwarders page in the Mail group
  • Hit ‘Add Forwarder’ – you’ll be taken to the Add New Forwarder page
  • Enter the email account name in the ‘Address to forward’ box (and pick the correct domain if your cPanel’s set up to manage multiple)
  • Click the Advanced Options link to expand the form out
  • Pick the ‘Pipe to a program’ radio button and, in the text field next to it, enter the path (relative to your home directory) of the processing script created from the above steps
  • Click ‘Add forwarder’ – your new forwarder should be created

Now any email sent to the address you specified will be pumped to the script just created. We’re then free to do any processing we want.

Gotchas and things to check when it doesn’t work

There’re a lot of things that can go wrong with this – some of the ones I encountered are below.

All script output is considered to be an error
Anything your script outputs is considered by the mailer to be a problem and will cause the email to bounce back to the sender. While it might seem like you’re not outputting anything, the following non-exhaustive list are some examples:

  • Anything you output using PHP’s echo method (or equivalent)
  • Any ‘die’ statements that are called
  • Any HTTP headers (hence our using the -q switch)
  • Any syntax errors output by the PHP binary when it’s fired up
  • Any stray whitespace before the opening <?php tag or after the closing ?> tag

That last one caused some significant issues for me. By any whitespace I mean any whitespace – and not just in the PHP file that’s the target of your pipe but in anything it subsequently includes through require_once or similar.

One way to test this is to run your script on the command-line. If it outputs anything at all you’ll see it in the console – perhaps as stray whitespace, perhaps as more obvious parser errors.

Failing to read all of STDIN may be interpreted as an error too
This one I wasn’t expecting but failing to read all of STDIN may also be interpreted as an error state. The easiest way to handle this is to simply read the whole damned thing.

Debugging

Given that pretty much anything that your script does wrong will cause a bounce, and that the bounce may or may not contain useful information, it’s useful to split out the bulk of your processing logic into a separate file and have your endpoint responsible only for parsing STDIN into a structured email and calling your actual processing logic. Then you can have a test harness around your actual processing logic (perhaps a simple webpage that lets you paste in an email) and test it offline, without having to try to diagnose issues solely from having emails bounce.

It can also be handy to store all incoming emails in a file or database table for a short period of time to help you diagnose issues in conjunction with your test harness. If you get a bounce during development, you can dig up the email that caused the problem and fire it into your test harness.

Improving the C# Ants AI Challenge starter kit for debugging support

The C# starter kit for the current AI Challenge works well out-of-the-box, but doesn’t easily let you debug your bot live. There’re a number of ways of supporting live debugging, so I chose one of the more interesting.

The aim is to host the bot in a Visual Studio debugging session, then write a proxy that the Python game script runs which relays commands and responses between the Python script (which expects IO through stdin and stdout) and the bot being debugged.

To start this, we need to do a little refactoring to the starter kit classes. Our aim is to decouple the starter kit from talking directly to Console.In and Console.Out. This only happens in a couple of places so is reasonably straightforward, and the methods called are few.

So, let’s define an interface representing a bot’s connection to the game:

public interface IGameDataPipe
{
   string ReadLine();
   void WriteLine(string s);
}

We’ll also implement a console-based version of this:

public class ConsoleGameDataPipe : IGameDataPipe
{
   public string ReadLine()
   {
      return Console.In.ReadLine();
   }    

   public void WriteLine(string s)
   {
      Console.Out.WriteLine(s);
   }
}

We then want to modify the Ants class to use our interface instead of talking directly to the Console class. First off, we add a member variable of type IGameDataPipe and default it to our console version. Then we find everywhere that calls Console.* and replace it with an equivalent call on the IGameDataPipe member.

There’s one other place in the starter kit that talks to the console directly – the base class for our bot implementation in Bot.cs. Here, IssueOrder is directly calling Console.WriteLine. It’d be great if we could avoid having to expose the Bot class to the concept of how to present its data to the game layer, so I’d like to avoid pushing an IGameDataPipe into the Bot. Instead we’ll expose this through a delegate and event – when the bot wants to issue an order to mutate the game state it raises the event with appropriate arguments and the Ants class figures out what to do with it.

That takes our definition of Bot to looking something like this:

public abstract class Bot {    
   public delegate void OrderIssueHandler(Location loc, Direction direction);
   public event OrderIssueHandler OrderIssued = delegate { };
   public abstract void DoTurn(IGameState state);

   protected virtual void IssueOrder(Location loc, Direction direction)
   {
      this.OrderIssued(loc, direction);
   }
}

We then modify the Ants class to hook the event upon entering PlayGame, with the handler delegating to the IGameDataPipe.

So after all that we’ve achieved nothing functionally different but we have managed to decouple the bot implementation from the actual mechanism by which it talks to the Python game script.

This lets us replace our console in-out implementation with something else. We could use IPC here, WCF remoting, anything. I’m going with TCP for this example. Our server and client implementations can be very simple because the way our IGameDataPipe is consumed is also very simple. The original Console-based client worked just fine using the blocking IO provided by the Console class, so we can replicate the same. First things first – let’s modify Ants.cs with a new constructor that lets us override the IGameDataPipe to be used for comms with any other implementation.

// Default to the console, allow caller to override as required.
IGameDataPipe dataPipe = new ConsoleGameDataPipe();

public Ants()
{
}

public Ants(IGameDataPipe pipe)
   : this()
{
   this.dataPipe = pipe;
}

Now on the server side (that is, the side that’s hosting the bot) we simply wrap a TcpClient with a StreamReader and StreamWriter and expose ReadLine and WriteLine methods delegating their work to the two Stream* classes.

public class TcpIPGameDataPipe : IGameDataPipe
{
   StreamReader _reader;
   StreamWriter _writer;
   TcpClient _client;    

   public TcpIPGameDataPipe(TcpClient client)
   {
      _client = client;
      NetworkStream stream = _client.GetStream();
      _reader = new StreamReader(stream);
      _writer = new StreamWriter(stream);
   }

   public string ReadLine()
   {
      return _reader.ReadLine();
   }

   public void WriteLine(string s)
   {
      _writer.WriteLine(s);
      _writer.Flush();
   }
}

On the client side (that is, the shim that the Python script talks to) we need to relay stdin over a TCP connection, and relay incoming data from the TCP connection to stdout. This is marginally more involved than the server implementation as we need to thread these operations since they’re being initiated from one of the Python script on one side and the TCP connection on the other. Still, the code’s straightforward:

public class ConsoleTcpIPForwarder
{
   TcpClient _client;
   StreamWriter _writer;
   StreamReader _reader;    

   Thread _readThread;
   Thread _writeThread;

   public ConsoleTcpIPForwarder(int port)
   {
      _client = new TcpClient(new IPEndPoint(IPAddress.Loopback, 0));
      _client.Connect(new IPEndPoint(IPAddress.Loopback, port));

      NetworkStream stream = _client.GetStream();
      _reader = new StreamReader(stream);
      _writer = new StreamWriter(stream);
   }

   public void Start()
   {
      _readThread = new Thread(ReadForwarder);
      _writeThread = new Thread(WriteForwarder);

      _readThread.Name = string.Format(“{0} read thread”this.GetType().Name);
      _writeThread.Name = string.Format(“{0} write thread”this.GetType().Name);

      _readThread.Start();
      _writeThread.Start();
   }

   private void ReadForwarder()
   {
      while (true)
      {
         _writer.WriteLine(Console.ReadLine());
         _writer.Flush();
      }
   }

   private void WriteForwarder()
   {
      while (true)
      {
         Console.WriteLine(_reader.ReadLine());
      }
   }
}

Now we’re almost home. Our shim implementation simply needs to kick off a ConsoleTcpIPForwarder and leave it running, which will in turn forward all console input to the network and network traffic back to console. Our server implementation that hosts the bot just needs to listen for incoming TCP connections made by the shim and when one’s initiated kick off a new Ants game using the TcpIPGameDataPipe. We’ll keep this in a busy-loop that runs a game then waits for more connections to avoid having to keep kicking off the Visual Studio side of things.

The only final comment here is that we want to keep out TcpIPGameDataPipe in a separate assembly to avoid breaking compilation of our main bot assembly when uploaded to the server (as it’s unlikely that it has the required System.Net* assemblies available for compilation against).

I’ve combined both the forwarding client and the game-host server in the same executable for straightforwardness, deciding whether to spin up the server or forward from the client based on whether any command-line params are supplied (none = client, anything, say ‘-server’ = server).

To run, we start the shim process in Visual Studio (with a command-line parameter of ‘-server’ to force it to go into server mode) and then run the Python game script pointing it at our shim exe instead of the actual bot exe. We can now put breakpoints into our bot code and have them fire just as we want.

The modified package is available here.

Improvements to Telerik’s input controls

Telerik’s RadControls for ASP.NET are reasonably good from an ‘out of the box pretty’ point of view and expose a fair bit of functionality with minimal configuration (hence the ‘RAD’ part of the control name, one imagines). However, such quick-to-throw-together functionality comes at a cost, as the controls can render down to some quite clumsy HTML.

For example a RadTextBox called ‘RadTextBox1’ with a prompt message within it renders to three input tags and a span tag:

<span id="RadTextBox1_wrapper" style="white-space: nowrap;">
  <input value="type here..." size="20" id="RadTextBox1_text" name="RadTextBox1_text" style="width: 290px;" type="text">
  <input id="RadTextBox1" name="RadTextBox1" title="" style="visibility: hidden; margin: -18px 0pt 0pt -1px; width: 1px; height: 1px; overflow: hidden; border: 0pt none; padding: 0pt;" value="" type="text">
  <input autocomplete="off" value="{&quot;enabled&quot;:true,&quot;emptyMessage&quot;:&quot;type here...&quot;}" id="RadTextBox1_ClientState" name="RadTextBox1_ClientState" type="hidden">
</span>

The element that actually shows on-screen isn’t the one with ID ‘RadTextBox1’ which is annoying when trying to tie together jQuery and Telerik controls. For example, to attach a jQuery Autocomplete to the above Telerik RadTextBox you’d have to either do it by class (fine, unless you’re using this in a user control in which case you’d have to class each one differently just to get different autocompletion lists working) or assume that Telerik stick with their current convention of appending ‘_text’ to the element ID that actually shows.

So it’s fairly good news for those faced with combining Telerik controls and jQuery (or anything else javascripty) that they’ve introduced ‘single input rendering mode‘ that turns the above into something marginally clearer:

<span id="RadTextBox1_wrapper" style="width: 290px;">
   <span id="RadTextBox1_display" style="display: inline; color: rgb(138, 138, 138); font-family: &quot;segoe ui&quot;,arial,sans-serif; font-size: 12px; line-height: 17px;">type here</span>
   <input style="color: transparent;" id="RadTextBox1" name="RadTextBox1" size="20" type="text">
   <input autocomplete="off" value="{&quot;enabled&quot;:true,&quot;emptyMessage&quot;:&quot;type here&quot;}" id="RadTextBox1_ClientState" name="RadTextBox1_ClientState" type="hidden">
</span>

It’s still generating a bunch of tags for something fairly straightforward, but at least now the visible input control is the one with the ID you assign in the page making interactions with jQuery a little easier.

AI Challenge 2011

I’ve not taken part in an AI Challenge before but with the next few weekends looking pretty quiet I decided to sign up. The latest challenge, run as a collaboration between Google and the University of Waterloo Computer Science Club, simulates a competitive environment in which two or more ant colonies fight for survival on a selection of maps.

The contest is distinctive for the broad range of languages supported. You code up your entry, zip it and upload it where it is compiled (where necessary) on the server and put into a queue to fight against other players’ creations. There are starter packages provided so that you don’t have to parse all of the incoming data and wrap up the output.

You control an ant colony located somewhere on a map. By commanding your ants to move to and collect food you spawn new ants from your anthills. Survival is key, but a scoring system also encourages attacking behaviour – get close enough to another ant and you’ll attack it, and if you manage to land on an undefended ant-hill that hill is lost and no longer produces ants for the opposing player.

Your code has a limited amount of time to perform its processing per time-step, and if it exceeds that time then your bot is eliminated. Equally if your bot crashes during execution then you’re also out. Winning or losing changes your rank based upon the Trueskill algorithm and your bot will keep being queued for more battles in a round-robin fashion for a number of days after you upload it.

I’ve decided to try a diffusion approach which I suspect many others will be trying too. The idea is reasonably simple – goal items in the world exude a digital pheremone that makes them attractive to your ants, and this pheremone is diffused throughout the map over time by simple averaging in adjacent cells. For example, a food item might have a score of 1000 scent units in time step 0; in time step 1 its neighbours will have attributed themselves some proportion of that scent, and in time step 2 their neighbours will also be imbued. If you consider the scent of food to be a surface then we iteratively smooth out sharp peaks caused by goal items throughout the rest of the surface.

That means that we can find food or unexplored squares (if we associate a suitable attraction score to cells we’ve not yet visited) by simple hill-walking per-ant. When an ant is deciding where to move it examines the squares to the north, west, east and south of it and picks the direction with the highest scent value. It also means that we get pathfinding for free by clamping the ability of wall-cells to propagate scent to zero – any other cell adjacent to a wall cell will have a higher attractiveness, so the ant will not consider the move.

There are substantial kinks to work out in my implementation but I’ve only just started. For now, my bot (not yet uploaded) seeks only to explore the map without bothering to look for food (though there is enough food that it’ll stumble upon some accidentally fairly often). This video shows the pheremone trails over time; the green on the left is the ‘unexplored’ pheremone, while the red on the right is the ‘food’ pheremone. Just now the ants are configured to consume the unexplored pheremone when they are within a cell such that any ants following behind will seek different routes.

The ants are starting from near the top-left of the map. The green pheremone quickly ramps up across the map even in regions the ants cannot yet see because we do not yet know where the walls are in those sections – ants have a limited field of view. Once we start exploring these regions the wall cells start to clamp correctly and the map topology becomes a lot clearer.

On the red side the food pheremone pulses as new food items are added randomly. When an item is consumed its red signal falls off over a few time units.

The contest closes for new entries in December on the 18th, after which the tournament-proper starts – I’ve enough time to hopefully get something based on this diffusion approach uploaded by then!

An homage to Pages from Ceefax

I always loved Ceefax as a kid, and in particular (for reasons that probably indicate some manner of mental deficiency) Pages From Ceefax, the small non-navigable subset that was broadcast after-hours accompanied by a variety of big-band and easy listening music. And with a spare bank holiday weekend at my disposal and a girlfriend who is more accepting of my eccentricities than is necessarily prudent, I put together an HTML5 canvas + Javascript version that pulls in RSS feeds from the BBC news website and renders them in a hand-made sprite font:


Click image for project page – requires HTML5 canvas support

Two sprite fonts (one for character data and one for the graphics blocks from the header section) are combined with feeds from the BBC periodically downloaded by a cron job. Three feeds are in use in the above sample – the Top Stories, Sport and Business feeds. Each of these map to a single page (signified by the Pxxx number at the top of the view).

These are then split down into subpages so that both headline and byline are rendered together. For a given page there will be one or more subpages that contain the actual content – these are cycled between on a 15 second interval, and when all of the subpages of a given page have been shown the next page is slotted into place.

Graphing the Whisky Fringe with Visiblox

The Whisky Fringe Tasting Tracker generated a couple of hundred datapoints over the course of two days (after Neil gave it a crack on the Sunday, and a few of us had played with it on the Saturday). The data we have available is pretty simple, and associates opinions about whiskies against the times that those opinions were recorded.

This lets us look at a couple of things from the off:

  • What was the sampling rate like over the course of the day? Did it get quicker near the end when time was constrained, did we start off quickly and wind down early?
  • Did our sentiment towards the drams being tasted change over the course of the afternoon?

I charted some of the data using our company Scott Logic’s Visiblox charting component in a pretty rough-and-ready fashion. By way of a quick summary:

  • A PHP file generates a JSON object containing the various metrics
  • A custom Silverlight control hosting a Visiblox chart is coded to expose some ScriptableMember methods that directly manipulate the axes and data-series of the hosted chart, allowing Javascript on the HTML page hosting the Silverlight control to programmatically add data and configure axes
    • Note that this isn’t the way the Visiblox component’s intended to be used – I did it in this way as a fun experiment, to be written up in a subsequent post

For now here’s some static images:

Sampling rate

Sampling rate over the Saturday

First off we see the aggregate sampling rate. Each bar represents the total number of drams sampled (across all people using the system) in a given 15 minute time period. While the event started at 2pm, we didn’t actually get started on sampling until about 2.20pm (with the rest of the time spent actually getting into the venue and waiting for a few stragglers). Further, the tables started getting cleared at 5.45pm so there’s a corresponding lack of data there.

  • Around 4pm there’s a sharp drop-off as everyone camped out at their chosen ‘half-time orange’ stand – these are the rarer whiskies of which you can try one. We also seemed to use the subsequent wee while taking a breather.
  • Once the half-time orange period had ended things pick up substantially, possibly as people realised that time was no longer on our side.
  • There was another minor panic right near the end of the day as the time when stands packed up approached, though it’s clear that by about 5.30 everyone was pretty much done.

Likes and dislikes

Likes and dislikes over the Saturday

Here we’re charting the same data as above, but split out into those drams that were marked as ‘liked’ vs those marked as disliked. Dislikes are given a negative score to push them below the axis. It seems like there was a period about 45 minutes after the half-time orange where we were more critical of what we were trying, perhaps as a result of having recalibrated as a result of really liking the half-time whiskies. However, it’s a little difficult to tell (and in fairness, we don’t have enough data points with so few people using the system).

Sentiment

Sentiment over the Saturday

Sentiment’s a tricky thing to gauge. My initial attempt subtracts the number of whiskies marked as ‘disliked’ from the number marked as ‘liked’ in each 15 minute bucket, giving a net number of liked whiskies in the period. This is then divided by the total number of whiskies sampled during that 15 minute window, giving a normalised value between 0 and 1:

  • Values close to 0 suggest that more whiskies were disliked than liked in the period (that is to say that sentiment was negative overall). Zero values also occur when no tasting took place (such as the 4pm half-time orange lull).
  • Values close to 1 suggest that more whiskies were liked than disliked in the period (or that sentiment was positive overall).

With some of the buckets having so few samplings within them (for example, the 5.30ish bucket has only three samplings), the measure is very sensitive to noise but it’s an interesting diversion nonetheless.

Whisky Fringe Tasting Tracker

Each year the Royal Mile Whiskies Whisky Fringe showcases the wares of the whisky industry in the picturesque settings of Mansfield Traquair. For £20, you have four hours in which to wander round 30 or so stands sampling any of the ~250 whiskies and ~30 rums on offer.

Upon entry you get three things – a printed programme booklet, a biro and a tasting glass. I’ve tended in the past to try to take notes in the booklet, but it requires an unusual manoeuvre of trying to hold the tasting glass and booklet in one hand while scrawling (with less success the later in the day you are) with the other.

So on Thursday night I threw together a quick mobile-friendly website for myself and a few friends to use to track which whiskies we taste on the day and which we particularly like or dislike.

Screenshot of the Whisky Fringe Tasting Tracker UI

The interface is as simple as possible to allow one-handed operation on a phone – phone in one hand, sampling glass in the other. The list of whiskies is presented without much fanfare with a alphabetical jump-list at the top for navigation. Users can say that they either liked, disliked or simply tasted a given whisky which is recorded in a MySql database along with a timestamp such that we can piece together the events of the day after the fact.

It remains to be seen whether it’ll hold up for the day (having effectively had to teach myself PHP in the space of a night for this one, it’s probably not the most robust implementation I’ve ever been involved in)…