Category Archives: NewRelic

nrconfig – Automatically generating custom instrumentation configuration files for NewRelic with .NET

New Relic’s a pretty good monitoring service and its support for .NET is pretty good – both integrated instrumentation and support for custom runtime metrics.

However, configuring custom instrumentation gets frustrating if you have to instrument more than a handful of methods:

  • You need to specify each method to be instrumented down to the parameter list (if you have method overloads) – this is something of a time vampire
  • There’s no way of saying ‘just instrument everything in this class for me’
  • There’s no link in Visual Studio between the instrumentation config file and your code – refactoring breaks your instrumentation silently

To help, I’ve written a build-time tool that post-processes .NET assemblies and generates custom instrumentation files for NewRelic very quickly. It’s available on NuGet as two packages:

It’s got a GitHub project page that contains documentation and use-cases.

It’s got two modes:

Coarse-grained filtering on the command-line – no code changes required

Here we’re assuming that we don’t want to (or can’t) change the source of the projects we want to appear in the configuration file. The tool has a mode that does a more freestyle reflection over assemblies and generates instrumentation entries for every method (or constructor or property) that matches some very coarse criteria – it’s not perfect, but it’s intended to quickly bootstrap the instrumentation process with a file that can be manually tailored.

The workflow’s very simple:

  • Run the tool, specifying which code elements should be instrumented
    • One or more of {all, constructors, properties, methods}
    • Append + or – to the end of each class of code element to specify inclusion of public (+) or non-public (-) members
    • Use them in combination – for example, methods+- properties+ constructors+ will generate a configuration file to instrument all methods (public or otherwise), and public constructors and public properties
  • Either use the configuration file straight-off, or adjust it manually

Marking up your code with attributes

Need more control? The most fine-grained way to use the tool is to mark up assemblies, classes or methods (with properties and constructors also supported) with an [Instrument] attribute.

  • If you mark up an assembly with the attribute, it’s assumed you want to instrument every method in every class in that assembly – heavy handed but brutally straightforward
  • If you mark up a class with the attribute, it’s assumed you want to instrument every method in that class, and every method in any nested classes
  • You can limit the scope of the attribute to constructors, methods or properties (or combinations thereof), and to public and non-public members in each case

The workflow is still pretty straightforward:

  • Mark up your assemblies, classes or methods with [Instrument] attributes as required
  • Run the tool as a post-build step – it reflects over the assemblies and generates an appropriate custom instrumentation file

Download and Documentation

Binaries are available on NuGet – first, the tool itself (which can then be referred to as ‘nrconfig.exe’:

PM> Install-Package NRConfig.Tool

Then, optionally, the library that contains the [Instrument] attribute that can mark up assemblies, classes and methods:

PM> Install-Package NRConfig.Library

You can also download the tool and optional the library directly.

Documentation is available on the nrconfig GitHub page. Source is available at https://github.com/Pablissimo/nrconfig.

Limitations

New Relic’s instrumenting profiler has some limitations with respect to generics and the tool can’t do much about them:

  • It doesn’t appear possible to specify that a particular method overload should be instrumented if that method overload takes an instance of a generic class as a parameter and the generic class is itself parameterised by more than one type parameter. For example, IEnumerable<string> is fine, but IDictionary<string, string> isn’t as it has two type parameters.
    • The tool handles this by generating an instrumentation entry for all methods with the same name in the same class when a complex (i.e. >= 2 generic type parameters) generic type is detected – it means some methods get instrumented when you don’t want them to, but appears at present to be the only way to get instrumentation to work
  • It’s not possible to instrument generic classes at all.

Disclaimer

This project has nothing to do with New Relic the company – it’s neither supported nor sanctioned by them, so if it breaks come crying to me and not the nice folks on their support desk.

Ninject + NewRelic + Windows Azure Worker Role = horrific crash?

We recently encountered a weird issue – using Ninject for our DI needs, in a Windows Azure Worker Role with NewRelic instrumentation. All good-practice stuff, using proven technology.

And it didn’t work.

The role wouldn’t start up, RDPing in to look at the Event Log showed horrific errors in the .NET runtime itself:

Application: WaWorkerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in 
the .NET Runtime at IP 000007FEF6076B81 (000007FEF6070000) with exit 
code 80131506.

Application popup: WaWorkerHost.exe - System Error : Exception 
Processing Message 0xc0000005 Parameters 0x000007FEFD08718C 
0x000007FEFD08718C 0x000007FEFD08718C 0x000007FEFD08718C

Faulting application name: WaHostBootstrapper.exe, version: 
6.0.6002.18488, time stamp: 0x4fcaabe9
Faulting module name: ntdll.dll, version: 6.1.7601.17696, 
time stamp: 0x4e8147f0
Exception code: 0xc0000374
Fault offset: 0x00000000000a0d6f
Faulting process id: 0x4a8
Faulting application start time: 0x01cd9fb18d252932
Faulting application path: E:\base\x64\WaHostBootstrapper.exe
Faulting module path: D:\Windows\SYSTEM32\ntdll.dll

Using the Ninject source code and a bit of detective work the likes of which House would be proud, we found that creating a new AppDomain on an Azure Worker Role with NewRelic installed causes a pretty horrible crash. In fact, the issue had nothing to do with Ninject at all – simply having a Worker Role whose code created a new AppDomain caused the crash.

The solution we found was to:

  1. Construct Ninject kernels using either a list of INinjectModule instances or a list of assemblies where the modules live – importantly do not use the Load method with a set of string filenames as Ninject will create a new AppDomain to host the assemblies while it reflects over them which causes the above crash
  2. In the INinjectSettings object passed to the kernel constructor, turn off LoadExtensions as this causes the same code path as above to be run through, irrespective of whether any extension assemblies exist on disk

Both solutions aim to avoid the creation of a new AppDomain and thus avoid the crashing behaviour.

While we hope it’s an issue NewRelic will resolve in due course, hopefully the above’ll keep you going in the meantime.