Wednesday, June 29, 2011

I Don’t Have Time for Unit Tests

I’ve helped several organisations adopt Test Driven Development (TDD). The initial worry that almost everyone has, is that it will hurt productivity. It seems intuitively correct, because of course, the developer has to write the unit tests as well as the production code. However when you actually look at how developers spend their time, actually writing code is a small part of it. Check out this study by Peter Hallam from the Visual Studio team at Microsoft:

http://blogs.msdn.com/b/peterhal/archive/2006/01/04/509302.aspx

According to Peter, developers actually spend their days like this (while not reading Code Rant of course :):

  • Writing new code 5%
  • Modifying existing code 25%
  • Understanding Code 70%

If some technique allowed you to half the modifying/understanding parts of the job by doubling the new-code part, you’d now be taking only ~60% of the time you previously took to deliver a feature, almost doubling your productivity.

TDD is that technique. In my experience the productivity gains come from:

  • Allows safe modifications. If you break something when you modify some code, the unit tests fail.
  • Shortening the iteration cycle between writing/running code. No more need to step through your application to part where your new code gets exercised.
  • Mistakes in your code are shallow and obvious. No more need to step through code in the debugger, wondering which part of your application is broken.
  • Code is self-documenting. The unit tests explicitly show how the author expected the code to be used.
  • Code is decoupled. You can’t do TDD without decoupling. This alone makes the code easier to understand as a unit and much safer to modify.

Note that I’m just talking about feature-delivery productivity here. I haven’t mentioned the huge gains in stability and drop in bug-counts that you also get.

Now I don’t deny that TDD is a fundamentally different way of working that takes time to learn. Undoubtedly productivity may drop while a developer is getting up to speed. But in my experience the argument that a developer doing TDD is slower than a developer who doesn’t do it is simply not true.

Monday, June 27, 2011

RabbitMQ, Subscription, and Bouncing Servers in EasyNetQ

If you are a regular reader of my blog, you’ll know that I’m currently working on a .NET friendly API for RabbitMQ, EasyNetQ. EasyNetQ is opinionated software. It takes away much of the complexity of AMQP and replaces it with a simple interface that relies on the .NET type system for routing messages.

One of the things that I want to remove from the ‘application space’ and push down into the API is all the plumbing for reporting and handling error conditions. One side of this to provide infrastructure to record and handle exceptions thrown by applications that use EasyNetQ. I’ll be covering this in a future post. The other consideration, and the one I want to address in this post, is how EasyNetQ should gracefully handle network connection or server failure.

The Fallacies of Distributed Computing tell us that, no matter how reliable RabbitMQ and the Erlang platform might be, there will still be times when a RabbitMQ server will go away for whatever reason.

One of the challenges of programming against a messaging system as compared with a relational database, is the length of time that the application holds connections open. A typical database connection is opened, some operation is run over it – select, insert, update, etc – and then it’s closed. Messaging system subscriptions, however, require that the client, or subscriber, holds an open connection for the lifetime of the application.

If you simply program against the low level C# AMQP API provided by RabbitHQ to create a simple subscription, you’ll notice that after a RabbitMQ server bounce, the subscription no longer works. This is because the channel you opened to subscribe to the queue, and the consumption loops attached to them, are no longer valid. You need to detect the closed channel and then attempt to rebuild the subscription once the server is available again.

The excellent RabbitMQ in Action by Videla and Williams describes how to do this in chapter 6, ‘Writing code that survives failure’. Here’s their Python code example:

rabbit_mq_in_action_failure_detecting_subscriber

EasyNetQ needs to do something similar, but as a generic solution so that all subscribers automatically get re-subscribed after a server bounce.

Here’s how it works.

Firstly, all subscriptions are created in a closure:

public void Subscribe<T>(string subscriptionId, Action<T> onMessage)
{
if (onMessage == null)
{
throw new ArgumentNullException("onMessage");
}

var typeName = serializeType(typeof(T));
var subscriptionQueue = string.Format("{0}_{1}", subscriptionId, typeName);

Action subscribeAction = () =>
{
var channel = connection.CreateModel();
DeclarePublishExchange(channel, typeName);

var queue = channel.QueueDeclare(
subscriptionQueue, // queue
true, // durable
false, // exclusive
false, // autoDelete
null); // arguments

channel.QueueBind(queue, typeName, typeName);

var consumer = consumerFactory.CreateConsumer(channel,
(consumerTag, deliveryTag, redelivered, exchange, routingKey, properties, body) =>
{
var message = serializer.BytesToMessage<T>(body);
onMessage(message);
});

channel.BasicConsume(
subscriptionQueue, // queue
true, // noAck
consumer.ConsumerTag, // consumerTag
consumer); // consumer
};

connection.AddSubscriptionAction(subscribeAction);
}

The connection.AddSubscriptionAction(subscribeAction) line passes the closure to a PersistentConnection class that wraps an AMQP connection and provides all the disconnect detection and re-subscription code. Here’s AddSubscriptionAction:

public void AddSubscriptionAction(Action subscriptionAction)
{
if (IsConnected) subscriptionAction();
subscribeActions.Add(subscriptionAction);
}

If there’s an open connection, it runs the subscription straight away. It also stores the subscription closure in a List<Action>.

When the connection gets closed for whatever reason, the AMQP ConnectionShutdown event fires which runs the OnConnectionShutdown method:

void OnConnectionShutdown(IConnection _, ShutdownEventArgs reason)
{
if (disposed) return;
if (Disconnected != null) Disconnected();

Thread.Sleep(100);
TryToConnect();
}

We wait for a little while, and then try to reconnect:

void TryToConnect()
{
ThreadPool.QueueUserWorkItem(state =>
{
while (connection == null || !connection.IsOpen)
{
try
{
connection = connectionFactory.CreateConnection();
connection.ConnectionShutdown += OnConnectionShutdown;

if (Connected != null) Connected();
}
catch (RabbitMQ.Client.Exceptions.BrokerUnreachableException)
{
Thread.Sleep(100);
}
}
foreach (var subscribeAction in subscribeActions)
{
subscribeAction();
}
});
}

This spins up a thread that simply loops trying to connect back to the server. Once the connection is established, it runs all the stored subscribe closures (subscribeActions).

In my tests, this solution has worked very nicely. My clients automatically re-subscribe to the same queues and continue to receive messages. One of the main motivations to writing this post, however, was to try and elicit feedback, so if you’ve used RabbitMQ with .NET, I’d love to hear about your experiences and especially any comments about my code or how you solved this problem.

The EasyNetQ code is up on GitHub. It’s still very early days and is in no way production ready. You have been warned.

Thursday, June 02, 2011

Some Thoughts on Windows 8

Microsoft is the rabbit caught in Apple’s headlights… and about to be run over by the Google juggernaut. Microsoft’s income comes from two major sources, Windows and Office. The need to maintain the stream of licence fees for these two products is at the very core of everything Microsoft does. Windows has three major groups of customers: consumer PCs, business PCs and business servers. Microsoft is the incumbent in all three markets, it can’t grow any more by taking market share, its income can only increase at the rate of growth for these markets as a whole. The only way Microsoft can break out of this static lock is by creating new markets for its products. But it must be careful not to injure it’s existing Windows franchises in attempts to get market share elsewhere.

But it’s a tough world to be selling operating system licences. Unfortunately for Microsoft, the situation is far from static. It’s core source of income is under threat. The iPad has created a new class of consumer product that is eroding Microsoft’s market for consumer PCs. Many people have no need for the full power of a desktop PC. For browsing, reading and writing email and watching YouTube, an iPad, or one of the many Android competitors is perfectly adequate. Android in particular is becoming more and more able to do PC like tasks with every release. For many people a PC is overcomplicated and unreliable.

Windows is also under threat in business server rooms. The main challenge here is from cloud based services. Why employ expensive people to look after servers running Exchange when you can simply sign up for Google Apps? For many small and medium businesses, cloud based services are a very attractive alternative to running in-house IT.

Windows is probably safest of all on the business desktop. There’s no real alternative for running productivity applications like Office. However, with many line-of-business applications becoming cloud based, there is a risk that a Chrome OS style browser-only desktop might look attractive to some business. Also when everyone’s got an iPad or an Android tablet at home, having a similar device at work will start to make more sense.

So is Windows 8 an answer to any of these challenges? It’s obviously designed to answer the first, the erosion of the consumer PC market by iPad and friends. Fundamentally it looks like a simple UI layer, derived from WP7, stuck on top of Windows 7. Microsoft’s strategy seems to be to offer a simple touch-UI for ‘consumer’ tasks, but which allows you to switch back to Windows 7 for running desktop applications like Office. John Gruber makes the point here, that simply skinning Windows 7 for tablets is probably a mistake.

Microsoft is not the leader in the tablet market, it’s playing catch-up from quite a long way behind. Will consumers be willing to pay the Windows tax when they can simply buy a more mature iPad or Android device? Microsoft can’t start offering it for free like Google does with Android because they would immediately kill one of their main sources of income. There is no way they can do anything other than ask people to pay more, for what will, at least initially, be an inferior device. It doesn’t strike me as a winning strategy.

The Windows 8 developer story is somewhat odd to say the least. No mention of WPF or Silverlight. Instead developers are being asked to build apps for the Windows 8 touch UI in Javascript and HTML. There are more Javascript developers out there than Silverlight ones, but the people who already care about Microsoft platforms have put considerable investment into WPF and Silverlight. The Windows 8 announcement is a real slap-down for them. Just take a look at the anger on Channel 9. It’s also a snub to Microsoft’s developer division who have put considerable effort into building .NET. Is this a hangover from the Vista debacle? Was WinFX such a failure that the Windows team now want nothing to do with .NET? If so, what’s the point of the developer division?

It’s a confusing time for us developers. I think Microsoft still have a very strong position on the business desktop. If you are building line-of-business applications for a living with .NET, there’s probably still some millage in that. But the feeling is very much that .NET is now middle age, like many of its developers. No matter how nice the technology is, and it is very nice, it’s part of a platform that’s perceived to be part of the past, not the future.

But if you are building for the consumer market, or the cloud, then Microsoft is merely a niche player. It’s been obvious for some time that everyone should have Javascript as a core skill. The Windows 8 announcement reinforces that. It’s also clear that UNIX/Linux operating systems (including iOS and Android) are now ubiquitous in the same way that TCP/IP is. You’d probably want to make sure you know how to find your way around a UNIX system. For building server side applications for the cloud the field is wide open. Ruby, Python & Javascript are all well established and for some exotic applications, things like Erlang, Scala and Haskell are very interesting. Personally I’m hoping, optimistically I think, that Mono and Xamarin are a success. The excellent .NET platform deserves more than to be chained to Windows’ declining market share.