Archives for the month of: June, 2011

In a project I am working on at the moment I have designed and helped implement a MSMQ based service bus which allows us to run processes that have to retrieve data from external systems on machines outside of our web-farm (it’s a SharePoint 2010 Publishing solution). This allows us to offload the multi-threaded socket gobbling monster webservice calls to somewhere that wont impact the operations of the rest of the site. Also the way it is written made me think of how we could use it for CQRS and other high volume storage implementations.

To achieve all of this we basically have this sort of design:

This allows us to send messages through a generic sender class which will raise an event through a delegate whenever a message is received. In our usage of this we have messages that ask for a web service call to be executed and when the data is returned down the bus messages are routed into the database or returned to a waiting thread depending on some app logic.

A simple implementation of this bus is as follows:

    public class MSMQReceiver : IDisposable
        readonly MessageQueue queue;

        public MessageReceived OnMessageReceived { get; set; }

        public MSMQReceiver()
            queue = new MessageQueue("queue_name");

        public void StartListening()
            queue.ReceiveCompleted += queue_ReceiveCompleted;

        void queue_ReceiveCompleted(object sender, ReceiveCompletedEventArgs e)


        public void Dispose()
            queue.ReceiveCompleted -= queue_ReceiveCompleted;

    public delegate void MessageReceived(object data);

The above code creates a message queue (or attaches to one if it exists) that will asynchronously listen for messages. When a message is received it will put the body of the message into an object and fire off the delegate for you to what you want with it. This is only a very simple example, but it allows you to expand it and it provides the pattern to create a larger disconnected architecture that leads to interesting challenges like horizontal scaling in a cloud environment or maybe your first home-brew CQRS implementation.


So the main problem some people have when coming across LINQ-to-Objects is that they think it will work in the same way as LINQ-to-SQL. The fabled John Skeet has written a very in-depth blog series into how LINQ-to-Objects is implemented, and this post is intended to give you a quick overview and some handy pointers.

With LINQ-to-SQL, and other well implemented IQueryables, your LINQ statements compile down to the most efficient query possible. This often takes advantage of SQLs aggregation abilities. LINQ-to-Objects however only operates on the previous enumerable input. This means that sometimes it can be easy to write statements that iterate through your collections many times more than necessary. Often this isn’t an issue as your collections might be small, but the efficiency gains learnt from understanding LINQ-to-Objects can also lead to more readable code.

So without further ado, lets look at some examples!

Checking that a collection contains at least one item

So maybe you have written something like the following to test that a collection contains items?

_collection.Count() > 0;

But this will cause you to iterate through the entire collection to get a Count and then do a comparison of that to 0. If you have 100,000 items in your collection (maybe your caching something?!) then you’re going to have a long running for/foreach loop in there eh?

Why not do the following instead?


Checking that a collection contains at least one specific item (i.e. using a where clause)

Ok, so it seems simple when you know about it, but sometimes you start learning these things and it’s just not explained to you. This method simply tries to start an iteration, and if items are available then it returns true. If it can’t iterate it returns false. So you instantly get a huge performance saving if you have more than 1 item in your collection.

With SQL you might write something like this:

SELECT TOP 1 ISNULL('1', '0') FROM Table WHERE SomeField = @SomeValue

This is just testing if a table contains at least one of our value. I’ve often seen this implemented as this:

_collection.Select(x=> x.SomeField).Where(x=> x == SomeValue).Count() > 0;

I bet you’re thinking “well that should at least finish with .Any() eh?!”, and you’d be right of course. However in our query we have iterated through the collection fully to get an IEnumerable<T> of SomeField for the Select, then again to get the subset (although iterating fully) of IEnumerable<T> for the Where and then iterated through it fully again to get a Count. Obviously using Any() provides a way of reducing these iterations. However there is further scope for reducing these iterations …

Most of the methods for LINQ-to-Objects selection methods also take some kind of filter, we can use this with Any() to produce a way to iterate only part way, if a match exists, of the Collection.

_collection.Any(x=> x.SomeField == SomeValue);

Getting the first item out of a list using a predecate

So just like the previous example you might want to get the first value from the list in it’s entirety. Maybe you copied your previous query and just put the following in:

_collection.Select(x=> x.SomeField).Where(x=> x == SomeValue).FirstOrDefault();

As with the previous example, this will cause at least 2 full iterations of our collection (or intermediate collections). To our rescue though is the fact that we can add predicates to FirstOrDefault() just as we could with Any(). So if we re-write it like so:

_collection.FirstOrDefault(x=> x.SomeField == SomeValue);

Then we get a performance gain as well as reducing the amount of code we have to read and write.


I sincerely apologise if this post might be obvious to you, and if it wasn’t and it came across as patronising then I am sorry for that too. However hopefully it has highlighted that poking around a little with the method signatures for LINQ-to-Objects will give you some great performance gains as well as increase the readability of your code.

As a final note you might want to stay away from doing ToList() or ToArray() unless you absolutely need to convert your collection as those types. Almost all of the LINQ selectors use Yield, which delays execution of the query until you start iterating through the Collection. If you do call ToList() or ToArray() then it will iterate through the entire collection and copy it to an in-memory object, which negates any benefit from using LINQ.

I have such great admiration for all those people out there who manage to create the consistently interesting blogs I read on a daily basis. Why does it always seem like such an afterthough for me? Afterall I’ve been able to access the internet for over 17 years. I must have read thousands of blog posts and articles people have written, yet my contribution always seems so scattered and infrequent. Because of some interesting developments in my life recently I have promised myself to try and correct this. I’ve been devoting a lot of time to expanding my development skills and knowledge in recent months, and with a couple of interesting ideas in the pipeline hopefully I’ll have a bit more to say next time I post here …

I have recently had the mad idea that I want to be able to start prototyping apps from anywhere with a stable internet connection. Also it was a nice thought to be able to do it from my iPad (or even iPhone if I am feeling brave), which meant that I couldn’t really think in terms of traditional IDE environments.

After a fair bit of thought I had a minor brainwave while working on a Node.js script in vim on Ubuntu (this is relevant, but not right now!) and the following questions needed answering rather quickly:

  • Is there a decent iPad / iPhone SSH client?
  • Can that client use Amazon EC2 keys?
  • Could I really learn Vim?
  • Where are the biscuits? (I get distracted easily)

After a little bit of searching it turned out that all of the above questions could easily be answered. So you may be thinking what exactly was that brainwave? Well it basically boils down to this: Use Ubuntu via SSH from the iPad to develop and just man up and use Vim for it!

I first tried to use “SSH Term Pro” from the Apple App Store, but this turned out to be a no-go as it doesn’t support Amazons “pem” keys. However with a little googling I found “Prompt” and with the aid of jchen’s guide I was up and running with the minimum of fuss (ok I had to check up on this to see how to transfer the keys).

The set-up part of EC2 was relatively straightforward. I just provisioned the official Ubuntu image (I used Natty Narwhal from their guide) and ran it on a extra large instance for all the installs of Ruby/Rails/Node.js/Redis etc and then scaled it down to a micro instance afterwards.

All I have left to do is buy a wireless keyboard and my plan is complete!

Now where did I leave those biscuits ….