Leedshack2 (6-7th August 2011)

First off the obvious one: What an absolutely awesome weekend!!

I had to get up at a ridiculous hour in the morning to get from Nottingham to Leeds in time for the iOS talk, after spending the previous day at a wedding getting rather drunk. This meant I was already feeling pretty ropey, but fortunately the infectious passion that Dave Verver had for iOS apps meant that the talk was really fun.

The day was then kicked off by Dom Hodgson who was the main organiser of this rather insane event. I think that without his fantastic personality and drive this event wouldn’t have been anywhere near as good. The fact he doesn’t try to be too serious but still got everything done and had time for everyone is absolutely amazing and he is a shining example of how to act when running these kind of events.

People then settled down to some pretty hardcore hacking, although there was lots of breaks for vast amounts of food and a little bit of Dom baiting. An especially fun episode was wrapping him up in bubble wrap and then the “incident” where he tried to chase Caius but unfortunately he didn’t quite succeed due to the fact I couldn’t resist standing on a trailing piece of bubble wrap which lead to a rather hilarious fall (It’s all on youtube somewhere …).

There were some truly fantastic hacks presented on the Sunday. My personal favourite was the PleasePledge app which did mean I was in no way going to win an Apple TV with Sausagefest, but that wasn’t really surprising was it? One of the really inspiring parts of it was the fact that it was written in PHP which is often derided for being a little bit of a “Teenage-hackers” / “Script Kiddies” language, but it had to be one of the most polished sites I’ve seen for a long time. In fact I would go as far to say that it kicks the pants off JustGiving and Virgin Money Giving.

I’m sure if I put my mind to it I could list a lot more that happened during this weekend, but the fact of the matter is that I still haven’t fully recovered, and that’s a good thing. There were so many people there that I talked to and it was amazing to be somewhere that had so many people around that were passionate about what they did. The only way you can truly understand how amazing one of these weekends is would be to attend, and to be honest when they are kept free like this what’s your excuse not to?

Thank you for the sausages for my “Mighty Meaty Prize”, I’m glad I could provide a bit of a laugh. Also a big thanks to everyone else for making this such an enjoyable experience! Thanks to all of the sponsors for all the food, even if I didn’t feel like eating at all on Sunday thanks to being force fed for the previous 24 hours. I now know what ducks must feel like in France when being subjected to gavage!

Finally all that is left to say:

Thank you Heather!

Please please please continue to keep Dom in check and thanks for ensuring that everything stayed organised and on track. Even though Dom is the face of Leedshack, we all know that you’re the one who kept it afloat!

Barcamp Nottingham 2011

I’ve pretty much just returned home after attending Barcamp Nottingham where I was locked inside for over 32 hours (apart form when the fire alarm went off). It was hosted at Hackspace Nottingham which is a truly amazing place to be. It all started yesterday at 9am when I met Dominic Finn (@cleverfinn) from North 51 for breakfast and then we headed over to the Hackspace for 10am. Getting registered and settled in was made really easy by the organisers and I felt very welcome.

After a rousing introduction by Gemma Cameron (@ruby_gem) and Dom from Hackspace (@ChickenGrylls / @HSNOTTS) people rushed over to put their talks on the boards. Each talk was (supposedly) limited to 20 mins with a little bit of discussion and swap over to basically fill a 30 min slot. Overall while it wasn’t the most tech-heavy event I’ve been to, which initially was a bit of a disappointment, it turned out to be thoroughly entertaining.

Most of the talks focused on peoples real passions or issues that they faced at work. It was great to see how much interaction each talk generated, and how well each subject was received. Where else could you go and see talks on a variety of computing topics intermingled with lace-making and how to be a Ninja! I even managed to get an extended lock picking lesson while @ruby_gem tattooed a sheep onto an orange!

As for my involvement? I gave a lightening talk about hosting Web Applications in the cloud, got really involved in some great chats with lots of people on a variety of topics, helped Dom Finn thrash the Esendex boys at arcade games, got eaten first when playing Werewolf, created a Node.js Werewolf server, stayed up until 5:30 coding a ruby web-crawler to scrape a BBQ web-ring (and playing games on Facebook and chatting) and drank beer while eating pizza.

All in all it was an absolute blast! You can read a lot of the shenanigans on the Twitter feed by using the hashtag #bcnott.

Finally a massive thanks to everyone that was there, to the Hackspace collective for hospitality, Gemma for organising it, North 51, Esendex and Nottingham Uni for sponsoring it and to Nescafe for making sure I didn’t fall asleep for too long. 🙂

So what am I upto?

Well quite a large change has happened in my life and I have started out as a contractor based initially in Nottingham, but soon I’ll be moving to Wetheral (which is near Carlisle). This has come about because my girlfriends family and friends are up in Newcastle / Sunderland and she wanted to be a bit closer to them all. It was quite an easy decision for me as my favourite places to go mountain biking are about 1 hour from Carlisle, as well as having easy access to the Lake District for some crazy open water swimming.

Initially I am consulting for North-51 (http://www.north-51.com), the company that has been my permanent employer for nearly 3 years, which is great for me as I really love the company. In fact I’ve said many times that if it wasn’t for the move, then I don’t think I would have considered becoming a contractor as life is pretty great here at N51 towers.

I’ve started pushing myself to become a bit more active in the local development community so I have started ensuring that I attend the events like GeekUp, Notttuesday and Second Wednesday which are proving to be quite fun. Hopefully I’ll be able to time my visits to Nottingham to also take in some of these events, and with any luck there will be something similar up in Carlisle.

The great thing about having my own company is that I can have my own goofy website: http://www.code-burst.co.uk. Although I think it’ll be quite obvious that I definitely fall into the stereotype that middleware developers don’t often make good designers.

Change is coming …

While I am awaiting the final details to be drawn up, all I can say is that my life is starting to change quite dramatically. Hopefully I’ll be able to say more soon!

Using MSMQ as a service bus

In a project I am working on at the moment I have designed and helped implement a MSMQ based service bus which allows us to run processes that have to retrieve data from external systems on machines outside of our web-farm (it’s a SharePoint 2010 Publishing solution). This allows us to offload the multi-threaded socket gobbling monster webservice calls to somewhere that wont impact the operations of the rest of the site. Also the way it is written made me think of how we could use it for CQRS and other high volume storage implementations.

To achieve all of this we basically have this sort of design:

This allows us to send messages through a generic sender class which will raise an event through a delegate whenever a message is received. In our usage of this we have messages that ask for a web service call to be executed and when the data is returned down the bus messages are routed into the database or returned to a waiting thread depending on some app logic.

A simple implementation of this bus is as follows:

    public class MSMQReceiver : IDisposable
    {
        readonly MessageQueue queue;

        public MessageReceived OnMessageReceived { get; set; }

        public MSMQReceiver()
        {
            queue = new MessageQueue("queue_name");
        }

        public void StartListening()
        {
            queue.ReceiveCompleted += queue_ReceiveCompleted;
            queue.BeginReceive();
        }

        void queue_ReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
        {
            queue.EndReceive(e.AsyncResult);
            queue.BeginReceive();

            OnMessageReceived(e.Message.Body);
        }

        public void Dispose()
        {
            queue.ReceiveCompleted -= queue_ReceiveCompleted;
            queue.Close();
            queue.Dispose();
        }
    }

    public delegate void MessageReceived(object data);

The above code creates a message queue (or attaches to one if it exists) that will asynchronously listen for messages. When a message is received it will put the body of the message into an object and fire off the delegate for you to what you want with it. This is only a very simple example, but it allows you to expand it and it provides the pattern to create a larger disconnected architecture that leads to interesting challenges like horizontal scaling in a cloud environment or maybe your first home-brew CQRS implementation.

How not to suck at LINQ-to-Objects

So the main problem some people have when coming across LINQ-to-Objects is that they think it will work in the same way as LINQ-to-SQL. The fabled John Skeet has written a very in-depth blog series into how LINQ-to-Objects is implemented, and this post is intended to give you a quick overview and some handy pointers.

With LINQ-to-SQL, and other well implemented IQueryables, your LINQ statements compile down to the most efficient query possible. This often takes advantage of SQLs aggregation abilities. LINQ-to-Objects however only operates on the previous enumerable input. This means that sometimes it can be easy to write statements that iterate through your collections many times more than necessary. Often this isn’t an issue as your collections might be small, but the efficiency gains learnt from understanding LINQ-to-Objects can also lead to more readable code.

So without further ado, lets look at some examples!

Checking that a collection contains at least one item

So maybe you have written something like the following to test that a collection contains items?

_collection.Count() > 0;

But this will cause you to iterate through the entire collection to get a Count and then do a comparison of that to 0. If you have 100,000 items in your collection (maybe your caching something?!) then you’re going to have a long running for/foreach loop in there eh?

Why not do the following instead?

_collection.Any();

Checking that a collection contains at least one specific item (i.e. using a where clause)

Ok, so it seems simple when you know about it, but sometimes you start learning these things and it’s just not explained to you. This method simply tries to start an iteration, and if items are available then it returns true. If it can’t iterate it returns false. So you instantly get a huge performance saving if you have more than 1 item in your collection.

With SQL you might write something like this:

SELECT TOP 1 ISNULL('1', '0') FROM Table WHERE SomeField = @SomeValue

This is just testing if a table contains at least one of our value. I’ve often seen this implemented as this:

_collection.Select(x=> x.SomeField).Where(x=> x == SomeValue).Count() > 0;

I bet you’re thinking “well that should at least finish with .Any() eh?!”, and you’d be right of course. However in our query we have iterated through the collection fully to get an IEnumerable<T> of SomeField for the Select, then again to get the subset (although iterating fully) of IEnumerable<T> for the Where and then iterated through it fully again to get a Count. Obviously using Any() provides a way of reducing these iterations. However there is further scope for reducing these iterations …

Most of the methods for LINQ-to-Objects selection methods also take some kind of filter, we can use this with Any() to produce a way to iterate only part way, if a match exists, of the Collection.

_collection.Any(x=> x.SomeField == SomeValue);

Getting the first item out of a list using a predecate

So just like the previous example you might want to get the first value from the list in it’s entirety. Maybe you copied your previous query and just put the following in:

_collection.Select(x=> x.SomeField).Where(x=> x == SomeValue).FirstOrDefault();

As with the previous example, this will cause at least 2 full iterations of our collection (or intermediate collections). To our rescue though is the fact that we can add predicates to FirstOrDefault() just as we could with Any(). So if we re-write it like so:

_collection.FirstOrDefault(x=> x.SomeField == SomeValue);

Then we get a performance gain as well as reducing the amount of code we have to read and write.

Conclusion

I sincerely apologise if this post might be obvious to you, and if it wasn’t and it came across as patronising then I am sorry for that too. However hopefully it has highlighted that poking around a little with the method signatures for LINQ-to-Objects will give you some great performance gains as well as increase the readability of your code.

As a final note you might want to stay away from doing ToList() or ToArray() unless you absolutely need to convert your collection as those types. Almost all of the LINQ selectors use Yield, which delays execution of the query until you start iterating through the Collection. If you do call ToList() or ToArray() then it will iterate through the entire collection and copy it to an in-memory object, which negates any benefit from using LINQ.

Why does social media and blogging seem like a chore?

I have such great admiration for all those people out there who manage to create the consistently interesting blogs I read on a daily basis. Why does it always seem like such an afterthough for me? Afterall I’ve been able to access the internet for over 17 years. I must have read thousands of blog posts and articles people have written, yet my contribution always seems so scattered and infrequent. Because of some interesting developments in my life recently I have promised myself to try and correct this. I’ve been devoting a lot of time to expanding my development skills and knowledge in recent months, and with a couple of interesting ideas in the pipeline hopefully I’ll have a bit more to say next time I post here …

Developing applications on the move

I have recently had the mad idea that I want to be able to start prototyping apps from anywhere with a stable internet connection. Also it was a nice thought to be able to do it from my iPad (or even iPhone if I am feeling brave), which meant that I couldn’t really think in terms of traditional IDE environments.

After a fair bit of thought I had a minor brainwave while working on a Node.js script in vim on Ubuntu (this is relevant, but not right now!) and the following questions needed answering rather quickly:

• Is there a decent iPad / iPhone SSH client?
• Can that client use Amazon EC2 keys?
• Could I really learn Vim?
• Where are the biscuits? (I get distracted easily)

After a little bit of searching it turned out that all of the above questions could easily be answered. So you may be thinking what exactly was that brainwave? Well it basically boils down to this: Use Ubuntu via SSH from the iPad to develop and just man up and use Vim for it!

I first tried to use “SSH Term Pro” from the Apple App Store, but this turned out to be a no-go as it doesn’t support Amazons “pem” keys. However with a little googling I found “Prompt” and with the aid of jchen’s guide I was up and running with the minimum of fuss (ok I had to check up on this to see how to transfer the keys).

The set-up part of EC2 was relatively straightforward. I just provisioned the official Ubuntu image (I used Natty Narwhal from their guide) and ran it on a extra large instance for all the installs of Ruby/Rails/Node.js/Redis etc and then scaled it down to a micro instance afterwards.

All I have left to do is buy a wireless keyboard and my plan is complete!

Now where did I leave those biscuits ….