Bjarte.Com

Javascript – this, new and prototype

Javascript is a special language with somewhat unusual features. Often you see people struggling with the concept of “this”, “new” and “prototype”. This is my attempt to shed some light on these mysterious concepts. If you are coming from a statically typed language like C# or Java, you will quickly realise that they are not what they seem to be.

This and that

A common mistake is to think that the “this” keyword in Javascript refers to the instance of the object you are in. This is the common use in C# and Java.

You can think of the “this” keyword as an implicit argument to the function. In a browser this implicit parameter will be set to the root object by default, e.g. the window object:

function foo(){
	return this;
}
	
alert(foo()===window) //true

However, you can pass the “this” reference to the function you are calling by using the call (or apply) function:

	
function foo(){
	return this;
}
	
//calling foo with “this” assigned to the value “hello”
var result = foo.call(“hello”) 

alert(result) //”hello”

As you can see, the “this” keyword depends on the context in which you call the function foo.

The new keyword

To construct objects in Javascript you use the “new” keyword together with a constructor function. Functions that are used as a constructor usually starts with an upper case letter (best practice)

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = new Foo();

alert(obj.helloWorld()) //”Hello world”

When you call a function with the “new” keyword, a new empty object is created and then the constructor function is called with “this” set to the new object. In the example above, this will cause the new object to get the method “helloWorld” assigned to it. The prototype hierarchy is also set up, but I’ll come back to that later.

The following code demonstrates another way to get the same effect as the new keyword (without the prototype hierarchy)

function foo(){
	this.helloWorld = function(){return "Hello world"}
}

//emulate "new"
var obj = {};
foo.call(obj);

alert(obj.helloWorld()) //"Hello world"

Forgetting the new keyword

If you “forget” to use the new keyword on a function that is meant to be used as a constructor you will get strange behaviour:

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = Foo();

alert(obj.helloWorld()) //fail
alert(window.helloWorld!==undefined) //true

You just assigned the function helloWorld to the window object. Probably not the intention :)

The prototype hierarchy

When you use the new keyword, the prototype hierarchy is set up. This means that the new object will have the methods in the prototype object of the Foo function:

function Foo(){
	this.helloWorld = function(){return "Hello world"}
}

//define a method on the prototype
Foo.prototype.helloWorld2 = function(){return "Hello world 2"}


var obj = new Foo();

alert(obj.helloWorld()); //"Hello world"
alert(obj.helloWorld2()); //"Hello world 2"

The new object will also have methods defined higher up in the prototype hierarchy.

Well. That’s it for now.

The beauty of open source

This post is just about me sharing some of the stuff I have looked at in the open source community lately. There might be some ruby stuff in here. First out is rack.

Rack

Imagine for a minute that there were a lot of different web servers out there like IIS, Apache and perhaps even Kayak. Also imagine that there were a lot of different web frameworks out there, like Fubu MVC, ASP.Net Web Forms (don’t get me started) and Asp.Net MVC.

Wouldn’t it be cool if we could put any web server together with any web framework just by putting a common abstraction in between ?

The web frameworks would only need to implement an adapter working with the abstraction and the same for would go for the web server. To get n web frameworks working with m different web servers, we would only need to implement n+m adapters as opposed to n*m without the abstraction. We could pick our favorite server and our favorite web framework and start developing with a smile on our face. (Ignore for a minute that this is not really possible with the servers and frameworks I just mentioned)

What would this abstraction look like ? Well, a web server is at its hart a very simple thing. You get a request and return a response. A response is simply a status, a set of headers, and a body.

In .NET a very simple implementation of a web “framework” adhering to the abstraction would look something like this.

public Response Call(Environment env)
{
  return new Response(
    status: 200,
    headers: new Dictionary<string, string> { { "Content-Type", "text/plain" } },
    body: new List<string> { "Hello world" });
}

or the more terse ruby version

def call(env)
  [200, {"Content-Type" => "text/plain"}, ["Hello world!"]]
end

Now I finally came to the point of this entire rant. In ruby they actually have this abstraction in place. It’s called rack. You can mix and match web servers and web frameworks to make your coding life a dance on roses.

I like it.

You would think the story ends here, but it doesn’t. There is also something called rack middleware. It takes advantage of the common abstraction between the web server and the web framework and offers you the ability to put logic in between the two. It is entirely possible to create rack middleware for debugging, logging, authentication, the list goes on. You now have reusable components for your web site that is independent of both the web server and the web framework you have chosen. Cool huh?

I like it.

(This part of the post was mostly inspired by this introduction to rack)

Sinatra

Sinatra is a rack enabled web framework. Its implementation is very small and the sample application is even smaller. I will not say much about Sinatra, because the web site says it all.

require 'sinatra'
get '/hi' do
  "Hello World!"
end

Ahhh, the beauty of simplicity. Describe the URL, define the action, and you’re done.

Since Sinatra supports rack, you get the power of rack middleware in addition to the simplicity of Sinatra. I like it.

Tilt

Any serious web framework needs a nice templating engine. In ruby you have a lot of choises like erb, haml and my current favorite mustache. It would be a pain (in the ass) to implement support for all these engines in every web framework, so the ruby guys have (mostly) agreed on tilt. Tilt is a common abstraction that goes between the web framework and the templating engine. Sounds familiar ? To use a tilt enabled templating engine, you need to supply the template and some context for the template (the data). Easy easy.

template = Tilt::ERBTemplate.new('templates/foo.erb')
joe = Person.find('joe')
output = template.render(joe)

Nice stuff.

Well, that’s it for now. See you later.

Improving software quality

NDC 2010 was great. This post is inspired by Tom Gilbs ”Lean quality assurance” talk:

Often we set out to build high quality software without really considering what that entails. How can you really know, when project has finished, that you have created quality software?

Clean code, test coverage and executable specifications created iteratively together with the primary stakeholders seem to be the main focus for most agile projects, at least, in agile theory. There are other concerns that can easily be overlooked when people go into “agile mode”.

Numeric entry and exit

As a stakeholder in a project you don’t really care about testing and executable specifications. You care about the ilities. Usability, Maintainability, Reliability, Security and Scalability are some that come to mind. Often requirements emerge as the software is developed, but many of these are well known in advance, at least to the stakeholders.

How can we identify these requirements and ensure that they are met at the end of the project?

A coder will leave it to chance. A software engineer will measure and verify that the requirements are met.

To identify the requirements you first need to identify the stakeholders. This is a process in its own right. Later you can start identifying the requirements.

According to Tom Gilb you can measure *any* requirement numerically. Later in the process you can use these numbers to evaluate the quality of the software (or some other item) you are producing. A requirement is considered done when your software meets the numeric target for that requirement. “Numeric entry and exit” is a term that describes this way of thinking.

How can you measure things like usability of a feature? The answer is obviously “it depends”. One metric can be the percentage of grandmothers that understand the feature on the first try. If 80% get it, you have a usable product, and you have met your target for that specific requirement.

Obviously dissecting your requirements this way requires a lot of time and effort. For some projects doing it this way makes no sense, for some it’s too cumbersome and expensive, and for some it totally makes sense.

You get what you measure

Later.

Writing maintainable integration tests for web applications

I guess I need to supply some context here: I’m still in the JavaScript world and I’m still writing desktop-like applications that run in the browser. I only load the DOM once. Here we go:

Every test you add to your application creates a dependency on it. More tests mean more dependencies. Dependencies spell trouble, especially if you depend on code that changes a lot, a.k.a. unstable components.

If you have a test to verify the correctness of a feature, the test should touch a minimal amount code, and it should ideally only touch stable components.

A common problem with integration tests is that they break for reasons having nothing to do with the feature they are testing. You often run in to this problem if you are testing thru the UI. The UI is very unstable (it changes a lot). Minor design changes can break tests. If you cannot easily change your application without creating extra work, the ROI of the integration tests is diminishing.

One solution is to ignore the outer layer of the UI and instead test one layer in. When I write desktop-like web applications I have a thin view layer that encapsulates all the UI interaction (http://bjarte.com/post/effective-javascript-unit-testing). This means that I can invoke every user action thru the view layer.

Let’s look at the following example of a search box:

The view would look something like this:

function SearchView(callback) {
  
  $("#search").click(function () {
    callback($("#q").val());
  });

  function doSearch(){
    $("#search").click();
  }

  return {
    doSearch: doSearch
  }
}

Somewhere else in my code I will make all my views visible globally thru a “views” collection. From my tests I can now invoke behaviour by talking directly to the SearchView:

views.searchView.doSearch();

I actually use WatiN to test thru the browser. The only line of code I need in my test (actually specification) to invoke the behaviour is

Browser.Eval("views.searchView.doSearch();");

As you can see, some of my application code is there only to support integration tests (e.g. the doSearch method on the view). I don’t have a problem with that.You can plug a computer into a modern car to diagnose it. Why shouldn’t you be able to diagnose software?

The point here is that my tests depend on more stable code. The layout can change. The button can move, or perhaps become an image. The search behaviour will stay the same. Hopefully.

Obviously you are not testing EVERYTHING when you don’t run the tests thru every layer of the application. But you know what ? You cannot test everything anyways. You come to a point where writing more tests will not give you more benefits. At some point more tests become a problem, a maintenance problem. Where you draw this line is dependent, not only on your technical challenges, but also the people in your team, the culture, the process and many other factors. Context, context, context.

Later.

Effective JavaScript unit testing

Most JavaScript testing frameworks run in the browser and are tightly coupled to the DOM.

I don’t care about the DOM and I definitely don’t care about the browser, at least not when I am (unit) testing my JavaScript components.

This probably needs some context; I’m not writing web sites with 5 lines of JavaScript in them you see. I’m writing desktop like applications that run in the browser. This means I have a lot of widgets and screens that need to talk to each other. I am also doing XMLHTTP calls to the server. All the interaction needs to be tested.

I don’t want to run the tests thru the GUI. It’s to brittle. Changing an a-tag to an input-tag or moving a component on the page can easily break a test. Testing is supposed to improve productivity, not slow it down.

I use the Model-View-Presenter design pattern to design my application layer. This means I have a thin layer, call “the view”, over my HTML that abstracts away both the DOM and with it, the browser.

The View

Let’s say I have the following component in my application. (It would probably say “search” and not “Google search”, but you get it)

The corresponding HTML would look something like this (ignoring the lucky-button)

<input id="q" type="text" />
<input id="search" type="button" />

I would now create a SearchView module to put an abstraction on top of this HTML (using some JQuery magic)

function SearchView(callback){
	$("#search").click(function(){
		callback($("#q").val());
	});
}

Now the rest of my code will depend on the SearchView function, and only the SearchView function will depend on the browser DOM. But that’s no problem. We can always create a fake SearchView function to act as a stand in for testing purposes.

The key here is that even if I build a very complex application with a lot of interaction, it is only the thin view layer that is depending on the browser DOM. The rest of the code is pure JavaScript.

Testing the code (on the Windows platform)

So I have A LOT of JavaScript code to test, and it does not depend on the browser DOM, so why would I need a browser to test the code ? Wouldn’t it be much better to run the tests thru a command line JavaScript interpreter? Or even better, have the tests run inside Visual Studio ? I definitely think so. Opening up a browser to run a unit test is way too much friction for me.

There is a JavaScript interpreter build into Windows. You can run it thru the cscript command line tool. To run a file, you do the following:

cscript myFile.js

Obviously if you are doing any meaningful testing you need to load multiple files into the environment, the testing framework, the tests, and the system under test. Luckily you can run multiple files at the same time by creating an xml file. Let's call the file test.wsf.

<?xml version="1.0" encoding="utf-8" ?>
<package>
   <job id="1">
     <script language="JScript" src="TestGui.js" />
     <script language="JScript" src="TestRunner.js" />
     <script language="JScript" src="Foo.js" />
     <script language="JScript" src="FooTests.js" />
   </job>
</package>

You can now run all the files

cscript test.wsf

We can now use this command line tool inside Visual Studio by creating an “external tool”

Tools -> External tools

By adding a shortcut to the external tool, I can run my JavaScript tests from anywhere in the Solution, only using a single keystroke. AND IT’S FAST!

The test runner / framework

As I mentioned previously, most JavaScript testing frameworks out there are tightly coupled to the browser DOM. That will not fly when you are using the Windows Script Host. I chose to build my own little test framework to run the tests. It is very small, and only have three method calls that are coupled to the environment it is running in.

//write to console
WScript.Echo(s); 	
...	
//quit application with error (which will break the build on TeamCity)
WScript.Application.Quit(1);	
...
//write to StdErr
WScript.StdErr.WriteLine(s);

As you might tell from the code sample above, I run these tests on our CI server as well. No problem there.

My home-made testing framework is in an early stage. You can look at it if you want to, but don’t expect too much: http://github.com/BjartN/BigUnit.

Well, that’s it for now. Later.

Object design in JavaScript

JavaScript has gotten a bad reputation in the software industry. Most of it is related to the browser DOM and incompatibility across browsers. This problem is starting to go away due to great frameworks like Jquery and Prototype. Web 2.0 sites with heavy AJAX functionality forces browser manufacturers to improve their JavaScript interpreters. JavaScript is even becoming a contender on the server side.

Designing objects

If you google ”object oriented design (OOD) using JavaScript” you will find that most people build JavaScript objects using prototype and the “new” keyword. Here is how you normally would create an object:

var Address = function (street) {
  this._street = street;
};

Address.prototype = {
  setStreet: function (street) {
    this._street = street
  },

  getStreet: function(){
    return this._street;
  }
}

I create the function Address. This function will work as my constructor. Then I extend Address’ prototype with two functions, setStreet and getStreet. I can now use the “new” keyword to create an address object with the two functions “setStreet” and “getStreet” baked in

var addr = new Address("Elm Street");

If I want to, I can change the street name in the address object

addr.setStreet("Old Berry Vale");

This is all good. However, there are some problems with this approach.

This and that

If you come from OO languages like C# or Java, you are tempted to believe that the “this” keyword refers to the current instance of the Address class. However that’s not always true. The “this” keyword actually refers to the context in which the function is invoked. If somebody for some reason invokes the setStreet function in a different contex you will get into trouble. This usually happens when we pass around function pointers. I can use the "call" function on the "setStreet" function to illustrate this by calling the function "setStreet" in the context of an empty object ({})

addr.setStreet("Old Berry Vale"); 
areEqual("Old Berry Vale", addr.getStreet()); //nothing strange here

addr.setStreet.call({}, "Cotton Acres"); //calling method from different context
areEqual("Old Berry Vale", addr.getStreet());  //hey.. the name didn't change

Are you protecting your privates ?

Well, are you?

addr._street = "Pleasant Island Stead"; //I can change internal state
areEqual("Pleasant Island Stead", addr.getStreet());

Ai ai ai. The “_street” instance variable isn't really private. Not having private instance variable when you create objects like this is problematic.

Closures. The road to a better approach.

From the description above you might believe that JavaScript is troublesome. However, as with any sharp tool, you need to use it right if you don’t want to get hurt.

JavaScript has the fine notion of closures. And yes, now it’s time to concentrate:

function main(foo) {

  runFunction(sayHello);

  function sayHello() {
    alert(foo);
  }
}

function runFunction(callback) {
  callback();
}

main("Hello world"); //=> “Hello world”

In JavaScript you can define functions within functions. This might seem a little bit strange, but you get used to it. The ”runFunction” is straight forward. It takes a function pointer as an argument, and executes that function . The “main” function is more fun, but still pretty simple. It just passes the “sayHello” function to the “runFunction” function.

The interesting thing here is that when the “runFunction” invokes the “sayHello” function, the sayHello function knows the value of “foo”. How can this be? The answer is that even after you pass the “sayHello” function it still has access to the parent scope where it was defined. The “sayHello” function is a closure. Cool huh ?

Another thing to mention here is that the “sayHello” function is private within the “main” function.

A better approach

By using closures and a couple of additional tricks you can create “objects” that doesn’t have the problems outlined in the previous solution. This is how you do it:

var Address = function (street) {

  function setStreet(s) {
    street = s;
  }

  function getStreet() {
    return street;
  }

  //explicitly make private methods public
  return {
    getStreet:getStreet,
    setStreet:setStreet
  };

}

You can now create an “instance” of the “object” this way:

var addr = Address("Elm Street");

Notice we are not using the “new” keyword, and we are not using “prototype”. We are basically calling the Address function which returns an object containing the getStreet and the setStreet function. The “street” variable acts as an instance variable. The function getStreet and setStreet has access to this variable since they are closures.

The setStreet and getStreet functions are initially private so we need to return them in an object to make them public. The act of making functions public is explicit, which I think is a good thing.

We get the expected behaviour:

var addr = Address("Elm Street");
addr.setStreet("Old Berry Vale");
areEqual("Old Berry Vale", addr.getStreet());

addr.setStreet.call({}, "Cotton Acres"); //different context doesn't create problems
areEqual("Cotton Acres", addr.getStreet());

addr.street = "Pleasant Island Stead"; //private variables cannot be tampered with
areEqual("Cotton Acres", addr.getStreet());

Good good. One other thing to notice is that the code is now visually much cleaner. Well, that’s it for now. [UPDATE] Look at the “yahoo javascript module pattern” if you want to learn more [/UPDATE]

A small note on methods and functions

In mathematics a function always returns the same value on the same input. In this post I have used the word in a more loose fashion because the of the “function” keyword in JavaScript. Methods and functions are not the same. Just so you know :)

Don’t document your code

Did I get your attention? Good.

To many of my fellow programmers what I am about to say is second nature. However, taking a recent discussion into consideration it is not second nature to everybody.

I don’t really think my point of view is very controversial. It basically goes like this:

Write comments in your code if it adds value

Obviously this is a very vague guideline. When does a comment really add value?

Comments on classes

In well written object oriented software your objects play one or more roles. I find it valuable to add a comment to the class declaration explaining the objects role in the system. This helps you see the big picture, which usually cannot be described by a simple class name. Sometimes it can.

It’s easier to explain my attitude towards writing comments by going thru the cases where I don’t write comments, as opposed to talking about the ones where I do.

Comments on and inside methods

In my opinion you should usually not comment your methods and the code inside them. Public APIs is an exception.

In well written code a method should only have one or two responsibilities. This means that by carefully selecting a method name you can give the reader a good idea of what is going on inside the method. Now, obviously you cannot explain the nitty gritty details, but that is what code is for. If you want the details, you should read the code. Method level comments cannot really add value. Putting a short description of what the method does in the comment is redundant because the method name is already doing this. Putting details in the comment gives you a maintenance hell when you need to keep the comment in sync with your code. If you want the details you better read the code.

Inside a method your often feel the urge to insert some comments to make the code more readable. Why don’t you just make the actual code readable instead? If you have a section of code that is doing some complicated calculation you can always put the calculation in its own method and give that method a meaningful name. Problem fixed.

Obviously there are some exceptions to my above comments. They are few. If you find yourself commenting a method or code inside a method usually it is a problem with your code. Comments are often a code smell.

Another problem with comments is the fact that you need to maintain them as you change your code. Up to date comments is not a big problem. Out of date comments is a big one.

When it comes to the title of this post, it’s not completely accurate.Today I’m pretending I’m running a tabloid.

(Please see my comments in the context of good old OOD. They might not be totally suitable for functional programming)

Limiting work in progress

In software development we often end up with local optimization. Developers focus on getting the development work done, testers focus on getting the testing done, and the analysis team focus on getting their work done. This is a problem, because local optimizations does not optimize the whole.

The development process should be optimized for global productivity. Work should flow thru the development pipeline without batches of work accumulating within the pipeline.

In the following diagram imagine the work flowing from left to right. First it enters the analysis team, then it’s developed, then tested, then finally it is delivered to the customer. A work item is indicated using the letter “x”.

----- Flow ------->

Analysis	Development	Testing	Delivered to customer
xx	xx	xxxxxxxx	xx

----- Flow ------->

This shows a situation where the testers are a accumulating work because they cannot process all the work the developers are handing over. Notice every team is working with top efficiency. This way of working introduces problems.

Problems caused by large batches

What are these problems we are creating ? If you look at the entire development process as a queue of work going from left to right, you will see that more work in process creates a longer queue. A longer queue means that the time it takes for one item of work to pass thru the entire queue is longer. This means that the delivery time of our software increases ! When work in progress increases, the delivery time increases. This is obviously a problem for many reasons that I will not go into right now.

There are other problems with the above work flow as well. Developers continue developing before their previous work is validated by the testing team. This means that they are creating work based on possible false assumptions. When this propagates thru the system it creates even more problems.

Limit the work in progress

The key is to limit the work in progress. In the previous scenario the testing stage of the software development process is the bottleneck. To avoid the accumulation of work in progress we need to increase the thruput in the test team, or lower the rate at which work is arriving. One solution could be to move some of the developers over to the testing team. Another would be to slow down the work upstream (to the left) of the test team, by giving the craftsmen there some “slack”.

Having people not doing the work they should be doing is often a problem for managers. They measure success by how efficient their team is. Suddenly we are back to local optimization. Having slack is actually a good thing. In the right enviroment it creates opportunities for learning and innovation.

In rigid functional teams, it is hard to do anything to modify the flow. People are happy with local optimization. In cross-functional teams it is easier to modify the process to accommodate the flow.

Some additional thoughts

1) It is easy to think of work in an software organisation as small items of work flowing thru the pipeline. In reality the work items varies in size as they flow thru the pipe. Also the issue of rework puts a little brake on the happy path. Keep this in mind as you take your next steps to improve the flow :)

2) There are things to be said about having the testing team upstream from the development team and making the batches of work smaller. That’s something for another post.

3) Focusing only on the work in progress withouth thinking about keeping the team members happy from a social point of view is a good way to fail in the long term.

Feel free to comment :)

Inspired by:
Kanban Chalk-Talk (http://vimeo.com/7814701)

The path to frequent deployments

Yesterday I was at a .NET User Group meeting starring Kent Beck. He did a presentation on the different considerations you need to make when increasing the speed of deployment.

Most companies deploy their software on a yearly to quarterly basis. The trend is going towards shorter intervals. Why would you deploy on a more regular basis? The reason is somewhat obvious. It makes the business much more flexible. If you can deploy faster, you can respond faster to the changing business requirements. Also I believe that companies that are ably to deploy on short intervals, are companies with a small amount of overhead.

Moving from annual to quarterly deployments

Add: Automated acceptance tests, Refactoring, Continuous Integration, and Subscription based model

When moving from annual to quarterly deployments you need to add automated acceptance tests. Making software ready for production by manual acceptance tests takes too much time (if you do it the right way). If you move to quarterly deployments, the time it takes to manually test the software will easily overshadow the development time. However if you are doing annual deployments the manual testing is not an overwhelming part of the development effort, and you can live with it.

You need to practice code refactoring at all times when developing for quarterly deployments. In the business world the term refactoring is (mis)used to mean changing the entire code base in one go. This is not what I mean by refactoring. Refactoring is changing the structure of the code one small step at a time. This is a continuous effort that ensures that the code base always is optimized for changeability.

The above considerations are technical. From a business perspective you can consider moving to a subscription based model for your software. If your software is released on a annual basis you would send your sales team out to sell “the big upgrade”. This model does not work as well if you are deploying every three months. Moving to a subscription based model makes more sense. This model also ensures a steady income.

Moving from quarterly to monthly deployments

Add: Developer testing, “Stand up meetings”, “Cards on a wall” and Pay per use

Remove: QA Department, Multiple deployed versions, Design document, Change request, Analysis team and the Build team

When moving to monthly deployments the handoffs between the developers and the QA department starts to become a bottleneck. You need developer testing to prevent bugs moving further down the production line. You could still have a QA team, but this team is much closer to the development team, and the software they are testing is of a much higher quality.

The communication bandwidth needs to increase between the team members. This can be done by adding visual aids like cards on the wall and by frequent short meetings.

Also you need to remove the heavy weight handovers that occurs between the developer team, the test team, the build team and the analysis team. The overhead caused by these handovers will start, again, to overwhelm the development effort. The team needs to be cross functional.

Having multiple versions of your software deployed will easily become maintenance problem. You cannot afford this, so you must have a smooth upgrade process for your software. This will ensure that you only have one version deployed at all times.

From a business perspective you have now the ability to introduce the “pay per use” model. With shorter deployment cycles you have the ability to adjust and change your software as the revenue varies. With longer deployment cycles you are stuck with your software longer and a “pay per use” model would not work very well because it takes a long time to change and adapt.

Moving from monthly to weekly deployments

Add: Live 2-way data migrations, Defect zero, Short lived branches, Keystoning, Kanban, Bootstrap financing

Remove: Test team, One-way data migration, Release branch, Patches, Up-front usability design, Venture capital

When moving to weekly deployments you are possibly deploying 52 times per year. This means that there is a good chance that some of the deployments will fail. You need to be able to roll out and roll back your changes automatically. Doing this manually is too much effort. In other words, you need, among other things 2-way data migrations.

Defect rates must be close to zero. Due to the sort deployment intervals, smaller defects can be fixed in the next deployment. Ideally they don’t occur at all.

Having code that live for weeks on a separate branch is a no-go. The cost of merging is too high. Branches must be short and sweet and usually merged into the main branch during the day.

Weekly iterations will not work. You need to be able to continuously change the priority of the work and work in a continuous manner. This can be done by applying Kanban. Developers will pull a task from the task list when the previous task is completed. There is no hard limit to the amount of work that should be done within a week. Developing the product is a smooth continuous effort.

Note: The idea of Kanban is much more involved than what I have outlined here. Do the Google dance if you want to know more. I strongly recommend this approach for developing software.

When deploying weekly and possibly using a subscription based model you are much closer to the revenue stream. For this reason alone this approach if very tempting to bootstrapped businesses.

Moving from weekly to daily deployments

Add: Immunization, A/B Testing

Remove: Staging, Operations team, Stand-up meetings

Is it getting hot in here? Did you just say daily deployments? Well. Yes :)

Immunization is the idea of automating rollout and rollback by using metrics. For companies with a constant revenue stream you can monitor the money flow on the servers you have upgraded. If you see drops in performance you automatically roll back to the previous version. There are also other metrics you can look at, e.g. the response time of the server.

Daily deployments enable a high level of A/B testing. By deploying different versions of your interface and monitoring the performance of each version, you can easily find the better solution.

“Inside the building there are only opinions”

…you need to test your software our there to really know what is the best solution.

Considerations

If you look at what is happening as you move to a shorter deployment cycle, you will see that all we do is removing friction. An organization that is only able to deploy on a yearly basis has a lot of waste in the system.

Moving to a shorter deployment cycle means learning new technical skills and having organizational backing. If your sales team is selling “big-upgrades”, having weekly deployments makes no sense. The technical skills also need to improve. The old software needs to support the new technical practices. This is not always trivial. You should move slowly to the next level, and not plunge into it.

I will finish this with a somewhat unrelated Kent Beck gem

“Your relationship with your customer is your most important asset”

Continuous integration with Team City and MSBuild

I am a firm believer that everything that can be easily automated should be.
Making a human responsible for repetitive IT work is not only inhumane, but also extremely inefficient and a waste of money.

Lately I have been working with automating deployment of new applications within our company. Deploying automatically to our customers is not within reach (yet).

The only thing that should be required for a developer in order to show his latest work to the rest of the company is to check in the code. The tests run and the application(s) should be deployed immediately after. A deployment package should also be created.

We are using TeamCity as our continuous build server, and it's doing a great job. The easiest way for us to build our code continuously, is to use the "sln2008" build runner. This means that TeamCity pulls the code from our code repository (Subversion) and compiles it. In addition we have set up the build runner to run our NUnit tests as well.

If we want to create an IIS application and deploy our code to it, we need to go for another option. Some common ones in .NET are NAnt, MSBuild, Rake and probably a lot more.

As a hardcore .NET guy my choice fell on MSBuild. I need to get started somewhere right?

An introduction to MSBuild

MSBuild is not that magical. Running MSBuild is as simple as running MsBuild.exe on the command line together with the path of your MSBuild script. The MSBuild script is an XML file. Actually the Visual Studio project files are MSBuild scripts. When you compile in Visual Studio, MSBuild is runs in the background.

The structure of an MSBuild script is pretty simple. It consists of a project. Inside a project you can have targets and inside each target you have one or more tasks:

<Project>
	<Target>
		<Task></Task>
		..
	</Target>
	..
</Project>

A task is some action you want to perform, like creating a directory or deleting an IIS application.

A target is just a collection of tasks to be executed sequentially.

A project is just a wrapper for a set of targets.
The following build script will write a “hello world" message when you run it on the command line using "msbuild fooscript.msbuild"

<Project DefaultTargets="FooTarget">
	<Target Name="FooTarget">
		<Message Text="Hello world" />
	</Target>
</Project>

Notice that the project defines "DefaultTargets", which is a list of targets to run. Inside the FooTarget there is only a single task "Message" that will output "Hello world" to the console.

You can make multiple targets and let them run depending on the success of the previous target.

<Project DefaultTargets="FooTarget">
	<Target Name="FooTarget" DependsOnTargets="FiiTarget">
		<Message Text="Hello world" />
	</Target>
	<Target Name="FiiTarget" >
		<Message Text="Hello world" />
	</Target>
</Project>

In the script above the default target is FooTarget, but since FooTarget depends on FiiTarget, FiiTarget will run before FooTarget. If FiiTarget succeeds, which it probably will, the FooTarget will run.

In MsBuild scripts you often have the need for defining variables that you can use later in the script. A variable is referred to using the $-sign. You can also define lists which are referred to using the @-sign. The following example demonstrates use of a variable:

<Project DefaultTargets="FooTarget">
	<PropertyGroup>
    		<FooVariable>Hello world</FooVariable>	
	</PropertyGroup>
	
	<Target Name="FooTarget">
		<Message Text="$(FooVariable)" />
	</Target>
</Project>

The above script will output "Hello World" (again) :)

Sometimes you don't want to put all of your targets in a single file. Maybe you want to use targets defined by third parties, like Microsoft or open source. I have used open source targets from tigris.org (http://msbuildtasks.tigris.org/). They have targets for copying files over FTP and editing XML-files. Both are valuable for automatic deployment.
To use targets defined in other files, you import them using the import statement:

<Project DefaultTargets="FooTarget">
	<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
	
	<Target Name="FooTarget">
		<FtpUpload .... />
	</Target>
</Project>

Here I import the MSBuild file located at C:\Program Files (x86)\MSBuild\MSBuildCommunityTasks. MSBuild uses the built in variable $(MSBuildExtensionsPath) to find the MSBuild root directory. By doing this I make the task "FtpUpload" available to my script. The built-in Microsoft tasks defined here (http://msdn.microsoft.com/en-us/library/7z253716(v=VS.90).aspx) are always available.

If you are moving a script from one machine to another, you need to make sure that you also move the imported scripts. Usually you can find these in the MSBuild directory. In my case it is C:\Program Files (x86)\MSBuild.

Now we are ready to look at how I actually implemented our MSBuild script to automatically deploy our web applications.

Creating a continuous integration script using MSBuild.

To deploy a web site you basically need to build it, create an IIS application and deploy the application to the physical location of the web site. In addition you would run your tests using the NUnit task, but I will not go into that here.

To create the build script for my web site I installed “Web Deployment Projects” for Visual Studio. You can add this project to you Visual Studio solution the same way you add any other project. (http://www.microsoft.com/downloads/details.aspx?familyId=0AA30AE8-C73B-4BDD-BB1B-FE697256C459&displaylang=en)

The web deployment project creates a .wsproj file which is actually an MSBuild script. Surprise. This script is configured to build the web application. I have removed a lot of noise from the script:

<Project DefaultTargets="Build">
  <PropertyGroup>...<PropertyGroup>
 
  <Import Project="$(MSBuildExtensionsPath)\Microsoft\WebDeployment\v9.0\Microsoft.WebDeployment.targets"/>

  <Target Name="BeforeBuild"> </Target>
  <Target Name="BeforeMerge"> </Target>
  <Target Name="AfterMerge"></Target>
   <Target Name="AfterBuild">...</ Target>
</Project>

Notice that the default target is “Build”, but “Build” is not actually defined in this script. The “Build” target is actually defined in the imported script Microsoft.WebDeployment.targets. This script was put in the MSBuild folder when I installed the Web Deployment Project. It is the “Build” target that is responsible for building my web site. I will not go into the details of that script here. Suffice to say that Microsoft has done a good job. Notice however that when the “Build” task finishes, the final web site is put in the “./source” folder. In an MSBuild script it is common practice to operate with relative paths. This causes the least amount of hassle.

Next step is to deploy the web site to the IIS server. I did this by putting a set of tasks into the “AfterBuild” target in the .wsproj file. (Same as above, but with some details removed and some added)

<Project ...>
   <PropertyGroup>
      <Customer>$(FOO_CUSTOMER)</Customer>
      <DeploymentDirectory>C:\ContinuousIntegration\$( Customer)</DeploymentDirectory>
      <WebSite>Default Web Site</WebSite>
      <AppName>$(Customer)</AppName>
      <FullAppName>$(WebSite)/$(AppName)</FullAppName>
  </PropertyGroup>  

....
  <Target Name="AfterBuild">
   <ItemGroup>
       <File Include=".\source\**\*.*" />
    </ItemGroup>

    <RemoveDir Directories="$(DeploymentDirectory)" />
    <MakeDir Directories="$(DeploymentDirectory)"/>

    <Copy 
      SourceFiles="@(File)" 
      DestinationFolder="$(DeploymentDirectory)\%(RecursiveDir)">
    </Copy>

     <Exec Command="appcmd delete app /app.name:"$(FullAppName)"" ContinueOnError="true"></Exec>
    <Exec Command="appcmd add app /site.name:"$(WebSite)" /path:/$(AppName) /physicalPath:$(DeploymentDirectory)"></Exec>
    <Exec Command="appcmd set app /app.name:"$(FullAppName)" /applicationPool:FooAppPool"></Exec>
   </ Target>
</Project>

First I copy the compiled files over to the deployment directory, making sure I remove previous files. Then I use the Exec task to run the command line tool “AppCmd”. AppCmd is used to create an IIS 7 application pointing to the same physical location as the compiled files.

Well that’s it. I have compiled the web site, moved the web site to a deployment directory and created an IIS application pointing at the web site.

A side note

I have omitted talking about the problems I had figuring out the difference between compiling a Web Application and a Web Site. Not sure I have the big picture on this question yet. It seems like you easily can build a Web Application Project using only a MSBuild task (within MSBuild)

<MSBuild
      Projects="$(ProjectFile)"
      StopOnFirstFailure="true"
      Targets="ResolveReferences;_CopyWebApplication;_BuiltWebOutputGroupOutput"
 Properties="WarningLevel=0;DefineConstants=TRACE;Configuration=Debug;Platform=AnyCPU;Optimize=true;DebugSymbols=false;OverwriteReadOnlyFiles=true;WebProjectOutputDir=$(DeploymentDir)\;OutDir=$(DeploymentDir)\bin\;DocumentationFile=;"/>

That's it. For now. In a later post I guess I will talk about getting the script to run in TeamCity.

My fresh new blog

This weekend I have published my new blog. Before my blog was hosted by Tumblr, but as a true programmer I needed to write my own. I have focused on making the blog as feature-free as possible. I have no admin interface, I just use Windows Live Writer. For comments I use Disqus.

The blog is running on my own virutal server hosted by ISPHuset ( in Norway). Having my own web server with the flexibility it brings is great.

The reason for writing a new blog framework is learning. At work you are confined to work with the tools that get the job done. When writing your own software you can experiment.

I decided to try out Autofac for dependency injection, Db4o as a storage and Moq as mocking framework. Usually I would use Structuremap, NHibernate and RhinoMocks.

Autofac

Autofac is ok. It has all the features that I usually use in StrutureMap. The only problem with Autofac is the documentation. Going from version 1 to 2 Autofac has changed its entire API and its conventions. Googeling for help is not easy, as you never know if what your read is version 1 or 2. Documentation is a problem with most OS projects and there should be a better way. I have been playing with the thought of creating a web site that can deal with this issue. All in all, I give Autofac a 4/6 ;)

Db40

I have not worked a lot with Document or Object databases before, so working with Db4o was interesting.

Getting started is very easy:

_db = Db4oEmbedded.OpenFile(Db4oEmbedded.NewConfiguration(), fileName);
_db.Store(myObject);

That’s all you have to do as long as you have a reference to the Db4o dll. Easy schmeasy. I did run into some problems. Consider having a Post object with a list of Tags. So you insert all your posts into the Db and now you want a list of all tags without pulling out every single post in the database. Suddenly you realize you are not dealing with a relational database anymore.

Luckily Db4o has Linq support, so you can query your object without pulling them out of the database. In my case I created a Tag object and asked the database for a unique list of Tags. I’m no master of Db4o so the solution I found might not be the best one.

Using Db4o is very easy in the simple cases, but things easily gets tricky when dealing with relationships. For me to use Db4o effectivly in the future I need to learn more about it, and frankly, I don’t think I want to spend the time. I have more faith in document databases, so maybe I will try out Mongo next time.

Moq

I did’t get to use Moc a whole lot. My first impression is that Moq has a much cleaner interface compared to Rhino Mocks. I think I will be using Moq in the future.

Continous deployment is great

The smartest thing I have done is to set up a TeamCity server at home. Everytime I check my code into GitHub, TeamCity get’s my changes and deploys them automatically to the Blog. Continous deployment should be mandatory for everybody developing software. Now, you don’t nessecarily need to deploy the changes directly to your customer, but at least you should do in-house deployment. Every single repetitive development task that can be automated should be. The reason ? It saves money and makes programming more fun !

Problems and bugs

Most of the challenges I faced when developing the blog was related to porting the data from the previous blog. I needed to support the Tumblr url schema /Post/[PostId]/[Slug], and also my new one /Post/[Slug]. Also I needed to move my images and make sure that the comment integration continued working.

Making sure that all the RSS feeds continued working was also a challange. Some of you might have noticed that I suddenly released over 30 posts in Google reader. Not sure how I would prevent that behaviour. Hope it didn’t cause any problems.

Bjarte – the designer

I’m not really a web designer, but I have done some design work in the past. I focused on making the reading experience nice and I hope I have done a good job. When It comes to the rest of the design I am pretty happy with it considering the fact that I’m a developer :) I know the design is different between IE and Chrome b.c. I have deliberately used some CSS 3 features that is not supported everywhere. Also the site doesn’t look good on 1024*800 monitors, and monitors with low contrast. Can’t please them all :)

I have tried to make the HTML as clean as possible by relying on CSS to do most of the work, and letting the HTML be responsible for structure only. In the future I might spice up the site with some nifty JQuery action, but right now I’m content with maintaining a small feature set.

That’s it for now. See you later :)

Sometimes you are better off without comments

        /// <summary>
        /// Gets the constant value.
        /// </summary>
        /// <param name="exp">The exp.</param>
        /// <returns></returns>
        public static object GetConstantValue(this Expression exp){
           ...
        }

I found this code in an open source project. It kinda looks like a joke, but it’s not.

Spot dirty code without reading it

Sometimes you can identify bad code without reading it at all. If you have big arrows forming in your code you should reconsider your solution. It usually indicates a high cyclomatic complexity (i.e unredable code). Can you see the arrow ?

A little nasty one from the belly of the beast

I was just skimming thru the ASP.NET MVC code and found this little nasty guy.

You should never widen your interface to enable unit testing. Doing so is a code smell, and there is (almost) always a better way.

 
[SuppressMessage(
    "Microsoft.Usage", 
    "CA2227:CollectionPropertiesShouldBeReadOnly",
    Justification = "Property is settable so that the dictionary can be provided for unit testing purposes.")]
        protected internal ModelBinderDictionary Binders {
            get {
                if (_binders == null) {
                    _binders = ModelBinders.Binders;
                }
                return _binders;
            }
            set {
                _binders = value;
            }
        }

Linq for javascript in one minute

I do quite a lot of javascript and C# programming. Linq is one of the greatest things in C# and I was missing it in javascript, so I figured I could just implement what I needed. It took about one minute. Here is how I could improve some of my code:

var a =[];
for (var i = 0; i < elements.length; i++) {
    a[a.length] = {
        name: elements[i].name,
        id: elements[i].id
    }
};
return a;

becomes

return elements.select(function(item) {
    return {
        Name: item.Name,
        Id: item.Id
    };
});

and

var res = [];
for (var i = 0; i < ids.length; i++) {
     for (var j = 0; j < this._currentList.length; j++) {
           if (ids[i] == this._currentList[j].Id)
               res[res.length] = this._currentList[j];
     }
}
return res;

becomes

return this._currentList.where(function(item) {
    return ids.contains(item.Id);
});

The actual implementation I call Blinq (e.g.) Bjarte Linq :p

if (Array.prototype.where != undefined)
    throw new Error("Duplicatte method definition on Array");

Array.prototype.where = function(match) {
    var a = [];
    for (var i = 0; i < this.length; i++) {
        if (match(this[i])) {
            a[a.length] = this[i];
        }
    }
    return a;
}

if (Array.prototype.select != undefined)
    throw new Error("Duplicatte method definition on Array");

Array.prototype.select = function(selection) {
    var a = [];
    for (var i = 0; i < this.length; i++) {
        a[a.length] = selection(this[i]);
    }
    return a;
}

if (Array.prototype.contains != undefined)
    throw new Error("Duplicatte method definition on Array");

Array.prototype.contains = function(obj) {
    var len = this.length;
    for (var i = 0; i < len; i++) {
        if (this[i] === obj) { return true; }
    }
    return false;
};

Not alot of code really.

Nitpickers corner: LINQ-like implementations of Javascript already exists, but when it takes me a minute to implement I don’t go about learning a new framework to get the job done :)

Strongly typed Action Links, URLs and Redirect To Action in ASP.NET MVC 2 RC 2

This one goes out to all you who are spending valuable time to figure out where the heck you can find these methods:

ActionLink and BuildUrlFromExpression you can find in the MVC Futures:

<%@ Import Namespace="Microsoft.Web.Mvc"%>  
<% =Html.ActionLink<HomeController>(x => x.Index(),"Go home") %> 
<% =Html.BuildUrlFromExpression<HomeController>(x=>x.Index())%>

Strongly typed RedirectToAction you can find in MvcContrib

using MvcContrib; 
return this.RedirectToAction(x=>x.Index());

You can also read Phil Haacks uber lame exuse for not putting this inside MVC at the end of this post

Apple and Usablility

I am very impressed with the user interfaces Apple has on it’s products like iPod and the iPhone. Apple really do set the standard. Apple does have some really crappy software as well. The one that constantly annoys me is iTunes. But that’s not a problem Apple. I can forgive you for one mistake. However, what I got this morning when I tried to update my iTunes didn’t please me at all. As we all know when we get an software update we just press next, next and then we’re OK. Apple has realized this as well and is deliberately trying to push some software to you that you DO NOT WANT. This is not acceptable.

The problem is easy to fix. They could just have left the check-boxes unchecked.

Obviously I’m not new to this behaviour. A lot of non-user friendly companies does exactly the same thing. But Apple? Come on. You are supposed to be the usability guys.

The way to create great software is to think about the user experience from the users perspective. Nobody wants an extra browser or a mobile-something application installed when they are just updating their software.

Hopefully you, my readers, don’t do this kind of crap to your customers. Or do you ?

Later.

Code ownership and source control in action

We (the team) just had a chat about some ideas we should follow when it comes to source control and code ownership. Here is a short summary.

Code ownership

”Shared code ownership: ” To ensure that more than one person is familiar with a certain part of the code base we have shared code ownership. This will make the team more flexible, and making change easier. It is always allowed to refactor code to make it better. It is actually required to prevent the code base from deteriorating over time. That said, there might be people who knows more about the code base than you do. Also, somebody else might be working with the code right now. ”Consult before making radical change.” Communication is always important.

Check in

”Check in early, check in often: ” The code is what is in source control, not what is on your computer. By keeping the code in isolation on your computer you are setting yourself (and the team) up for big trouble. By updating(first) and checking in (second) often, you prevent merging and communication problems. Partition your current work into sub problems, and check in when each feature is created or updated. Work on one problem at a time. If you find yourself working on 10 features, where each one is impossible to check in, you are doing it wrong. It’s a code management anti-pattern. The philosophy behind this is simple. Problems are easier to fix right after they occur. In a day, or a week, solving the problem becomes exponentially harder. Considering this philosophy, keeping the code on your machine to avoid problems MAKES NO SENSE !

”If the code isn’t checked into source control, it doesn’t exist”

Write tests

Write tests for the expected behaviour and your usage of the code. This will ensure that new features and refactoring is compatible with existing usage.

Standard

“Standard” is what we do right now (or should do). The standard can be changed as we find better ways to solve our problems. However, we should all follow the current standard.

Feedback is king

The other day when I was up in the mountains in my cabin, I suddenly started thinking about development processes and rapid feedback loops. I sat down to write about it, and here is the result:

There is a saying “culture eats strategy for breakfast”. It’s true. I have seen it in action.

Introducing a new process often requires changing the culture in the company. This is hard or impossible. Radical change creates too much friction. Working with the existing process and modifying it slowly is the most feasible route. Identify your goals and see how you can slowly modify your existing process to achieve the goals.

Let assume that building software faster and with better quality is a good idea in your context. It usually is. Let’s say this is our goal, assuming that it in the end will lead to more profitability.

I firmly believe that to create maintainable software there is one term that is more powerful than all others. That term is rapid feedback loops (RFL). Rapid feedback loops is the common denominator of all agile processes. The only process I know of where RFL is not a central concept, is the waterfall model. It is not the details of the agile processes that make them succeed. It is the fact that they all embrace rapid feedback.

Test driven development tells us every minute if our code works. Continuous integration tells if our code is working in the bigger picture. Integration tests checks whether we are fulfilling the product requirements. Periodical demonstrations and conversations with our customers tell us if we are building the right product. It’s all about getting feedback. If you are writing bad code, or creating the wrong features, feedback nudges you back on the right track. Design reviews can prevent architectural erosion. You need feedback as soon as possible so that you don’t waste time working on the wrong thing. Working on the wrong thing is expensive.

Feedback is a good thing if the feedback you are getting is correct. If it’s not you have a problem. We can usually trust well written unit tests. Integration tests are harder, because they are only correct if they are actually testing a feature we need in the end product. We need to constantly check with our customer that we are on the right track. The customer might be wrong, so we re-check on a continuous basis.

If you want to improve your current process you should start looking at your feedback loops. Slowly introduce feedback on all levels. You don’t need to pick Scrum, XP, RUP, Lean, Kanban, Crystal Clear or ScrumBut. However, you should know roughly what they are about. This way you can pick from a toolbox of practices and make them fit into your existing culture.

I don’t believe in picking an out of the box process if it will crash with your current company culture. However I really do believe in feedback loops and you should introduce them slowly into your current process. There is obliviously more to creating a smooth software machine, but feedback is a key concept.

Peace.

Software related podcasts

Here is my ITunes export of podcasts I listen to. Most are software or “software business” related. I hope it’s helpful. Have I missed some crucial ones ? Tell me on twitter or in the comments :)

BTW, I just removed the StackOverflow podcast from the list. It was taking “brain dead” to another level :)

.NET Rocks!
Agile Toolkit Podcast
Elegant Code » CodeCast
Entrepreneurial Thought Leaders
Hanselminutes
Herding Code
IEEE Software’s “On Architecture” with Grady Booch
IEEE Talks Software Process
Lean Agile Straight Talk podcast
Mixergy - Online Business Tips from Successful Entrepreneurs » Interview
Pluralcast
Polymorphic Podcast
Software Engineering Radio - the podcast for professional software developers
TEDTalks (audio)
The Startup Success Podcast
This Week In Google
This Week in Startups - Audio Only
this WEEK in TECH - MP3 Edition
Venture Voice

Crush it !

I just finished the book Crush It! by Gary Vaynerchuck. It’s about how you can make a living using social media and doing what you love to do. In the book he summarizes my own philosphy in a couple sentences

Love your family
Live your passion

Can’t argue with that…

I give it a 4 out of 6. If I had finished it a year ago I might have given it a 5 or 6. It all depends on what kind of books, blogs and shows you have devoured before you read it. To me it wasn’t a lot of mind blowing stuff in there. Still, it’s worth reading due to the passion Gary brings to the subject. Get the audio book, and you will see what I mean :)

Freaky coincidence

Just sat down to draw some vectorized mountaintops in Illustrator to practice my design skills. I turned on Spotify and the first song started. It was Jeremy by Pearl Jam. It starts

“At home drawing pictures of mountaintops…”

Freaky :p

An abstraction on top of NHibernate

Prepare for a random thought > 140 chars:

I just read Ayendes post where he debates why we should not put an abstraction on top of NHibernate.

The reason to do such a thing would be to avoid coupling your code to the persistence layer. This in itself is not a bad idea until you take a deeper look at what is required to make a useful abstraction.

A simple persistence interface on top of NHibernate would make it easy to change the persistence layer but

What happens when you need other features from NHibernate. Will you add it to the abstraction layer ? Not a very solid abstraction really.
NHibernate is already an abstraction on top a relational database. Do you want to add another one ?

The term “Leaky abstraction” comes to mind.

Taking your game to the next level

Experience is the best way to learn, especially if you fail. However, you should not be reinventing the wheel and making mistakes people have made before you. Use the knowledge on the web for what it is worth, and stand on the shoulders of giants as you take the next step to become a better developer.

I have been learning new things about the business of software, project management and software architecture. I have read a lot of books, listened to hours and hours of pod casts, watched screen casts and been to conferences. Finding the best content among all available sources is hard. In this post I will share some highlights hoping that I can help you in your search for the best content.

The business of software

I was listening to the StackOverflow podcast when they were interviewing Jason Calacanis. I had never heard about this guy but he was apparently an internet celebrity and serial entrepreneur in the web space. He had just started a show “This week in startups” and I checked it out.

In the show he is interviewing guests on how you start up and run a successful internet business. He has a very interesting personality which makes this show a must watch. I am listening to many shows, like “This week in tech”, “This week in Google”, “Herding Code”, “Elegant code cast”, “Software engineering radio”, “HanselMinutes”, “StackOverflow” and many others, but this show is always the highlight of the week.

After I started watching this program I am really getting a taste for the business side of things. Turns out it is as complicated and fun as creating software. You need to be passionate, hard working and even a little bit smart to be a good entrepreneur.

Lately I have been reading all sorts of books on the subject. One of the most intriguing books I have read lately is “The purple cow” by Seth Godin. He talks about how you need to create remarkable products that people want to talk about. The traditional model of marketing is not working anymore because people is getting overloaded with information and It’s hard or even impossible to make your product stand out. Read the book or get it at audible.com. It is worth it. Audible, by the way, is great.

Right now I am reading a book called “Predictably irrational”. People act very irrationally when it comes to money. The funny thing about it is that you can actually predict their irrational behavior. In this book the author gives a unique insight to the human psyche. It’s fun to read. I recommend this one as well.

Software design and architecture
Software design and architecture has been one of my interests for a long time and this year is no different. The Norwegian Developer Conference was packed with great speakers presenting on parallel tracks. Despite having a choice between great speakers I always found myself listening to Udi Dahans presentations. Udi talks about SOA not only from a technical perspective but in the intersection between business and software. He has a great way of keeping things simple but not simplistic.

I have been following Greg Youngs blog on Code Better for quite some while, and was thrilled when I got the opportunity to attend his class on the CQRS architecture in Bergen. The CQRS architecture works great together with domain driven design to create and manage complex domain models. It’s worth reading up on.

The best software design book I have read this year is “Patterns of Enterprise Application Architecture”. The material is presented in a clear and concise manner and the book is a great read. No wonder it has cult status in the developer community.

Another great book is the free book “Getting real” by 37 Signals. It talks about every aspect of developing web software. The big takeaway from reading this one is “Keep it simple”. It’s separated into several small essays. Reading an essay takes 5 minutes and every one contains gems of knowledge.

The process of developing software
Thru my work I have seen many interesting takes on how you should develop software, some good, some bad. There is no “one true way” of developing software. I believe that your choice of process greatly depends on the culture in your company and the physical location of the people involved. Distributed teams have many limitations and should be avoided if possible.

Agile and Lean software development is interesting. I have been reading the Poppendieck books on Lean and some of the more business oriented books of Womack and Jones. However the highlight of the year is “The Goal”. It is written as a novella which makes the subject even more interesting. This book introduces “the theory of constraints” which is a very interesting concept, and a “must know” for every developer (and business person).

Seeing the big picture
Learning about business, process and software gives synergies. Each of the areas affects the others and you start seeing the bigger picture. This will make you a better developer, or as in my case a great one ;)

The roles of software development

Building software requires people with a certain skill set. To build a successful product you need a person with a clear vision of where the product is going. This person is the most important member of the team. Without him (or her) you will not be successful. This person, the champion, can guide the development towards the goal. The champion knows what features to include and what features not to include in the product. His clear vision will stop feature creep and lengthy unproductive discussions. The champion listens to others but has the final word. Democracy is not the way to go if you want to build remarkable software.

Developing a quality product is difficult, and a champion is only one piece of the puzzle. To build a great software product, you need great software. You won’t fool anybody by putting lipstick on a pig. This is a key aspect missed by a lot of companies.

To build great software you need a person who knows how to build software. It sounds obvious doesn’t it ? This person, let’s call him a developer lead, should be responsible for the development team. He is responsible for ensuring that sound software engineering practices are followed throughout the project and that the software quality is up to par. This is not a simple job.

Ideally the champion and the developer lead is the same person. The reason is simple: Communication. Communicating the vision is hard, and fewer people means simpler communication. If you want to develop software effectively you need to make trade-offs between the features you would like to have and the time it takes to implement them. Often you can make small adjustments to your requirements and gain a lot in terms of development time and better technical solutions. This translates to faster time to marked and eventually more revenue. Identifying these trade-offs are difficult, but if the champion knows the technical aspects of software development this problem disappears.

Having one person doing the work of a champion and a developer lead can be a recipe for failure if this one person fails at any of the two roles. A common mistake is to put a domain expert to the task of leading the development team, and thinking that if you put enough developers on the team the software will build itself.

I have seen many products become mediocre due to the lack of a champion or a dev. lead. The roles need not be explicit, but they need to be present in the team in some form.

What kind of people do you think is required to develop remarkable software ?

HTML 5 looks nice

I just had a look at HTML 5 the other day. The Chrome browser has support for a lot of the features and you can check them out here if you have Chrome installed. To me it looks like Silverlight and Flash can get in trouble in the future. The first question that hit me was how long time it will take before the majority of the browsers support it. According to Microsoft it might take 5-10 years, but I’m not so sure about that considering the speed at which the web is developing.

Check it out at http://www.chromeexperiments.com/

A simple exception handling strategy

Time to scribble down some of the things I don’t like seeing when reading C# code.

try {
    something();
}
catch(Exception e){
    //doNothing
}

and

try {
    something();
}
catch(Exception e){
    logSomething();
    //do nothing
}

and

try {
    something();
}
catch(Exception e){
    logSomething();
    return null;
}

and

try {
    something();
}
catch(Exception e){
    //throw base Exception class
    throw new Exception("Something didn't happen");
}

All of these are exception handling anti-patterns and the reason should be obvious. Still I see them all the time. I might even have written some code like this back in the days.

One reason not to write software this way is that your system will have side effects that are impossible to spot. Also, I’m guessing the users of the software won’t like strange things happening in the UI.

To avoid problems like this I’m usually using a combination of custom exceptions and a back stop. When I see the possibility of an exception occuring in my code and it makes sense to create a more specific exception I do just that. Remember to keep the original exception as an inner exception of the new one. To prevent exceptions from bubbling onto the user interface I have a back stop where I catch all exceptions before they are exposed to the user. Since I am throwing custom exceptions in my code I can reason about the exceptions and give the user the proper feedback. Sometimes the exceptions require more attention than just giving feedback to the user. In these cases my exception might become an event that is picket up by the appropriate handler.

I guess this is the common way to handle exceptions, but who knows. Do you have any other strategy ?

Cucumber and the Gherkin language

If you are able to read this entire post I will owe you a beer ;)

The other day I was at a Norwegian .NET User Group (NNUG) meeting in Bergen. Aslak Hellesøy was showing off his Cucumber tool. I will give a uber quick recap skipping a lot of detail and correct terminology.

Cumcumber is a mature BDD testing framework written in Ruby. When using Cucumber you describe your features in English (or the language of your choice). Cucumber will parse the feature and generate test templates (a.k.a. step definitions) that the developer will be completing.

Feature

# language: en
Feature: Division
  In order to avoid silly mistakes
  Cashiers must be able to calculate a fraction
 
  Scenario: Regular numbers
    Given I have entered 3 into the calculator
    And I have entered 2 into the calculator
    When I press divide
    Then the result should be 1.5 on the screen

The language used to describe a certain feature must follow a set of structural rules. The language is developed for Cucumber and is called Gherkin.

Step definitions

Given a feature you would use Cucumber to generate a template for writing your tests, called step definitions.

Before do
  #todo
end
 
After do
end
 
Given /I have entered (\d+) into the calculator/ do |n|
  #todo
end
 
When /I press (\w+)/ do |op|
  #todo
end
 
Then /the result should be (.*) on the screen/ do |result|
  #todo
end

This example is in Ruby but you can generate the steps definitions in other languages as well. Some languages seems to have good support, others not so much.

When you have generated your step definitions and filled out the missing code you can run the feature and see if your code is meeting the business specifications.

Cucumber is pretty sweet and it’s a shame it’s not mature enough for usage with the .NET platform. Also there is to much friction installing it and getting it to run properly.

Digging deeper -Parsing

I’m not the kind of guy that needs to know every technology to feel complete. There are just way to much out there. When it comes to theory around software I don’t like to be out in the dark. Things like patterns, architecture, processes and the business surrounding the software is very interesting to me.

The question that caught my attention in the Cucumber presentation was

How do you take a feature and build test templates from it?

One thing is for sure, you are not using “find and replace” :) I felt a gaping hole in my knowledge and needed to do some research. I was not interested in the technical aspects, but the theory behind it. I often like to start at the bottom and work my way up. I knew I had to do some reading about parsing, but I did not quite know where to begin. After some googeling I found some nasty papers. I quickly realized I was missing the required background so I needed to find my old book from the university called “Introduction to the theory of computation” by Michael Sipser. The book is theoretical but still not very hard to read for someone who has read similar books before. I am now about to give a summary in plain English of my research, so that you don’t have to read the book.

Warning: I might be skipping some details and simplifying.

Languages

A language is basically a set of strings. A given string can be in the language or not. Looking back at Cucumber you can say that a feature (the string) is in the Gherkin language if it complies with the rules of the Gherkin language. But how do you describee these rules that decide if a given string is in the language or not? Also, if the feature is in the language, how do you break it down into it’s components so that you can reason about the semantics,i.e the meaning, of the feature ?

There are different ways to describe languages. Depending on your method of choice, you are able describe a language or not. Let me clarify.

Regular languages

A regular language is a language that can be described by using a regular expressions.

If you want to describe the language consisting of all strings starting with “a”, then continuing with any number of “b”s and ending with an “a”, i.e the set {aba, abba, abbba, ..}, you can describe this language with the regular expression “ab+a”.

Now you might think that you can describe any set of strings using regular expressions, but that is not the case. The language consisting of all string with ”a”s and continuing with the same number of “b”s cannot be described by a regular expression. (More formally you would describe this language as B={aⁿbⁿ | n>=0})

Context free languages

Context free languages are a superset of regular languages. Some of these languages cannot be described using regular expressions. You describe these languages using a set of rules (also known as productions). Let’s look at the following rules.

A => 0A1
A => B
B => #

For a given rule, the left side of the arrow contains a variable (upper case letter). The right side contains a string of variables and determinants. In this case the determinants are 0,1 and # and the variables are A and B.

To build a string in the language using these rules follow the following steps:

Write down the start variable. It is the variable on the left-hand side of the top rule unless specified otherwise.
Find a variable that is written down and a rule that starts with that variable. Replace the written down variable with the right-hand side of that rule.
Repeat step 2 until no variables remain.

Using the grammar above you can generate the string 000#111. The generation would go something like this, starting with the first rule.

A => 0A1 => 00A11 => 000A111 => 000B111 => 000#111

Actually using these rules you could generate any string starting with any number of zeros, then a hash sign, and then ending with the same number of ones. This language cannot be described using a regular expression.

A parse tree

Building the string 000#111 above can be represented as a parse tree:

A parse tree is decomposing a string in the language into it’s separate components. Now it starts getting interesting. If I somehow could create a parse tree from a feature in the Gherkin language I could do all kinds of sexy stuff with it. Cool!

Ambiguity

The problem with context free languages is that you in some cases can have different parse-trees giving you the same string in the language. In other words applying the rules in a different order will give you the same string. If I am using the parse tree to reason about the meaning of the string, two different parse tree would give two different meanings to the same string. Not a good thing.

Parsing expression grammar

A grammar is another word for the rules describing the language. I.e a grammar is a set of rules. A parsing expression grammar (PEG) is very similar to the context free grammars I described above. The big difference is that the languages described using a PEG are not ambiguous. For a given string in a language there is only one parse tree. Sweeeeeeeeeeet. Now we’re talking :) (And no, I’m not a nerd)

It turns out that Gherkin is described using a PEG. Cool huh? In ruby there is a tool called TreeTop that takes the rules of the grammer as input. If you give it a string from the language, i.e a feature in this case, you will be able to generate a parse tree for this string.

This is the point I am at in my research currently. I have also found a .NET tool that is able to create a parse tree of a given string if you supply the rules to it. My next step is to steal the rules of the Gherkin language and move them over to this tool, and see If I am able to build a parse tree for a Cucumber feature using this tool. That would be cool.

Also I need to look at the theory of how you actually build a parse tree for a string. Not by starting with the rules and building a string, but the other way around.

Did this make sense ? Am I on the right track ? Please tell :)

Till next time.

Set based validation in the CQRS Architecture

Today we’ll get our hands dirty.

If you are following the CQRS design pattern you might run into trouble when it comes to set based validation. Consider the task of registering a new user. How would you in your domain figure out if that user name is unique? You can obviously not query your event store, because the event store is not built for that purpose.

Querying the reporting database
The solution is actually querying the reporting database on the read side to figure out if the user name is unique. There is (in my opinion) a little problem with this approach. By querying this way you are making the read side responsible for knowing domain concerns. You are effectively making the read-side a part of the domains bounded context and adding more responsibilities to it. Before we introduced set based validation the separation was clean and nice. It is not that clean any more.

You could keep the design clean by having a separate database with the only purpose of answering set based queries and keep this database within the bounded context of the domain. This would keep the design cleaner (perhaps), but it would not be a practical choice in most scenarios.

Another oops
In a scalable architecture you are publishing events asynchronously. They are picked up by the read-side and applied to the reporting database thru the denormalizer. Consider the situation where the UserCreatedEvent(UserName=”BjartN”) is in the denormalizer queue, and the domain is instructed to add yet another user named “BjartN”. The reporting database knows northing of the first event resulting in the domain publishing the same event twice. When the second event hits the reporting database the entire thing blows up, because the user already exists. What to do next ? The event has already occurred (twice) and has been published to many other systems. Basically the events have occurred and you need to deal with it. The only way to fix it is to issue an compensating action. This could be issued by the denormalizer to the domain as a command.

Be consistent if you can
Dealing with these kinds of concurrency issues is not something you want to do for fun. If you have the option to publish the event and save to the reporting database in a single transaction (e.g. synchronously) you should consider that.

Why would you store the entire history of the domain ?

I got this great question in one of my comments:

Modeling the entire history of the domain seems to only be valuable if the domain requires it as part of your domain? I guess the biggest concern with a pattern like this would be how you would effectively deal with different versions of code that have existed at different points along the path. The report one would generate from 10 releases ago might not be the same if you are reapplying events as you go. This could certainly have business implications too, how do you suggest dealing with this?

If the the application of an event changes, do you simply create new events and never modify old ones?

I will try to answer some of it:

Storing the entire history
It’s not so much the domain that usually requires you to keep the entire history, but you keep it for auditing. If you only keep your current state in your database, how do you know that the data is correct ? Anything could have happened to keep it from being correct, like bugs and evil people :) With an add-only model you even have the possibility of writing the events to a write-once-storage. Now it is impossible to manipulate the data. If you write a bug that creates false events, you would later execute compensating actions to undo what you did. Remember that an event is something that has happened (past tense) so it makes no sense to change it in any way.

There is another good reason to keep the history as well. When saving events you are not saving data to your current data model, but you are actually storing all the user behaviour. If you only store the current state of the domain, the reporting you can do on this data is limited. The information stored in an add-only model is much richer. You can do reports on things you didn’t event think of when you created the application.

Versioning
If your events changes you would create a new version of that event, and keep the old ones. To keep your domain code form being bloated with handling of all versions of events you would basically introduce a component that converts your events from previous to newer versions, and then apply them on the domain. Remember that events are things that actually happened in your domain so in most cases the information in deprecated events are valuable.

That said, there are of course situations where you would choose different architectural patterns.

Creating an event storage

The event storage is basically where you persist all the events your domain is publishing. The only thing that makes it a little bit tricky to implement is the fact that you need to worry about concurrency. Concurrency is tricky, but not that tricky.

Event storages can be created using various technologies. According to Greg the best way is to write the events onto the disc, circumventing the file system. Of course then you’ll need to write your own indexing as well. That’s just crazy stuff and I’m not digging into those APIs any time soon :p In this example I’ll be showing an event storage for a sql database. We need to be able to store our serialized events and keep track of what aggregate root published the event. Also we need to maintain the concurrency version of the aggregate root.

Database

First let’s look at the database schema

EventProviders

EventProviderId (uniqueidentifier)
Type (nvarchar)
VersionNumber (int)

Events

FkEventProviderId (uniqueidentifier)
DateTime (datetime)
Data (varbinary)

The name “event provider” refers in this case to the aggregate root. Each event provider has an id, a type name and a version number. The version number is used to do optimistic locking, making sure we don’t update an aggregate root that has been changed in the meantime. The version number also serves another task; the version number is always equal to the number of events published by the aggregate root.

Now let’s look at the events table. Each event has a reference to the event provider that published the event. We also store a serialized version of the event and the time it was persisted. Notice we have no “eventId”. We don’t need one.

Event storage

We will access the event storage using the IEventStorage interface.

public interface IEventStorage
{
    IEnumerable GetAllEventsForEventProvider(Guid id);
    void Save(IEventProvider provider);
}

To save an aggregate root we pass the a reference to the IEventProvider interface (implemented by the aggregate root) to the event storage.

public interface IEventProvider
  {
    IEnumerable GetChanges();
    void ClearChanges();
    Guid Id { get; set; }
    int Version { get; set; }
  }

The Save method

To save the IEventProvider (i.e. Aggregate Root) we need to do the following

Start transaction
- Get the event provider from the database
- If the event provider does not exist in the database, create it
- If the event provider exists, check that the database version matches the current version of the aggregate root. If not throw a concurrency exception.
- Save all events the event provider has published after previous version
- Update the version number in the event provider. The version number is now the previous version number plus the number of events published since previous version.
End transaction

The implementation is pretty straight forward. As you see all the complexity in this operation comes from the fact that we need to check for concurrency violations.

Included project

I am including a project with an implementation of the event store and some other things like the repository and the aggregate root base class. This was written quickly at the DDD Course by me and some other peeps, and is far from production ready and it has no UI. However I think it’s better to post this (crappy) code than to post nothing at all. The idea is that you can have some code to look at. There are some integration tests you can run if you set up the database. If you want to use an implementation like this, you can get some speed optimizations by using stored procedures. It’s evil, but still.

Some things to keep in mind

I have not talked about implementing snapshots to improve performance but it is really not that hard. I would use the Memento Pattern and not store the actual aggregate root. From there on it is just plankekjøring ;) Mark Nijhof is publishing an extensive example at some point in the near or far future.

Creating a Microsoft Surface application for Norwegian Television

I had a great conversation with MVP Einar Ingebrigtsen. He was the one who invited me to Wave, and I figured why not use Wave to do an interview on his latest project. He has been working on an application for a Norwegian television show called “De Ukjente”. It is an WPF application for Microsoft Surface. In this interview he talks about all the challenges he faced creating this application.

It is apparent that Einar is a developer with a taste for developing quality software. It really shines thru in this interview.

First a couple of “teaser” screenshots

The first one is for the actual application, and the next ones are from the admin console

You can even watch the show here

..and then the interview

1) What problem does your software solve ?

The television program was a gameshow, and their alternative was to have physical items on a table for the interaction. The purpose of the gameshow is to match known professions with unknown persons in studio. The application was used to solve this interaction, and more. The administrator could send clues to the tables that was displayed per face when touched. In addition there was a monitor application that the gameshow host used to monitor the interaction.

2) What did you use to develop this software ? WPF?

The project was originally not targetting surface, but Windows 7 multitouch and HP Touchsmart desktop machines, so we started out in WPF 4. But 2 weeks before we were supposed to be finnished, we got hold of 2 surface tables and decided to go for Surface. We then needed to rewrite the touch logic, which was fairly simple since I had pretty much abstracted that away. And we ended up doing a WPF 3.5 SP1 solution using the Surface SDK 1.0.

3) How did you develop the user interface ? Is it all your work, or did you do it togheter with a designer ?

The television channel provided the graphics elements in Adobe Illustrator format - they controlled the look. I did not do any graphics myself, but was in control of the layout of things and had some “artistic freedom”.

4) Assuming you used XAML. Did you write the raw XAML or did you use Blend ?

Handwritten in Xaml for the most part - only did a couple of storyboards in Blend. I feel a lot more productive in Xaml. When I first started doing Xaml based development, I started in Blend, but after å while I ended up feeling a lot more productive hand-writing Xaml.

5) Did you use any design patterns to shape the user interface. MVVM?

The entire solution was built around MVVM and was 100% test driven all the way from the backend to the UI. All in all the solution consists of 7-10 views, all with different viewmodels, some user controls with their own viewmodels and data from a database with an abstraction layer built on top of DevExpress Persistent Objects. It uses commands for handling behavior. It was crucial to go with test driven and use well known patterns, as this was to be on television and we needed to have control of code quality right from the start.

6) Did you have any challanges related to the fact that you developed for Microsoft Surface ?

My main challenge was the lack of hands on with an actual surface during the main development. I was pretty much doing this blindfolded. Had a chance the last day of development to sit down with the actual surfacetables and tweak things. The simulator worked out pretty good for me though. The challenges in the project was more related to other technical difficulties, such as concurrency, realtime data transfers between all the clients and such.

7 ) Did you have any bugs in production, that is in the TV show ?

Bugs - what do you mean? :) Well, the first show we had a couple of issues. The entire solution was using UDP broadcasting to let all the clients communicate with each other, they shared this code. That all worked out great, only thing was I left some experimental code in the receiverthread that allocated 10MB of data everytime an UDP packet was received, needless to say - OutOfMemoryException occured a couple of times. But thanks to the fact I did it all TDD, we got lucky and didn’t have much bugs and the ones that occured was really easy to get fixed and tested. After the second show I never heard back from the TV channel, which is a good thing. Only thing I heard from them was that they wanted to do a nother program with surface in it, so look out on NRK2 at the end of October, there will be a program called Tekno with a WebBrowser created for Surface that can be rotated and scaled and moved around on the table. Also something I will be putting up a blogpost on how to do, btw.

8) I have been wathcing a couple of episodes of the show, and it seems the participants are having some problems moving items on the screen. I’m guessing this has nothing to do with you, and everything to do with the nature of MS Surface

Both yes and no. We had some issues with lighting in the studio. The technology behind surface uses cameras within the table that captures the interaction on the table, if one is not careful with the lighting it will think that shadows on the table are interaction. That being said, we had a bug we didn’t have time to fix - the snap mechanism didn’t take into consideration the scaling of the pictures being moved. So in some cases the faces did not snap. It is not that apparent on the show, since we prepped the contestants to be aware of this. But some times you can actually see this problem. You can also see they have some problems moving things at all, and this is because of lighting in the studio.

People and software design

I my last posts I have been talking about DDD and the CQRS Architecture. There are certainly situations where the CQRS Architecture would be a good solution form a theoretical point of view. I know from experience that rolling the best suitable architecture is not always the way to go. We need to consider the people as well. It is really dependent on the culture in the company if a “new” architecture can be rolled out.

Company culture

In some companies being a craftsman is valued. You are working in an environment where creating unreadable code will get you in trouble and major design decisions are well thought thru. There’s process, there’s design and there’s coding. The goal is to create code that can be maintained over time and at the same time meeting the needs of the business.

In other companies there is chaos. The recognition that software needs to be built with care does not exist. The goal is to create the next feature without any thought of the maintenance hell you will experience in the future. The developer who implements the next feature in the shortest amount of time is the one who gets rewarded. Spending time creating SOLID software with well proven development practices is not valued.

I’m painting a black and white picture here, and as we all know, the world is grey :) Now, let me continue my rant.

Both these companies can grow and gain marked share. If you are in a marked where you have monopoly, or are one among a few you can keep delivering mediocre software and still be in business. If you are in the “chaos company” category you will start seeing problems when competitors starts entering your marked or when you just can’t deliver more features in reasonable time. Suddenly you are dealing with crappy software, unmotivated developers and a bad business. Not a good place to be if you are in the business of making software.

The best design decicions are not always the best design decisions.

Back to my original thought. Yes. I was planning to advice you not to apply the CQRS Architecture, or any architecture that requires a certain amount of thought in companies following the chaos model. You will probably cause more problems than you solve. You need to be able to control your environment to roll these kinds of solutions. Eric Evans talks about creating bubbles of quality software in your organization. These kinds of bubbles burst quickly if you are in the wrong company. In other companies a bubble is a viable solution.

Introducing new ideas in a company must be done with care and the first thing you need to consider is the people and the culture, not the best design from a technical perspective.

…and that was todays rant. Makes sense ? Hit me with a comment :)

Implementing an aggregate root

This post is related to the following ones

In one of my previous post I explained the testing benefits of the CQS architecture.

I arrange my tests by replaying a set of events.
I act by executing a set of commands.
And I assert by comparing the events published by the aggregate root to the events I expected to be published.

I am doing Act-Arrange-Assert testing in the simplest most readable way and at the same time creating business value.

In this post I will look at how I would implement my aggregate root to enable this kind of testing. What we need to look at is how the aggregate root implements the replaying of events. This is the key to loading the aggregate root into it’s current state.

The base class

In addition to managing it’s domain logic each aggregate root is responsible for

Keeping list of all the events that the aggregate root has published after it was loaded.
Maintaining a mapping between events and the methods for applying the events to the aggregate root.

When saving an aggregate root to the event store you would ask for the events that has been published since it was loaded. That is why we need to store this information. In a testing scenario it is also useful to be able to assert against the events published by the aggregate. For each event, the aggregate root has a method for applying that event. A dictionary of these event-method pairs is maintained by the aggregate root. These responsibilities are common for all aggregate roots, so we will put it in a base class. Before I elaborate on this base class (called HandlesEvents) lets look at an example.

The aggregate root

The aggregate root has a behaviour (i.e. method) that corresponds to a command. Given a ChangeDescriptionCommand coming from the UI you would typically have a ChangeDescription method on the aggregate root. In the command handler you would map a command to a behaviour on the aggregate root.

The implementation of a given behaviour is the key here. In a behaviour you would

validate the input,
create an event
and call a method that applies the event to the aggregate root.

The ChangeDescription behaviour implementation would look something like this:

public class Foo:HandelsEvents {

  private string _description;
  
  public Foo(){
    RegisterHandler<ChangeDescriptionEvent>(applyChangeDescripionEvent);
  } 
      
  public void ChangeDescription(string description){
    if(description == null)
      throw new InvalidDescriptionException();

    ApplyEvent(new ChangeDescriptionEvent(description));
  }

  private applyChangeDescripionEvent(ChangeDescriptionEvent e) {
    _description = e.Description;
  }
}

When the ChangeDecription behaviour is called, the input is validated, the event is created and applied to the aggregate root. Notice I’m am calling the applyChangeDescriptionEvent method thru the base class in stead of calling it directly. This is to enable the base class to do some bookkeeping on what events are called and how they are applied to the aggregate root. Notice that we register the mapping between the event and the method in the constructor. This way the base class knows which method to call when applying the event, both when you are loading an aggregate root into it’s current state and when your applying the event thru a behaviour.

More on the base class

Now lets look at the base class HandlesEvents. The HandlesEvents base class is here for convenience. If I want to replay a set for events I will only need to call the method LoadFromHistory(IEnumerable<IEvent> history) on the HandlesEvent class. The HandlesEvent class will then get the aggregate root in the required state by applying the events to the appropriate methods.

public void LoadFromHistory(IEnumerable<IEvent> history)
{
  foreach(var e in history)
      ApplyEvent(e,false);
}

The ApplyEvent method finds the given handler method for the event and call this method with the event as an parameter. The boolean parameter false is an instruction to the ApplyEvent method not to keep a record of this event. We are only interested in the events that happend after the aggregate root was put in its current state. The ApplyEvent in the HandlesEvent base class will look something like this.

protected void ApplyEvent(IEvent e, bool add)
{
  AppliesEvent handler;

  if(!_lookUp.TryGetValue(e.GetType(), out handler))
    throw new HandlerNotFoundException();
  
  handler(e);

  if(add) _events.Add(e);
}

The HandelsEvents base class will also have a public method GetChanges to expose all the events that where published after the aggregate root was loaded into it’s initial state.

So lets sum up what we’ve got here.

You can load a set of events into your aggregate root to get it to the required state
You can invoke a behaviour on the aggregate root
You can get a list of events that the aggregate root has published

You are ready to start testing.

Should I go deeper into this stuff ? Was this post just confusing ? Is this English ? Hit me with a comment :)

Using the same model for reporting and transactions

I got this question in my comments section. I’ll try to answer it the best I can, but I might be leaving out good arguments. Who knows..

Why cannot a single model serve reporting, transactions and searching at the same time? I guess it has something to do with scalability and performance of the application to do. What I have read about CQS this far I have not seen any list of all the benefits of it and I have not yet found the real arguments that I can use to convince the other guys in the team.

The main reason for applying the CQS architecture and DDD is to model complex behaviours in your organization. If your only goal is to put a set of forms on top of your relational database, this approach is not for you. In that case you would go with Active Record or Row Data gateway, or smack a MS Access application on top of your database. Also if you wan’t to utilize the CQS Architecture it’s best done in green field projects.

Behaviours and data

In object-oriented programming behaviours are the things the objects can do. If I wan’t to create a scalable object model of my (complex) domain I am interested in exposing the behaviour of my objects and encapsulating the internal state of the object (the data). I aim to keep my object seams as tight as possible. A seam in this context is the exposed interface of my object.

I will now try to explain the difference between exposing data and exposing behaviour.

Let consider the entity “Employee” in a Human Resources Management application. At some point you would like to increase the salary of the employee. In a typical system you would update the “Salary” property on the employee to reflect the changes. Obviously there was a reason for the employee to get his salary changed. Maybe he got promoted, maybe it was just the yearly correction, or maybe he finally completed his master thesis. Let assume the reason was that he finally completed his master thesis. In many companies (at least in Norway) the salary is partly based on education. To capture the new education you would update the “YearsOfEducation” property on the Employee as well. (Simplified, but still.)

The person who is managing the employees need to update different properties in the HRM application to complete his task. Maybe he even has a written note of all the things he has to do in the system in order to update the education of the employee:

In a behaviour centric system this behavior would have been captured by exposing an AddEducation(..) behaviour on the employee. This behaviour (i.e. method) would be responsible for handling all the things that happens when an Employee adds on his education, like getting his salary increased. It would even publish an event so that the salary application could pick up the changes in the employees status.

We want to expose behaviours not data. As a consequence

Setters (i.e public settable properties) are a domain anti-pattern

Since I am interested in modelling behaviours in my domain and keeping a maintainable object model I do not wan’t to expose the data the behaviours is operating upon. This would lead to hard times when I need to change the implementation of my behaviors. As a consequence

Getters are a domain anti-pattern.

To summarize: Do not expose data on your domain model

…and that is the very short answer to why you would have one model for reporting and one for transactions on your domain.

This is the best resource on the web for really getting why you would model behaviour in your application (and yes, I have checked the entire internet): http://herdingcode.com/?p=189

Performance:

I am hoping to go deeper into consistency, partitioning and availability in an upcoming post. Still, what is interesting about the CQS Architecture is that in many cases you can fire off your commands on the domain asynchronously. Fire and forget. You don’t need to wait for the entire processing to have occurred before returning control to your user.

I hope this answers at least some of your questions Anders.

Testing the domain

One of the most powerful features of the CQS Architecture is the ability to create highly expressive tests.

When a set of commands are executed on an aggregate root the expected result is one or more events.

Before you execute your commands you must get your aggregate root in the required state, in other words you need to set up the context of the test. Loading an aggregate root into the required state is done by replaying a set of events.

Do you see the power of this way of testing ? You can express your test only using events and commands ! Yes. It deserves an exclamation mark. A specification/test would look something like this:

Given
UserCreatedEvent {Name=”Greggy”, Age=12, Address=”Bergen”}

When
ChangeUserNameCommand{Name=”Greg”}

Then
UserNameChangeEvent{Name=”Greg”}

OK, I’ll admit to the fact that this wasn’t a very good example :) Ignoring that, I’m hoping you see the power of writing tests this way. Look at how the language of the events in the past tense and the commands in the imperative form goes together with the “Given-When-Then” paradigm. It’s beautiful how things just fit together using BDD and DDD this way. Of course you would expect that from two so similar abbreviations. In case your were wondering, it was Greg who pointed this out :)

Creating the specifications this way fits well in with DDD idea of the ubiquitous language. You are really getting closer to the BDD dream of having your business people writing your specifications. Sweet.

Should I give some code examples perhaps?

Edit:

In the post I omitted the fact that it’s the command handlers in front of the aggregate root that is actually getting the commands and mapping them to behaviors in the aggregate root. It won’t change the post that much, but it’s important information.

Great DDD course with Greg Young.

I have been on a two day intensive DDD course with Greg Young. The course was great both from a social and educational perspective. To make sure I don’t forget all of what I’ve learned at the course I will try to write a couple of blog posts to remind myself, and perhaps give others some valuable information.

The majority of the course revolved around the Command Query Separation (CQS) architecture promoted by Greg in various presentations.

<EDIT>
We now call it command and query responsibility segregation CQRS :)
</EDIT>

http://www.infoq.com/presentations/greg-young-unshackle-qcon08
http://www.infoq.com/interviews/greg-young-ddd
http://www.vimeo.com/3171910

The CQS architecture follows the same principle as Bertrand Meyer’s Query Command Separation principle ( http://en.wikipedia.org/wiki/Command-query_separation ), but they are not the same.

Overview of the CQS architecture
One of the main ideas of the CQS architecture is keeping a clear separation between changing the state of the domain and querying . The key is that:

“A single model cannot be appropriate for reporting, searching and transactional data”

In box style the architecture it will look something like this:

The write side
The UI will query the read model to get the DTOs it wishes to display. In the UI the user will build up commands and send them to the domain. The commands will be in the imperative form like “ChangeDescriptionCommand”. The domain will process the commands and events. The events will be in the past tense like “DescriptionChangedEvent”. The language is important. It does not need to be a one-to-one relationship between the commands and the events as in the example I just gave.

The read side
On the “read” side the denormalizer subscribes to the events from the domain, and changes the model according to the events published by the domain. The read model is usually optimized for querying, and is often kept in 1. normal form. Depending on your non-functional requirements, the read model can be fully consistent or eventually consistent with the domain.

Add only model
The domain events are stored in the event store. The events are serialized and stored in the order the are published. The event store is important. Notice we have a add-only model. As opposed to the common CRUD paradigm we are only adding information. We are not updating or deleting anything in the event store. That means that we don’t loose any information. Consider a shopping cart. A user adds an item to the shopping cart, and then he removes it again. In a common CRUD scenario we would insert the item and then delete it. When inspecting the data we would see an empty shopping cart. The fact that it was added and then removed is lost. In a add-only model we would still have this information. For “the business” these things can be very valuable.

Of course in a CRUD model you could start recording this information and in a couple of months you could start giving “the business” reports about users adding and removing items. In an add-only model your could create a report of user behavior from the beginning of time. Do you see the difference ? Using an add only model you don’t loose information. You are capturing user behavior as opposed to just updating the data. It’s sweet.

Loading an aggregate root
Loading an aggregate root is a little bit different. Since you are not storing the current state of the domain, but the just the events that have happened you need to load up the aggregate root by supplying all the events that have occurred since the beginning of time and replaying these. This will obviously give an performance problem if you need to load 500 000 events to get your aggregate root to the current state. To solve this problem you can store a snapshot of the aggregate root at certain intervals. When loading the aggregate you load up the last snapshot of the aggregate and replay only the events that happened after the last snapshot.

Implementing the event storage and the snapshots is straight forward, but as in all systems you need to handle concurrency issues which makes it a little bit tricky.

To be continued..
I’m planning to make a series of blog posts digging deeper into various topics and looking at how you would solve some of the obvious problems you will run into using this architecture. I am hoping to cover the following topics, but no promises :)

Building up commands
Handling commands
Building up aggregate roots using the repository and event storage
Creating the event storage
Concurrency issues
Set based validation
Testing the domain in a BDD style
Testing the denormalizer
Disconnected clients
Contract first development
Enriching events and moving domain logic into independent components.

The subtleties of leaky abstractions.

I just read a great example of a leaky abstraction at Ayendes blog

public interface IQueue
{
    void Enqueue(T o);
    T Dequeue();
    bool IsEmpty { get; }
}

Ayende is using this abstraction to handle queuing. What if the queue gets so big it starts eating into your memory ? You would need to save the queue to the disk right ? Well, then suddenly T needs to serializable. But it does not end there. All objects that are referenced by T also need to be serializable. Suddenly you’ve got a big problem on your hand due to a leaky abstraction.

I love the subtleties of software design :)

What we've got where is a failure to communicate.

Shu-Ha-Ri is a Japanese martial arts concept describing the three stages of learning. When you start doing martial arts you learn by repeating the same techniques over and over. You are at a Shu level of learning. Later you start using other techniques and putting them together. You are reflecting over your techniques and figuring out what works for you. You are now at a Ha level. At some point it all comes natural to you and you don’t even think about what you are doing. It’s all reflex. You are at a Ri level.

The same concept can, with some imagination, be used in the software industry. Often I find myself in situations where I’m having problems explaining my points of view when it comes to technical decisions. This mostly occurs when I am talking to PMs (of course) and sometimes when I am talking to fellow developers.

When talking to business minded PMs the answer is always to focus on the business aspect of the problem. This is a good way to think about software in general by the way. You are in a business, and even if you think unit testing is the most important thing in the world, it is not. Money is. Luckily making money and making quality software usually coincides. (But not always)

Discussing technical practices together with my fellow developers is a whole other issue. Let me give you a contrived example:

In most cases I advocate using some kind of presentation pattern when developing a user interface. To me the reasons are pretty obvious; separation of concerns, testability, maintainability and so on. Sometimes you will find complex logic stuffed inside the code behind file with no separation of anything (Yes I’m talking about you Mr. ASP.NET Web Forms). Data access, business logic and user interaction logic all stuffed into a big pile of mud. This is where I step in and tell the developer that he should separate the code into several classes to keep the concerns separated and the code testable and maintainable. I continue my rant by explaining some of the intricacies of the different presentation patterns and how lovely test first development is. The developer will sit quietly and try to absorb all the information. The next time I look at the code nothing has changed. I got what I deserved.

Why?

I am talking to a Shu level developer as he was a Ha or Ri level developer. The attempt to communicate this way is futile. It is blatantly obvious that you need to communicate differently depending of the level the developer is at. Still I keep forgetting it. This post is to remind myself.

A little tip:

If you find yourself in an argument with a fellow developer and your crushing arguments seem to have no impact, consider that you might be on different levels. You might even be the Shu level person in the discussion. Well. No. That doesn’t make sense ;)

Do I really need to make my own blogging software?

Any developer will at some point get fed up with their blogging platform and start dreaming about writing their own. I got to this stage today when the tumblr text editor started crapping on my posts. It reformatted my nice html examples and started doing crazy shit. Also there is not a buildt in commenting feature in Tumblr which also causes some friction. The last thing that pissed me off is the fact that I cannot upload pictures.

What I need from a blogging platform is not much:

RSS feeds
Plain text editor. I know HTML and wan’t to write my posts using it. And I don’t want the editor to change my HTML.
Comments.
Uploading files.
Changing the theme.
Authentication for editors.

It’s very tempting to fire up Visual Studio and start making a sexy simple MVC blogging app for myself. But I just know It will consume way too much of my time. I’m in deep pain. Please comfort me.

The ugly green form

(The indentation got crapped on by the editor. Will fix it soon.)

If you work with javascript (+jquery), html and css and always seem to make overly complex and unmaintainable code this little rant is for you :)

The task

Imagine yourself getting the task of creating the most ugly registration form in the world. Let’s call it “the ugly green form”. I’m guessing you would create something like this guy here:

Name
Age

Even if you are creating the ugliest form in the world, you would not implement it using the markup below. Why? Because you care about the separation of concerns and you care about the people maintaining the code in the future.

Don’t do this

function save() {

var name = $(“#name”).val();

var age = $(“#age”).val();

//do something with the data

alert(name + ” - “ + age);

}

</script>

<tr>

<td>

Name

</td>

<td>

</td>

</tr>

<tr>

<td>

Age

</td>

<td>

</td>

</tr>

<td>

</td>

</tr>

</table>

We are not doing a good job of separating the concerns here. This page is doing too much. What ? Too much ? Oh yes! This piece of markup contains the structure, the styling and the behavior of the form. We need to split them up.

Heeeeeeeeeeeeeelp meeeeeeee!

The solution is simple. Put the css skinning in a css file. Put the javascript behavior in a javascript file. Put the html in html file. (Also fix up the HTML to use nice clean markup instead of overly verbose table markup.) You end up with something like this:

Html file

<div>

</div>

<div>

</div>

</div>

Javascript file

function init() {

$(“#submit”).click(save);

}

function save() {

var name = $(“#name”).val();

var age = $(“#age”).val();

//do something with the data

alert(name + ” - “ + age);

}

..and the CSS file

#form

{

background-color:Lime;

width:200px;

}

#form span

{

width:60px;

float:left;

}

Are we done ? Nope. We’re not. If we wan’t to test the behavior of a form like this, it would be nice to be able to separate the view (html) from the javascript completely so that the javascript code could be tested in isolation. I will write another post about this in the near future.

The difference between supervising controller and passive view makes my head spin

First let me supply some context by supplying you with the two reads leading to this rant:

Now over to my rant:

So say you are making a passive view with the task of registering a new user. The view might look like this:

    public interface IRegisterUserView {
	    string UserName{get;}
	    string Password{get;}
	    event Action OnSubmit;
    }

Now the presenter would look something like

  
    public class RegisterUserPresenter
    {
        private readonly IRegisterUserView _view;
        private readonly IUserService _service;

        public RegisterUserPresenter(IRegisterUserView view,  IUserService service)
        {
            _view = view;
            _service = service;

            _view.OnSubmit += submit;
        }

        void submit()
        {
            _service.CreateUser(_view.UserName,_view.Password);
        }
    }

This would be an implementation of the passive view. What if I now, instead of having a two properties UserName and Password in my view, created a whole new object and called it the “Model”

    public class Model
    {
        public string UserName { get; }
        public string Password { get; }
    }

And then I create a new view that looks like this. To make it even more superviser-controllerish I could remove the event and replace it with a method “Submit”

    public interface IRegisterUserView2
    {
        Model Model { get; }
        void Submit();
    }

Now the presenter would look something like this:

public class RegisterUserPresenter2
 {
        private readonly IRegisterUserView2 _view;
        private readonly IUserService _service;

        public RegisterUserPresenter2(IRegisterUserView2 view, IUserService service)
        {
            _view = view;
            _service = service;
        }

        public void Submit()
        {
            _service.CreateUser(_view.Model.UserName, _view.Model.Password);
        }
 }

The first example is an implementation the “passive view” pattern, and the other is an implementation of the “supervising controller”. The thing separating the two is that in the last example the view works intimately with the model. In the “passive view” example the view knows nothing about the model (which in this example is the IUserService).

But are we really talking about the same model here ? No. We are not. So my claim is that the difference between supervising controller and passive view is minimal. Maybe non existing.

I’m guessing the reason I am having trouble seeing the difference here is due to a lack of understanding of this pattern. I’ll be back with a new post when I finally figure out why the above rant is wrong.