A system is a set of interacting or interdependent components forming an integrated whole. In organizations, systems consist of people, structures, and processes that work together. In the software world a system might be a web site or a framework of interconnected components that work together.

To understand a system it is common to take an analytic approach where you decompose the system and gain understanding by understanding its parts.

An organization is typically split in to functional units such as sales, marketing and product development.

A web site might consist of a database, a front end layer and maybe integration with external services like Facebook.

By understanding the components of a system you can answer “how”-questions. For an organization you can answer questions like how does the organisation approach the customers, and how does the organization develop its products? In software you can answer questions like “how is the front-end talking to the database” and “how does the system integrate with the external service?”

By solely taking an analytic approach to a system you are bound to run into trouble. To understand a system you need to look at how the components of the system is interacting, and also how the system interacts with the larger system which it is a part of. A system is always a component in a larger system. Looking at the synergies that form a system is called synergetic thinking.

By taking a synergetic approach, you can answer “why”-questions. These are the more important questions.

In organization you can answer questions like “Why is the sales team only approaching customers thru cold calling?” and “Why do we have unhappy customers?”. You can answer these very important questions by looking at the system in which the organizations operates. The larger system consists of its customers, its competitors, and the legal system and so on.

For software products it’s the same. Why does Amazon feature the Kindle on the front page? Why does the Kindle ad take half of the valuable real estate on the front page? You can’t answer this question by looking at the components of the web site. You need to look at the larger system and see how the web site fits in this system.

Synergetic thinking is one of the corner stones in systems thinking. Toyota pioneered this thinking early on and applied it to the Toyota production system. As time went by they outperformed their competitors by working continuously with the system, ensuring that its components worked seamlessly together. This work was not limited to the Toyota system, but also the system as a component in a larger system. Toyota actively engaged their suppliers and dealerships to ensure synergies. Working this way is not easy. It requires a continuous focus.

When it comes to software we can see analogies of too much analytic thinking when applying the single responsibility principle. By focusing on the single responsibilities of the system we often lose the algorithm of the system. By the “algorithm” I mean the way the components work together to solve the problem. James O. Coplien and Trygve Reenskaug try to remedy this problem by introducing the concept of DCI (Data, Context and Interaction) in which the interaction between the parts is an explicit concept.

Organizations as systems and software as systems are different beast on different levels, but there similarities, as I’ve shown. I find this intriguing.

In large organizations an analytic approach is often seen. The different functions of the organisation are run by different people, each trying to optimize their part of the organisation. One of the corner stones of systems thinking is the claim that by optimising each part, you are not optimising the whole. There are numerous examples of this. Russel Ackoff likes to use the analogy of cars. Take the best car parts from all car manufactures and put them together to form the ultimate car. It will not work. They don’t work together. It’s not only the components of the car, but how they work together that matters most. The same goes for organizations.

Getting the parts of a organizations to work seamlessly together to create an optimal whole is something that few organizations explicitly work on. This is because analytic thinking dominates. It dominates the education system and it dominates organizations.

You might wonder if most organizations are working on 20% of their true capacity. Some say they do.

Thanks for listening as I organize my thoughts.

Later.

Comments F Share

This is a story about failure demand.

Failure demand is a work product that does not meet the customers need and generates additional work. It is opposed to value demand, which is the customer wanting something new.

A couple of months ago we moved into our new house. I didn’t take long before we started noticing small things like missing paint, wrong paint, and a shower that didn’t close properly and so on. We contacted the contractor and they sent people (sub contractors) to improve on the issues. The painter has been here four times now, and there are still issues with the painting.

In the bathroom they have mounted the lighting in the wrong place. This means they have to replace the tiles, and mount the light again. How do they solve this problem?

First they made an appointment with us to come inspect the problem, then they sent the plumber, which among other things, took down the light. Then they made a new appointment for the mason to come and remove the tile. He didn’t show up on the first appointment, so we rescheduled. The next day, he showed up. After taking down the tiles and messing up the bathroom, a carpenter came to fix the hole in the wall. Then the mason came back to put up the new tiles. Now we’re waiting for the plumber to come back and mount the lighting. The plumber does a little bit of the electrician work as you might have noticed. This is just one of several “work flows” I have seen in the process of getting my house up to par.

There are multiple problems with this way of working, as you’ve probably noticed.  The two big ones are unsatisfied customers and waste of money.

I don’t really like spending all my time waiting for people to show up for appointments. It kind of sucks. I don’t like cleaning up after the workers either.

Since we have already paid for the house, all this rework does not cost us anything (not, true but stick with me). The contractor needs to pay for all this rework.

My experience with the contractors is not unique. It’s seems to be the way they all operate. Handled properly the contractor could have made away with half the work, and in addition they would have had happy customers as well. Half the work you say? Well, then maybe the house would have been cheaper in the first place. So maybe I am loosing A LOT of money on this modus operandi.

What are the obvious issues with this workflow?

Inspection, inspection, inspection

In the process of taking over the house, we were invited to an “inspection”. We had a session with the supervisor, who wrote down all the obvious issues on a list. After the inspection the list was maybe 20-3o items plus.

This is a major issue. We, as laymen, were supposed to find all the problems in a short meeting. We found more than 20 without even trying! This means that the contractor had no plans to make a flawless house before we moved in. This leads me to believe the workflow is as follows:

1. Sell the house before it’s built
2. Make a half assed attempt to complete it
3. As the new house owner finds new problems repeat the following:
   1. Send a man to attempt to fix the problem in a half assed manner
   2. Send more men
   3. Repeat

This cannot be the best workflow.

Let’s attempt a better workflow:

1. Sell the house before it’s built (I guess we can’t do without this one)
2. Build the house while carefully and continuously verifying the quality of the work done
3. Have a thouroug final inspection
4. Let the new owners inspect the house, knowing there are no issues.
5. Take a cup of coffee

What is missing from the contractors workflow is careful and continuous inspection, and workers that take pride in their work. I guess the last one might be a problem, but the first one should be easily implemented.

It is not uncommon that work like this lacks proper inspections. I believe it has to do with the fact that you don’t get immediate feedback. Good work and inspections done right manifests  itself as nothing happening. No customer complaints. Who would like to spend a lot of time to make nothing happen. Seems counter intuitive. However, monetary gains and satisfied customers is the result you should strive for. So why not work better and inspect more ?

There is a phrase I like to use

You should build quality into your product. Not inspect it in later.

…but If inspection is your only option, do it all the time.

For my readers interested in software development, most of this applies to you as well

What’s next? Well, I am waiting for a sub contractor to come fix our leaky window

Comments F Share

When writing code I strongly believe in making interfaces between objects (and data structures) as small as possible. This makes life easier for both you and the people consuming your code.

The distinction between data strutures and objects is something only a few think about. To be clear: A data structure only carries data. An object carries state and behavior. This little rant is about data structures.

In C# it is common to create a data structure in the following way:

	
public class Product {
	public string Id {get;set;}
	public string Name {get;set;}		
	public Money Price {get;set;}
}

You would typically use this data structure to represent a product in your inventory. What you are actually saying (by having public setters) is that any piece of code can change the Product Id, Name or Price at any time. If your are lucky nobody actually does this, but hey, why make it hard on yourself?

If you are creating a data structure that is meant to be immutable, then make it immutable.

	
public class Product {
	public Product(string id, string name, Money price)
	{
        	Id = id;
            	Name = name;
		Price = price;
 	}

	public string Id {get; private set;}
	public string Name { get; private set; }
	public Money Price { get; private set; }
}

It's not much, but it makes your code more explicit by narrowing the interface of the data structure to represent what you intended. It might even get you out of trouble once in a while.

Any thougths ? Please share.

(Note: “Data structure” might be a poor choice of word here, since the term is often used to represent something slightly different in computer science.)

Comments F Share

The short story

You can get a fully working GDAL in .NET (x64) from here: https://bitbucket.org/bjartn/gdalgonewild 

For 32 bits windows you would need to do what I have done, but starting with the 32 bits C assemblies.

The somewhat longer story

Gdal is a translator library for raster geospatial data formats. It supports reading and writing many raster formats. One of them is GeoTiff, which is why I needed to use GDAL.

GDAL is written in C, but has bindings to many other languages thru SWIG. SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages.

For the .NET crowd this means that there are some .NET dll’s that talks to the  GDAL C assemblies thru P/Invoke.

The good news is that as a  .NET developer you don’t need to compile the entire GDAL library with the bindings yourself.  The bad news is that it is not straight forward to get it running in .NET (until now).

There are some different places you can get the assemblies needed. One of them is thru installing FWTools, another is by downloading from a build server http://vbkto.dyndns.org/sdk/ which is what I did.

Both these builds include many files you don’t need. What you *do need* is the dll’s and some data files for proj.4 and GDAL.

Setting up the environment the hard way

You also need to set up your environment variables so that GDAL knows where to look for the plugins, the data files and so on. The following environment variables need to be set correctly: PATH, GDAL_DATA,GDAL_DRIVER_PATH  and PROJ_LIB.

If you install FWTools the environment will be set up by running the “C:\Program Files (x86)\FWTools2.4.7\bin\setfwenv.bat”  in a command window. If you use the http://vbkto.dyndns.org/sdk/ build you can set up the environment using the “SDKShell.bat“ file.

Setting up the environment the easy way

We don’t want to set up the environment on every PC we are installing our software on. If we want to have an xcopy deployment of our .NET application we cannot rely on somebody manually setting the environment variables.

I have created a self contained Visual Studio project that runs GDAL without any external dependencies like the environment variables. You can download it at https://bitbucket.org/bjartn/gdalgonewild (x64 only). I created this project by picking and choosing the required DLLs and data files from a x64 build of GDAL found at http://vbkto.dyndns.org/sdk/.

I have created a class called GdalEnvironment that is responsible for setting up the GDAL environment. In addition I have added a “pre-build event” that copies all the required C and .NET assemblies into the bin directory when the application is compiled. This removes the need for setting up the PATH environment variable.

One little strange thing I noticed was that I was not able to set up the GDAL_DATA environment variable programmatically. So instead I used Gdal.SetConfigOption which overrides the environment variable.

Well. That’s all I can think of right now.

Comments F Share

I’m just finishing the book “Out of the Crisis” by W. Edwards Deming. The book is a treasure chest of ideas and thoughts. This one I found particularly interesting

"Experience without theory teaches nothing. In fact,
experience cannot be even recorded unless there is some
theory, however crude, that leads to a hypothesis and to a
system by which to catalog observations”

Source: Mind and the World Order. Clarence Lewis. 1956, Page 195.
Quoted in Out of the Crisis. Edward Deming. 1991. Page 317.

I wonder if this explains why developers with 10 or more years of development experience are still doing the same mistakes they did as junior developers. If you stop searching for knowledge, if you are no longer looking for theory to explain what is happening around you, you stop learning. After 10 years you are still a junior developer.

Comments F Share

Javascript is a special language with somewhat unusual features. Often you see people struggling with the concept of “this”, “new” and “prototype”. This is my attempt to shed some light on these mysterious concepts. If you are coming from a statically typed language like C# or Java, you will quickly realise that they are not what they seem to be.

This and that

A common mistake is to think that the “this” keyword in Javascript refers to the instance of the object you are in. This is the common use in C# and Java.

You can think of the “this” keyword as an implicit argument to the function. In a browser this implicit parameter will be set to the root object by default, e.g. the window object: 

function foo(){
	return this;
}
	
alert(foo()===window) //true

However, you can pass the “this” reference to the function you are calling by using the call (or apply) function:

	
function foo(){
	return this;
}
	
//calling foo with “this” assigned to the value “hello”
var result = foo.call(“hello”) 

alert(result) //”hello”

As you can see, the “this” keyword depends on the context in which you call the function foo.

The new keyword

To construct objects in Javascript you use the “new” keyword together with a constructor function. Functions that are used as a constructor usually starts with an upper case letter (best practice)

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = new Foo();

alert(obj.helloWorld()) //”Hello world”

When you call a function with the “new” keyword, a new empty object is created and then the constructor function is called with “this” set to the new object. In the example above, this will cause the new object to get the method “helloWorld” assigned to it. The prototype hierarchy is also set up, but I’ll come back to that later.

The following code demonstrates another way to get  the same effect as the new keyword (without the prototype hierarchy)

function foo(){
	this.helloWorld = function(){return "Hello world"}
}

//emulate "new"
var obj = {};
foo.call(obj);

alert(obj.helloWorld()) //"Hello world"

Forgetting the new keyword

If you “forget” to use the new keyword on a function that is meant to be used as a constructor you will get strange behaviour:

	
function Foo(){
	this.helloWorld = function(){return "Hello world"}
}
	
var obj = Foo();

alert(obj.helloWorld()) //fail
alert(window.helloWorld!==undefined) //true

You just assigned the function helloWorld to the window object. Probably not the intention :)

The prototype hierarchy

When you use the new keyword, the prototype hierarchy is set up. This means that the new object will have  the methods in the prototype object of the Foo function:

function Foo(){
	this.helloWorld = function(){return "Hello world"}
}

//define a method on the prototype
Foo.prototype.helloWorld2 = function(){return "Hello world 2"}


var obj = new Foo();

alert(obj.helloWorld()); //"Hello world"
alert(obj.helloWorld2()); //"Hello world 2"

The new object will also have methods defined higher up in the prototype hierarchy.

Well. That’s it for now.

Comments F Share

This post is just about me sharing some of the stuff I have looked at in the open source community lately. There might be some ruby stuff in here. First out is rack.

Rack

Imagine for a minute that there were a lot of different web servers out there like IIS,  Apache and perhaps even Kayak. Also imagine that there were a lot of different web frameworks out there, like Fubu MVC, ASP.Net Web Forms (don’t get me started) and Asp.Net MVC.

Wouldn’t it be cool if we could put any web server together with any web framework just by putting a common abstraction in between ?

The web frameworks would only need to implement an adapter working with the abstraction and the same for would go for the web server. To get n web frameworks working with m different web servers, we would only need to implement n+m adapters as opposed to n*m without the abstraction. We could pick our favorite server and our favorite web framework and start developing with a smile on our face. (Ignore for a minute that this is not really possible with the servers and frameworks I just mentioned)

What would this abstraction look like ? Well, a web server is at its hart a very simple thing. You get a request and return a response. A response is simply a status, a set of headers, and a body.

In .NET a very simple implementation of a web “framework” adhering to the abstraction would look something like this.

public Response Call(Environment env)
{
  return new Response(
    status: 200,
    headers: new Dictionary<string, string> { { "Content-Type", "text/plain" } },
    body: new List<string> { "Hello world" });
}

or the more terse ruby version

def call(env)
  [200, {"Content-Type" => "text/plain"}, ["Hello world!"]]
end

Now I finally came to the point of this entire rant. In ruby they actually have this abstraction in place. It’s called rack. You can mix and match web servers and web frameworks to make your coding life a dance on roses.

I like it.

You would think the story ends here, but it doesn’t. There is also something called rack middleware. It takes advantage of the common abstraction between the web server and the web framework and offers you the ability to put logic in between the two. It is entirely possible to create rack middleware for debugging, logging, authentication, the list goes on. You now have reusable components for your web site that is independent of both the web server and the web framework you have chosen. Cool huh?

I like it.

(This part of the post was mostly inspired by this introduction to rack)

Sinatra

Sinatra is a rack enabled web framework. Its implementation is very small and the sample application is even smaller. I will not say much about Sinatra, because the web site says it all.

require 'sinatra'
get '/hi' do
  "Hello World!"
end

Ahhh, the beauty of simplicity. Describe the URL, define the action, and you’re done.

Since Sinatra supports rack, you get the power of rack middleware in addition to the simplicity of Sinatra. I like it.

Tilt

Any serious web framework needs a nice templating engine. In ruby you have a lot of choises like erb, haml and my current favorite mustache. It would be a pain (in the ass) to implement support for all these engines in every web framework, so the ruby guys have (mostly) agreed on tilt. Tilt is a common abstraction that goes between the web framework and the templating engine. Sounds familiar ? To use a tilt enabled templating engine, you need to supply the template and some context for the template (the data). Easy easy.

template = Tilt::ERBTemplate.new('templates/foo.erb')
joe = Person.find('joe')
output = template.render(joe)

Nice stuff.

Well, that’s it for now. See you later.

Comments F Share

NDC 2010 was great. This post is inspired by Tom Gilbs ”Lean quality assurance” talk:

Often we set out to build high quality software without really considering what that entails. How can you really know, when project has finished, that you have created quality software?

Clean code, test coverage and executable specifications created iteratively together with the primary stakeholders seem to be the main focus for most agile projects, at least, in agile theory. There are other concerns that can easily be overlooked when people go into “agile mode”.

Numeric entry and exit

As a stakeholder in a project you don’t really care about testing and executable specifications. You care about the ilities. Usability, Maintainability, Reliability, Security and Scalability are some that come to mind.  Often requirements emerge as the software is developed, but many of these are well known in advance, at least to the stakeholders.

How can we identify these requirements and ensure that they are met at the end of the project? 

A coder will leave it to chance. A software engineer will measure and verify that the requirements are met.

To identify the requirements you first need to identify the stakeholders. This is a process in its own right. Later you can start identifying the requirements. 

According to Tom Gilb you can measure *any* requirement numerically. Later in the process you can use these numbers to evaluate the quality of the software (or some other item) you are producing. A requirement is considered done when your software meets the numeric target for that requirement.   “Numeric entry and exit” is a term that describes this way of thinking.

How can you measure things like usability of a feature? The answer is obviously “it depends”. One metric can be the percentage of grandmothers that understand the feature on the first try. If 80% get it, you have a usable product, and you have met your target for that specific requirement.

Obviously dissecting your requirements this way requires a lot of time and effort. For some projects doing it this way makes no sense, for some it’s too cumbersome and expensive, and for some it totally makes sense.

You get what you measure

Later.

Comments F Share

I guess I need to supply some context here: I’m still in the JavaScript world and I’m still writing desktop-like applications that run in the browser. I only load the DOM once. Here we go:

Every test you add to your application creates a dependency on it. More tests mean more dependencies. Dependencies spell trouble, especially if you depend on code that changes a lot, a.k.a. unstable components.

If you have a test to verify the correctness of a feature, the test should touch a minimal amount code, and it should ideally only touch stable components.

A common problem with integration tests is that they break for reasons having nothing to do with the feature they are testing. You often run in to this problem if you are testing thru the UI. The UI is very unstable (it changes a lot). Minor design changes can break tests. If you cannot easily change your application without creating extra work, the ROI of the integration tests is diminishing.

One solution is to ignore the outer layer of the UI and instead test one layer in. When I write desktop-like web applications I have a thin view layer that encapsulates all the UI interaction (http://bjarte.com/post/effective-javascript-unit-testing). This means that I can invoke every user action thru the view layer.

Let’s look at the following example of a search box:

The view would look something like this:

function SearchView(callback) {
  
  $("#search").click(function () {
    callback($("#q").val());
  });

  function doSearch(){
    $("#search").click();
  }

  return {
    doSearch: doSearch
  }
}

Somewhere else in my code I will make all my views visible globally thru a “views” collection. From my tests I can now invoke behaviour by talking directly to the SearchView:

views.searchView.doSearch(); 

I actually use WatiN to test thru the browser. The only line of code I need in my test (actually specification) to invoke the behaviour is

Browser.Eval("views.searchView.doSearch();");

As you can see, some of my application code is there only to support integration tests (e.g. the doSearch method on the view). I don’t have  a problem with that.You can plug a computer into a modern car to diagnose it. Why shouldn’t you be able to diagnose software?

The point here is that my tests depend on more stable code. The layout can change. The button can move, or perhaps become an image. The search behaviour will stay the same. Hopefully.

Obviously you are not testing EVERYTHING when you don’t run the tests thru every layer of the application. But you know what ? You cannot test everything anyways.  You come to a point where writing more tests will not give you more benefits. At some point more tests become a problem, a maintenance problem. Where you draw this line is dependent, not only on your technical challenges, but also the people in your team, the culture, the process and many other factors. Context, context, context.

Later.

Comments F Share

Most JavaScript testing frameworks run in the browser and are tightly coupled to the DOM.

I don’t care about the DOM and I definitely don’t care about the browser, at least not when I am (unit) testing my JavaScript components.

This probably needs some context; I’m not writing web sites with 5 lines of JavaScript in them you see.  I’m writing desktop like applications that run in the browser. This means I have a lot of widgets and screens that need to talk to each other. I am also doing XMLHTTP calls to the server. All the interaction needs to be tested.

I don’t want to run the tests thru the GUI. It’s to brittle. Changing an a-tag to an input-tag or moving a component on the page can easily break a test. Testing is supposed to improve productivity, not slow it down.

I use the Model-View-Presenter design pattern to design my application layer. This means I have a thin layer, call “the view”,  over my HTML that abstracts away both the DOM and with it, the browser.

The View

Let’s say I have the following component in my application. (It would probably say “search” and not “Google search”, but you get it)

The corresponding HTML would look something like this (ignoring the lucky-button)

<input id="q" type="text" />
<input id="search" type="button" />

I would now create a SearchView module to put an abstraction on top of this HTML (using some JQuery magic)

function SearchView(callback){
	$("#search").click(function(){
		callback($("#q").val());
	});
}

Now the rest of my code will depend on the SearchView function, and only the SearchView function will depend on the browser DOM. But that’s no problem. We can always create a fake SearchView function to act as a stand in for testing purposes.

The key here is that even if I build a very complex application with a lot of interaction, it is only the thin view layer that is depending on the browser DOM. The rest of the code is pure JavaScript.

Testing the code (on the Windows platform)

So I have A LOT of JavaScript code to test, and it does not depend on the browser DOM, so why would I need a browser to test the code ? Wouldn’t it be much better to run the tests thru a command line JavaScript interpreter? Or even better, have the tests run inside Visual Studio ? I definitely think so. Opening up a browser to run a unit  test is way too much friction for me.

There is a JavaScript interpreter build into Windows. You can run it thru the cscript command line tool. To run a file, you do the following:

cscript myFile.js

Obviously if you are doing any meaningful testing you need to load multiple files into the environment, the testing framework, the tests, and the system under test. Luckily you can run multiple files at the same time by creating an xml file. Let's call the file test.wsf.

<?xml version="1.0" encoding="utf-8" ?>
<package>
   <job id="1">
     <script language="JScript" src="TestGui.js" />
     <script language="JScript" src="TestRunner.js" />
     <script language="JScript" src="Foo.js" />
     <script language="JScript" src="FooTests.js" />
   </job>
</package>

You can now run all the files

cscript test.wsf

We can now use this command line tool inside Visual Studio by creating an “external tool”

Tools -> External tools

By adding a shortcut to the external tool, I can run my JavaScript tests from anywhere in the Solution, only using a single keystroke. AND IT’S FAST!

The test runner / framework

As I mentioned previously, most JavaScript testing frameworks out there are tightly coupled to the browser DOM. That will not fly when you are using the Windows Script Host. I chose to build my own little test framework to run the tests. It is very small, and only have three method calls that are coupled to the environment it is running in.

//write to console
WScript.Echo(s); 	
...	
//quit application with error (which will break the build on TeamCity)
WScript.Application.Quit(1);	
...
//write to StdErr
WScript.StdErr.WriteLine(s);    

As you might tell from the code sample above, I run these tests on our CI server as well. No problem there.

My home-made testing framework is in an early stage. You can look at it if you want to, but don’t expect too much: http://github.com/BjartN/BigUnit.

Well, that’s it for now.  Later.

Comments F Share