Geir Hedemarks blog

I did a lightning talk this friday on the problems people are experiencing with agile projects when trying to hand them over to a hosting organization. I have been working with this topic several years, and there is a basic problem in that development organizations are measured on change rate, while hosting organizations are measured on stability. There is a basic contradiction in how management goes about rewarding different parts of their organizations. I still think there is quite a lot that can be done at the process manager level, and I also think this is the harder task to solve.

There was an open space discussion on the topic afterwards. I am summarizing the opinions put forth in that discussion here - and adding quite a bit of myself in the process, I am afraid. This is one of the many problems with rewriting a discussion into a blog post. Caveat lector.

These are the problems the people in the open space wanted to raise as candidates for attention:

• The hosting organization needs to be involved early in an agile project. If we are agile, and want to try to communicate with our stakeholders (and the hosting organization is a major stakeholder in the success of a project - without running our code, our customer gets very little benefit), we actually need to have them present.
• ITIL is a set of best practices, but there does not seem to be a great willingness to change how ITIL is applied in the real world. I have had several people outside the open space come up to me and complain about the paperwork that is ITIL. I have also seen this myself - parts of the ITIL crowd has the same problem some agilists are struggling with: “We are ITIL, therefore all we do is good”. Pointing at the good book (in this case, the ITIL book and processes) and stating how the world should be is closer to religion than engineering. “This is how we do it” is never a valid reasoning when your customers raise legitimate concerns they are having. Instead, the paperwork and/or the tooling in a hosting organization should be changed if that is necessary to improve matters.
• Communication needs to happen. It is not enough to have a hosting resource in the project if parts of the organization places obstacles in the way of communication. There have been examples of project managers trying to apply a military chain of command to the flow of communication - a way of organizing communication that even the military has started to leave behind. What happens if a military chain of command is used is that the project manager becomes a bottleneck, slowing the whole project down.
• The hosting resource should deliver whatever technical infrastructure is needed to get the system into production. He also has a “weak veto” on technologies that will not work in a production environment. This will allow the hosting resource to veto bad moves like building the application in the production environment. If the application cannot be built by the developers and handed over, there is something wrong with the tool chain.
• More importantly, the hosting resource should work as a liaison with specialized technology providers inside his own organization. It is the hosting resource which needs to ensure that the load balancer technology can be used in the way planned (or, in extreme cases, dreamt up) by the developers.
• It should be possible to install the application within the first month of development. This will move all the noise that happens in the chaotic last part of the project forward in the project, where it is still possible to make changes.
• It is almost impossible to give any rules to how the workload should be split between the hosting organization and the development team. This depends on the expertise profile of the teams. Someone needs to be able to build a working tool chain for the developers, and someone needs to ensure that the application can be hosted without excessive costs. One person may be perfectly capable of fulfilling both roles, but such people are not common.

Some people reported that it was hard to get the three parts of an organization - the business owner, the hosting organization and the developers - to act towards a common goal.

One of the hosting people in the open space told stories of how some of their customers had ordered the development team not to bother making monitoring and hosting easy - the customer had paid the hosting people, and he expected them to work for the money. The project failed due to excessive hosting costs - the hosting provider had to pass on their elevated costs to the customer, or go out of business.

Other people in the open space told of hosting organizations where the project manager had told the customer straight out that the time of his technicians was too valuable to be wasted talking to the customer.

Next: What should change in the next project?

Read the rest of this entry »

Tomorrow, I am to give a lightning talk on “Agile vs ITIL”. Since I don’t just want to complain about how evil the world is, I am putting up a suggestion for an installation template for server-side, java-based applications. I have been using this template a number of years with good results, but I am sure it can be improved - feel free to give your suggestions.

The template is itself licensed under a creative commons/attribution license, but feel free to create proprietary installation instructions using the template.

The templates, in pages, word and RTF format, can be found here. Enjoy.

(Edited 2008-09-14 to include information from comments)

I have been working with a hosting company and seen what we developers pass off as applications.

These are some musings on what a developer needs to do in order to keep applications deployable.

The installation guide

All developers really, really should create an installation guide. They should then try to forget everything about their application and try to follow the guide.

Did it work? If not, it needs to be fixed. Modern development environments thrive on being able to have massive numbers of deployments. For a far better reasoning behind this than I can ever give in a short blog article, have a look at Refactoring databases by Ambler and Sadalage. Every project, team and developer should isolate themselves from errors made by someone outside their control. Virtualization technologies (or, if you are like me, I prefer to use my own workstation) have made this possible - but only if it is easy enough to deploy the application that people actually bother.

This is somewhat the same problem as testing. Architects need to make it very easy to write good tests, otherwise people won’t bother. So write that deployment guide, and try to make it as easy as possible to get from a source code repository to a running version of the system for someone who does not know the system.

Configuration should be split

A lot of developers seem to think that it is ok to place the configuration inside your application. Since I am a java developer, this often means inside some kind of java archive.

This is a really bad proposition for keeping an application deployable. Often, people will start to check in configurations for the test environments. There never seem to be provisions for configuring the production environment (or, when it comes to that, developer environments) without doing silly stuff.

It is perfectly ok to have all the static configuration inside the application. Whatever changes between installations - database logins and the like - need to go somewhere else. These are some guidelines I try to follow:

• Production passwords must never be checked into a developer repository. Developers should not have access to production passwords without a human giving it to them. Letting everyone tamper with the production database is a very good way to increase the risk of stability problems.
• The hosting organization must not be required to unpack an application, rewrite the configuration, and repack the application. Repacking leads to several different versions of the application floating around, with who knows what bugs. The release manager builds the application archive, period. Nobody messes with it afterwards.
• There must not be defaults for the installation-critical configuration bits that point to the test rig. If you do this, you are placing yourself in a situation where the only protection against accidentally putting production data into a test rig is a routing table entry. If the entry disappears, the best thing that can happen is a very costly reconciliation between the test and production data sets. Worst case, data is lost when someone tries to test the stuff that is in production - the stuff which by now is not behaving as expected - and clears the test database before the test starts. Please don’t write this off as far-fetched. I have seen it happen.
• If installation-critical configuration does not exist, the application needs to stop. This is one of the oversights of the JEE specifications. There is no way for an application to tell the application server that it wants to stop because continuing is not an option. This functionality usually needs to be written inside the application or be application server specific. If this condition is not met, people trying to install the application are going to spend lots of time reading logs, trying to find the log statement saying that it wasn’t possible to connect to the database. This statement should perferably be the last statement before the “I am stopping” statement in the application log.

I usually make some kind of provision for providing the name of a configuration file using either JNDI or command-line options to JAVA. Both are usually acceptable to hosting organizations.

A nice side effect of not hardcoding the location of the configuration file is that several instances of the application may suddenly run on one computer. This does away with a lot of virtual machines.

Use a production-like environment during development

The closer the system the developer uses is to the production enviroment, the fewer the problems.

How do you know that you are close to the production environment before this enviroment has been built? Specify how you want it to look. Lots of projects get into trouble because different team members think something is self evident - like how the production environment looks. It usually isn’t at this level. Knowing how the production environment needs to look also makes it easier to select a hosting provider.

Specifying how the application enviroment should be, seen from the development side, is the first step of starting a constructive information exchange with the hosting provider. A good, professional hosting provider will wish to make changes to the environment of the application in order to reduce the costs associated with hosting your application, and there is absolutely nothing a developer should attempt to do about this. A competent hosting vendor is the hosting technology expert, and needs to be able to exercise this kind of control.

Whatever changes to the application environment is agreed on also needs to be fed back into the development and test environments. As the first paragraph of this section started - the closer the system the developer uses is to the production environment, the fewer the problems.

Share the pain!

All developers should install a local copy of the application. If this is not possible, something is wrong. If it is too cumbersome to do so, only the developers are in a position to fix the problem - and hosting personnel is not much cheaper than developers, so it is better to fix the issue than to let the problem persist for the lifetime of the project.

Next: Installing deployable applications… When I get the time, at least. Shout if I should prioritize it.

Summer. Yup. Ice cream time. Cholocate ice cream.

• 6 tablespoons cocoa powder
• 4 dl skimmed milk
• 40 g sugar

Whisk the cocoa to a paste with some of the milk while heating, add rest of milk while whisking, let simmer until the “powdery taste” is gone. Add 100g of dark chocolate and a knob of butter.

• 3 egg yolks
• 40g sugar

Whisk the egg yolks with the sugar, then add the cocoa slowly while whisking vigorously. Put the stuff back in the pan and heat until the mixture goes more rigid. Do not boil - the result will be similar to an overcooked vanilla sauce.

• 1 ts granulated coffee
• 1 ts vanilla extract
• 2.5 dl cream
• sugar syrup (50% sugar by weight, boiled and chilled)
• 2 cl cointreau or other orange-flavored spirit

Stir in additional flavorings, then the cream. This will lower the temperature of the thing enough that the cointreau can be added without the alcohol disappearing. Chill further.

Adjust the consistency of the ice cream base with skimmed milk or water then it is cool - 20 to 10 centigrade. Milk will take away some of the chocolate-y punch. Your call. The consistency should be like a slightly thick vanilla sauce.

Freeze in an ice cream maker or put in freezer and stir every 30 minutes.

Use a small scoop. This thing really packs a punch - there is absolutely nothing subtle about the flavour.

I use “Ices” by Liddell/Weir as a base when making ice cream.

Here we go again. We are barely through with the SOA hype (See SOA-or-DOA or Does my bus look big in this) before the norwegian colored it press is starting to hype “software as a service” (SaaS). Again.

Old news apparently becomes good as new when big companies decide that they are looking into changing their licensing model. It seems that Microsoft is looking hard at the online office productivity tools made by Google.

The journalist then added some stuff of his own. In just a short while, all our applications are going to be online applications. We won’t store stuff locally any more, and we will rent our applications. Hosting your own mail server is going to be a thing of the past. If only I hadn’t heard this before. There are still lots of problems that need to be overcome before we end up leasing our productivity applications - even if I am writing this in an online editor in my blog tool, which incidentally is also free.

The hype has been here before under a different acronym. In 2001, ASPs were going to take over the market. I guess the half-life of a journalist in the colored IT press is shorter than six years, which means we now get a new round of the same hype with different acronyms attached.

There are still no critical questions being asked. And it seems that we like it. Someone has to buy these rags.

Whatever happened to critical journalism? Why are there no magazines out there that can actually provide me with pre-chewed overview information I can trust? The only IT magazines I have been able to lay my hands on seem to be tabloids with all the journalism removed.

In the non-IT world, there are some exceptions. The New York Times seems to give a fairly balanced view on american issues. The local newspaper “Morgenbladet” also seems to employ journalists that are worth something. So far, in the IT area, I have found nothing. Nada.

I am afraid this is how things are going to continue. We are still going to be subjected to waves of hype whenever some journalist wannabe wants to sell more copies. I would love to be proven wrong, though.

Today, I got my preconceptions challenged. As such, it has been a good day.

It concerns someone on the periphery of the people I know, who I am afraid I had a bad feeling about when I met her a couple of years back.

People usually form a very firm impression of people when they first meet someone. You have ten minutes, and if you haven’t measured up in that time, it is likely you never will unless you work very long and hard to fix whatever happened. Our brains love to do pattern matching, and this seems to be one of the uses that have been honed by evolution over the years. Will you trust someone you just met, or do you need to keep your distance so you don’t get burnt? Can you hunt with this individual, or will whatever you bring down just get stolen?

There is a long, long history if preconceptions like these through the ages. Humorism has had a long and fruitful life. The idea is connected to the idea of there being four elements - fire, earth, water and air. Therefore, or so the idea seems to go, there must also be four bodily fluids - blood, yellow and black bile, and phlegm. Or so the story goes. With these four humors, four personality types go: Sanguine persons, who are courageous, hopeful and amorous, had more blood. Those with more black bile were thought to be melancholic, and therefore suffer from sleeplessness, irritability and being despondent. But not to worry, if you eat right, you can balance things out. It almost seems like the precursor to the stuff you find in the glossies in the dentists waiting room. Would you buy something from the melancholic?

Thoughts like these have been with us all the way. We have tried to analyze and categorize what I think are the results of our first impressions. Some have even tried to ascribe personal characteristics based on measurements of the face and skull - and this is not too long ago.

These disciplines are mostly considered defunct today. I guess recruiters may still use some of this based on more modern psychological indicators like the Myers-Briggs test. I was subjected to a test like this by a recruiter when I applied for my first job, along with some other tests. I didn’t know the result of the test, but I had just finished my draft service and wanted to start a new job. In my final interview, I was told I would get an offer, asked to be allowed to read it and signed on the spot. Apparently, my profile read “has difficulty making decisions”. It was the last time we used that recruiting agency.

Ideas like these have been eagerly used by people with agendas in the past. In the not too distant past, Norway sterilized people who were in some way thought to be in the shallow end of the gene pool. There are lots of other examples. The idea that you can easily divide people into good and bad is a very seductive one. It takes away all the hard questions.

So, I don’t like these stereotypical models. I feel they simplify and cheapen the people around me into flat, cardboard versions. How can a one-hour test summarize the personality that has been developed over the course of many years?

And this is where I meet myself, having tried to work against my own stereotypical preconceptions against someone which are based on their looks and some short conversations. My preconceptions have now been proven right twice - she really didn’t follow up on her promises. Our trustworthiness detector works at least once in a while, and I can’t explain how easily.

What to do about it?

Nothing at all. Business as usual. I will probably still have to size people up on the spur of the moment once in a while, and run the risk of getting it wrong. If you have to, you have to. I will also continue to dislike stereotypical labels on persons.

But there may be a middle ground which just became a bit clearer.

During a talk with a colleague this week, I think I hit upon an insight into why SOA and domain-driven design seem to be causing so much pain in organizations out there.

First of all, I am writing about businesses where there is a fair bit of money on the line, and with a nontrivial complexity. In other words, I am talking about borderline “enterprise” businesses.

When a business first starts out, there is uncertainty about the business and marketing plans. As a result, the first couple of years of development in an emerging market is a hunt to find the right domain language, and how to best put it into software systems that help create value. During this period, the domain model is very plastic, and usually expands every week. However, since the business is still finding its feet, there are no IT-based processes to speak of - or they are IT-based, but not very complex. There are experts around to handle the processes.

Any business entering an unknown business area has a very strong economic motivation that pushes them towards finding a business model that works - quickly. If you don’t have a working business model, you are going to get outpaced by your competitors, or not get enough business to stay afloat. In other words, anything goes as long as the business is able to hit a sweet spot in the market. Who cares if there are people handling stuff manually if you aren’t getting any customers?

After the business has found one or more of these sweet spots, the game changes. At some point, someone is going to find that the business plan is good enough - it is time to cut costs and eliminate unnecessary manual steps. This is where business process modeling starts to make an appearance, and with it, SOA-style technologies. There is usually a fair bit of pain involved in introducing these kinds of technologies. They are costly, and in some cases hard to understand. Sometimes, there are consultants around to help with choosing the right path - and there is still a substantial risk of doing it wrong, even though significant funds are poured into finding the best expertise out there.

The good news is that a successful SOA project makes it easier to change the business processes. This is the upside which makes all of the pain bearable. However, making the business processes pliable usually also makes the business model rigid. It is no longer as easy to change the underlying domain model. This is a function of having exposed services, which by now have an unknown number of clients, and which are part of a set of processes which usually get complex very fast. Anytime something in the underlying data model needs to change, there is a significant risk of major pain or administration happening.

Now, imagine that the enterprise wants to expand into a new market, or if something major happens to the market. This is a clear change to the business model - new domain objects will usually be introduced, or at the very least new services.

This may be part of the reason why large organizations have a hard time keeping up with smaller, younger organizations when the market changes. They may be trying to fit a changing domain model into an existing environment geared towards efficient operations instead of building a “domain model nursery” where the focus is evolving the business model as fast as possible.

What to do about this? Not much, seen from the development point of view. Developers are generally not fond of having two ways of doing what seems like one thing. This is probably very hard to change.

From the management side, there may be lots of things that can make a difference. The new market may be handled by a separate group, possibly even by a separate organization. The differences between the markets may be communicated to the developers as a precondition before the work starts, possibly even accompanied by a strategic plan for how the system is expected to evolve.

This may seem obvious, but the idea that published services may fossilize your domain model in this way was not something I had been able to put in writing until now. I hope someone gets some mileage from this…

I was in Sweden last week, and picked up a couple of small (37.5 cl) bottles of Amarone. Masi Costasera.

We just opened one tonight. We had some ground beef (not the nasty, fatty stuff. I am talking about the good stuff, made from meat instead of gristle and fat) and some whole wheat hamburger buns, and after a quick look in the fridge, we ended up with oven fried potato slices and hamburgers.

The burgers were trimmed with tomatoes, onions, cress and homemade pesto. The potatoes were seasoned with salt and ground paprika, and a dash of yoghurt on top. This is how hamburgers should be, according to us. The hamburger chains don’t enter into it, although “Fyret” here in Oslo does a burger that comes very close. Anyway -  obviously a fairly hefty wine was needed. So we popped open the amarone, which turned out to be just what was needed.

Around here, it is almost impossible to get decent wines in small bottles. Why, oh why? I really only wanted the one glass of wine tonight, and 37.5 cl was more than enough - we still have leftovers. The only wines you can get in this kind of size in Norway is the kind of stuff I would hesitate to put into a sauce.

I can’t believe it wouldn’t be a good economic move for the importers. I would never have opened a whole bottle of amarone tonight. We didn’t want three glasses each - we wanted one. If I didn’t have the small’uns in the cupboard, there would not have been any wine for dinner.

Go figure. I’ll have to go and ask some friends of mine who are importing wines. For all I know, there may be some kind of taboo on anything but the normal-size bottles and up.

’till later.

Geir

I have just figured out why Apple seem to have cornered the market for java development hardware.

It has nothing to do with speed, ease of use, or glitzyness.

It has everything to do with annotations. People are starting to use annotations for loads of stuff now. The most extreme example is probably qi4j, where the annotations are running the show with bits of java showing through.

Around here, Spring have cornered the market for application infrastructure. Spring has also started to do annotations that are very useful.

This is where Apple comes in. They have put the “@” symbol on a separate key. That has to be good since it feels like java code now is consisting of close to 50% ampersands.

I really, really wish Sun had taken a different approach with annotations. My code is starting to look like perl, with the symbols you get by pressing shift and the upper row on a PC keyboard eating more and more attention.

At least I don’t have to press shift to get the @ symbol. It almost seems like Sun are also using Macs.

I am back in the SW business.

As of february 14, I am a principal consultant at Objectware in Oslo. Feels good. This also means I have started developing again.

One of the first things that happened is that grabbed hold of Testaco again. All of a sudden, the itch is back. Need to sort my own tests.

One of the ideas I have had is to drop the testing framework of Testaco and replace it with Unitils, leaving Testaco as an eclipse plugin that automates maintenance of tests. The tests would run using Unitils. I have talked to the developers of unitils, and they are not against the idea.

My problem is that I don’t know how large a user base Testaco has, if any.

Going through with this refactoring would mean breaking backwards compatibility for Testaco. Repairing it again could be done mostly by scripting, though.

So - what do you think? If you are a Testaco user, please comment this post. If you think Testaco is a good idea and would like to see it stabilized I would also appreciate some feedback.