The CDE model – and cultural musings

January 26th, 2010

I have a guy from Sweden on my team. He is an accomplished developer, and also a very reflected guy.

A couple of weeks ago, we had a discussion about cultural differences. He came up with an insight I have been thinking about for a while: norwegian developers are harder to lead than swedes. As I understood him, we tend to rebel against being constrained.

The kind of constraints I am talking about is the good kind. The kind where the product owner tells you that he needs some feature before some date for it to make sense. Can you do it? No? Is there something you can do to make it happen? I am not talking about the other stuff.

And here the rebellion lies. Norwegian developers tend to question the need to have a given feature out by some date. We question the need to save money – a couple of hundred thousand of whatever the local monetary unit happens to be can’t make a difference, can it? We spend so much already? We tend to question the need to cut corners instead of describing the horrific consequences which enable someone else to make the decision.

Mike Cohn has just written about removing team members. In his post, he describes something he calls the CDE model. The CDE model describes a container that is placed on a self-organizing team.

The container are the boundaries the team is to operate within. I view constraints on feature dates, which features to add, and cost to be the container we are placed in by the product owner. The product owner owns the budget, the resulting product, and all the problems that come of cutting corners. It isn’t really up to a developer to “adjust” this container. A good developer will tell the product owner what problems he will end up with, but the decision to actually suffer the consequences in order to achieve something else lies with the product owner.

I think this may be to do with the negative view of non-techies that is pervasive in some environments in Norway. If you have a background in economy, chances are that some techies will look down on you, using the somewhat derogatory term “blåruss” to describe your position. Yet I never saw a company succeed without some kind of economic control.

I am worried that norwegian developers are being described as what I view as container-averse. I don’t think we will get agile projects – or any other kind – to become successful products if this is actually correct. I hope this very insightful developer is wrong, or has been unlucky in his coworkers.

Scala and DI/IoC

January 22nd, 2010

I have been fooling around with Scala, Lift and Spring the last couple of months.Why? I want to see if this combination of libraries and languages can make me more productive than my current preferred application stack. I also like the prospect of having to write less linear text (stuff like getters and setters) in order to accomplish something.

Underlying this is a wish to kill off the factory (and, up to a point, the singleton) pattern. When I first started using Spring a number of years ago, a new world opened in terms of maintainability of my code. Inversion of Control (IoC) was the enabler of this leap forward. I think I have only written a handful of factories since.

I am basically looking for a stack that

  • has some kind of inversion of control that enables regression testing outside of a container/in an IDE without requiring deploying test structures in production, or changing code to run tests.
  • has declarative transaction control, along the style of Spring’s @Transactional annotation. Many of the Scala/Lift examples I see has little or no transactional demarcation.
  • has most of the infrastructure stuff I need already present – I do not want to even hear people talking about how easy it is to roll your own. I want stuff to be mainstream and recognizable. None of the companies I work for make money off of doing infrastructure like ORM. It might be symptomatic that Scala so far has two ORM frameworks (Mapper and Record), and three Actor implementations. I expect this flurry of library creation to die down after a while, though. Java wasn’t much better around 2001.
  • enables me to separate my applications along vertical layers, and which helps me separate presentation and business logic. This does not mean I want nonfunctional wrapper layers.
  • is efficient and bug-free.

It turns out other people is thinking along the same lines as I do. A good discussion can be found in this thread. This is a discussion between java/spring based people like me and the authors of libraries like Lift. One side wants IoC, the other says not to bother and use something like Lift’s Boot class instead. And at a superficial level, this isn’t too bad.

The boot class used by Lift looks like a stripped-down Spring XML bean definition list: “Class A needs to have an instance of class B”. I mostly don’t do these kinds of configuration items any more. I tend to declare my services using an annotation (@Component or @Service), and telling beans that use then that they need something injected (@Resource). I think this is cleaner since all the information about the wiring of my application is right there in the code, where it is easy to find for other developers.

My Spring config files deal with transactional support, databases, security, ORM – and test support. I also really, really like the pre-made test libraries that come with Spring. They make it so much easier to get people started with regression tests. None of these are available in Scala, and it seems to be somewhat less than trivial to retrofit these parts of Spring to Scala.

I haven’t decided yet, but I do know that I don’t like my production Boot class knowing about my test structures. Need to think about this for a bit.

Weak or strong typing?

January 20th, 2010

Developers are strange. We tend to spend hours discussing which feature is the best, what language to use, which color our laptops should be.

One of the longest and most polarized discussions is wether strong or weak typing is the better. Then we go on to select a language based on our preference.

Both camps have points. Weak typing allows you a more flexible domain object tree, where you very easily can add new stuff. If you can keep control of your design, this is clearly good. Strong typing allows for more error detection in your IDE – if you use an IDE. You do, right?

I could go on and on, but the problem we tend to create through these discussions is that one tool is not best for all applications. The domain I am going to write about today is integration.

If you come from the weak typing camp, you are probably used to the idea that you can accept any object that conforms to the interface you actually need. If you need version 7 of an interface, and the producer guarantees backwards compatibility, you are probably quite happy to accept even version 10 of the interface. In cases where very good designers have been around, I have seen the client ask the producer wether it can produce data that is compatible with the version it can accept. This very seldom happens in the software world. This is mostly the domain of hardware designers. My macbook is able to detect wether an external display only has a resolution of 800×600, and adjust accordingy. This type of negotiated versioning is my personal favorite – but then again, I have a hardware background.

Other camps accept whatever they get, store it away, alert someone, and then hope someone upgrades the application before the information that is available in the new version is needed. The benefit of this is that the problems gets pushed forward in time, and does not create problems until the very last moment when the information that is unparsed is actually needed. There are benefits and drawbacks to this solution, but it has worked very well for those who are able to view their integration points as a document.

Imagine adding strong typing proponents to this mix. Strong typing means knowing exactly what your objects can look like. My macbook would only accept the 30″ apple-branded displays, making it unusable when I want to present at a conference with an unknown projector. This is where 95% of the integration platforms out there are. To me, it seems like this branch of integration platforms has grown from the language-supported method call interfaces we use internally in our applications, through CORBA, then we did the same thing again only with XML and got web services, which have now evolved into JBI, where stuff looks like documents, but are very strongly typed. These technologies seem to cause teams to make only token efforts towards governance. I have found none who are forward-compatible.

I come from a company where people have been thinking about this for some time. These people have now moved on, but the ideas are still around. One of the ideas brought forward by this bunch of people is to do away with most of the web services stack and use xpath to pick out the bits of a message you need. If elements you don’t recognize and don’t need appear in the message, the extra information is ignored. More information from these people can be found at the Cantara wiki.

int-cat

So – where am I going with all this? I am a proponent of strongly-typed languages. I have seen too many projects fail because most developers lack the discipline necessary to avoid abusing a weakly typed language. That is why I prefer Scala to Ruby.

I am not a proponent of strongly typing everything, though. Strong typing is a good idea inside an application. Anything that is outside the application is going to be less under the control of the developers than whatever is inside. This is a fact. Nothing to do about it. Therefore a different approach is needed in order to avoid hideously large development/maintenance costs. This other approach is going to cost the team more, but the cost is going to be lower than trying to treat the outside world as something that can be controlled. I am not even sure I would want to do that – the implication is that the slowest team in the outside world is going to control the speed of your team, too.

This is where my gripe lies. I want to be able to selectively select how weakly an interface should be typed. When I assemble interfaces, the typing should affect how they interact. Right now, there does not seem to be a language that allows this. All languages have a one-size-fits-all approach to typing which is going to be wrong sometimes. The best compromise I have been able to find so far is the cantara or spring integration approach with the strongly-typed Java.

Perhaps in ten years…

“Is this the time to switch to scala?”

January 2nd, 2010

I have been getting more and more questions from both managers and techies about scala lately. Is this the time to switch to scala. This time, I am going to address the management/architect side of the question.

“Should I switch to scala” usually is the wrong sort of question. A better question is “what can I change that will improve my situation”. Techies usually have a habit of jumping the gun when it comes to technology-related issues. We dive right into the things we know best – the technology.

A lot of organizations are troubled by this. Why does the question come up at all? What does not work out for your organization? Very often, the question is a variation on “we need to get stuff done faster, and once it has been delivered, it needs to continue working”. Fixing the things that will give the biggest improvement first is usually the way to go. So where in all this does Scala – or any other “cool” technology – come into the picture.

This is my preferred list of todos before starting to fiddle with the technology stack, provided there is a workable technology stack around:

  • QA and testing. Many organizations do not have a workable regression test suite around, or have a regression test suite that gets in the way of changing the product. This will give a test workload that increases with the square of the number of features in the product. This will work out for the first couple of months, but the cost of not having a workable, automated regression test suite will quickly cause management or customer attention.
    There will still be a need to test every release. Regression tests are about ensuring that the stuff that worked in the last release is still operational. Anything that has been done in the current release, and anything that is of a visual nature, will have to be tested manually. Regression tests are there to reduce the cost of the tedious testing – the stuff that needs to be repeated every last release, and which gets bigger and more complex for every release.
  • Requirements capture. Many organizations are not very good at selecting which requirements should be implemented, and in what order. I think this is a sign of office politics – someone is out to increase their dominion without regard to the health of the organization. Mental visions of big, flaming eyeballs, orcs in their thousands, and tall, forbidding, black gates are appropriate here. The organization will never have any chance at being remotely efficient before the evil dominions have been reduced to a size where the overlord has to explain why there is a need to have big, black, forbidding gates inside of the organization, until the orcs turn out to be kids with poor complexions and butter knives, and the overload has to pay for the effect his black gates have on the organization from his pre-existing budget.
    Of course, the evil overlord may be the head of IT. Nothing makes an evil overlord like an IT guy. We are too well read in the science-fiction and fantasy worlds not to have picked up a thing or two. There needs to be a balance – read the Recluce books by Modesitt for a fantasy introduction to balance, otherwise have a go at “User stories” by Mike Cohn.
  • Knowledge  management. This has nothing to do with ITIL processes. This is about one part of the organization knowing what the other does. It is about developers understanding how tasks are performed by users of the system. It is about one developer being able to pick up where another left off. It is about management being willing to listen to arguments in non-managerial language.
    This is in direct conflict with being “more efficient” in the short term, and is therefore a problem with some current managerial trends which tend towards optimizing short-term profits to pump stock prices, even at the expense of long-term profits. Developers should get time out to clear up costly “technical issues”, they should work tightly with the users of the system so they understand what is needed, and so on. They are not creating new functionality while they are doing this, and will therefore be “less productive” and give less impressive results in the next financial report.
  • Lifecycle management of the software is something almost no development organizations think much about. I have had people yell at me when I asked what the life expectancy of a system was projected to be. The system was rewritten from scratch two years later. Switching technologies is not an interesting question compared to throwing away all the code every two years. So – how long is the system meant to live, and what should it cost? Should it be possible to make it into a product?
  • Personnel issues are a recurring problem in our profession. We had a visit from Uncle Bob a month or so ago, and he said something about not having weapons in the same room as two senior developers. Sadly, he was right – some of us are hard to work with. These issues can overshadow any benefits gained from moving to a new technology.

Any organization that does not have good control of all of these issues, or who have taken steps to get them under control, should not be thinking about adopting Scala. Fixing any of the above issues – if they apply to your organization – is going to have a much bigger impact than Scala ever will. So – make sure that introducing a new technology is a good idea.

Moving to Scala is going to require investments in new knowledge, there will be technical issues that will have a direct cost both in time and money, there may be downtime. Scala is also a very immature platform compared to Java or .NET. The benefits and risks should be carefully weighed before going for any new technology. This is my own list of pros and cons for Scala right now:

  • Pro: The language is “more orthogonal” than Java. This makes for a language that may require fewer workarounds and sticky-tape solutions.
  • Con: Developers have tendency to abuse new and “cool” technology in an attempt to show off. Unreadable or unmaintainable code is a problem whatever language it is written in.
  • Pro: Scala can reuse existing libraries from the java world. This should improve efficiency, especially if there is existing knowledge of Scala libraries in-house.
  • Con: There is very little knowledge about what bugs may be caused when using Scala with these old libraries. This, sadly, is a also developer-dependent. The organizations that are already using Scala will have solved their own teething troubles – but this does not mean your organizations will not create the same kinds of problems.
  • Pro: The designers of Scala have created a language that is less verbose than Java. This is significant because Java applications tends to get unmanageable by sheer codebase size.
  • Con: The development tools available for Scala are horrible. The best one while I am writing this is IntelliJ, with Eclipse in second place. I don’t like the compilation model of IntelliJ, so I am sticking to Eclipse. Netbeans is not a contender until the Oracle noise has been resolved.
  • Pro: Developers want to use Scala. Hiring talent is probably going to be easier, depending on what the previous technology in use was.
  • Con: Scala might not make it. What happens if Scala turns out to be a fringe language in two years? The risk of having to do another rewrite in another language should be factored into the decision to move to Scala. Personally, I believe in Scala, but there are no guarantees.

Now, don’t misunderstand me. I am going to try to find the time to write a small application in Scala during the next half year or so. I like the language. I am looking forward to getting rid of the non-functional fluff that is irritating me in Java. I just think organizations wanting to move to Scala should have a clear idea of why they are doing it, and what they will have to sort out along the way. Having a strategy or at least an explanation to the challenges posed here is a way to actually get to experience the benefits made possible by Scala.

Oh, and have a happy new year.

What is your new year´s resolution?

December 25th, 2009

I am getting old, I know. Old enough that I have allowed myself to slow down a bit and actually ponder what people around me say – and how they try to get their points across. It is an experience which sometimes ends up with strange insights.

I have just been at home a couple of days celebrating christmas, and got to thinking about people in power (architects, project managers, choose your own fancy name) who have turned out to be a bit off the mark.

These are the most memorable opinions I have heard the last couple of years:

  • “We don´t do it like that here”. This statement is wrong because it doesn´t actually address whatever caused a suggestion of whatever is “not done”. I am sure most developers have come across this one. It even has its own name: “Not invented here syndrome”. This is first since I come across it so often.
  • “I am here to create a counterpoint to your opinions”. This statement is wrong because it doesn´t take into account who is wrong. Taking the opposite viewpoint is never a good strategy – one opinion has to be wrong.
  • “We follow best practices”. The practices in question were out of date, and from the pre-EJB 1.0 time. This statement is wrong because it assumes that whoever has created the best practices have created something that is applicable to all situations. In this case, common opinion was that the pattern never was a best practice at all. This is a variation on invoking God – in this case, the deity in question was one of the original Sun blueprints.

All of these examples have thing in common. Whatever was being said was not backed by an argument based in long- or short-term gains for the project or the organization.

I know I have done this myself in the past. My resolution for the next year – to become a better professional – is to never, ever do this again.

What is your resolution for the new year?

Extensive expensive expertise

December 5th, 2009

I have had a search going the last decade or so at the biggest job posting site in Norway. I like knowing which companies are hiring, and what kind of people they are looking for.

The last year or so, I have seen more and more job postings with variations of “you need to have worked in  very expensive projects”. It usually isn’t worded exactly like this, though. The usual phrase is based on looking for  some kind of leadership role, but only if you have experience with major projects – and here major projects are measured in money. “Need to have been a <something> on projects which cost more than <some nine-digit sum in NOK>”. So far, we haven’t been able to run projects into ten digits here in Norway, and I really hope it isn’t just a question of time.

This mystifies me.

IT projects are – at least not after the .COM frenzy, where a bunch of moneypushers tried to tell us the usual rules were suspended – not supposed to burn off the maximum amount of money possible. You are not supposed to build the biggest empire you can. You are supposed to use the minimal amount of IT needed to increase your earnings. Every extra coin spent needlessly on IT eats into the earnings.

Very few companies consistently manage to hit the sweet spot. Some underspend on IT, and increase their costs or fail to get hold of the possibilities that are enabled by IT. People tell me a lot of companies who have tried to offshore development have learnt this the hard way. Others overspend on IT, and reduce their earnings through direct IT spending.

Going far enough in a job ad to say right out that employees are measured by the amount of money they have managed to spend it totally beyond me.

Why would anyone actively communicate that they want the biggest spenders in the class? Even if they are vetting the applicants during the interview process to sieve out the big spenders, the job posting itself sets an unacceptable standard for whoever is already employed, and this mindset is going to take a major amount of leadership to correct. “We have big balls because we are rolling in other people’s money”? In the future of these companies I see people delivering stuff that doesn’t work. The work ethic and the craftsmanship will have gone from the workplace. This can happen all too easily to a development team. When standards slip, they slip fast.

Fixing a culture that has has been degraded in this way reduces the efficiency of whichever leader was just hired, and increases the costs of the project because the other problems – the problems that presumably were there before the job was posted, and is the reason for the role being created – are going to get less attention.

I hope this kind of wording is a kind of slang – a slang created, probably unconsciously, by groups who have worked together in the past. I hope these groups learn to reassess their value in terms of worth. The alternatives are all far worse.

Scala: Test as a first class citizen?

November 23rd, 2009

I have been toying around with scala the last couple of weeks. I have read the book, wrestled with the gui, and I am now trying to build myself an application stack.

I think the last gamechanger to appear in the software business was TDD – in the sense of writing tests at all. All of a sudden, a team could accelerate instead of getting bogged down in endless regression tests.

So here I am, having a look at a brand new language, and what do I find? The scala libraries have not been instrumented for testing. A small example: If I send a message to an actor, there is no easy to assert in a test that a message should have been sent (that I can find), much less to access it and test the contents.

I remember way back, when I did hardware. We had something called JTAG. JTAG is basically a series of glorified registers that are placed on all outputs and inputs of a module. In normal operation, the registers just sit there. There is a trivial speed cost of having the extra hardware there, and as such, they are just in the way. Total waste.

But when enabled, the registers come alive, letting a tester sense the input and set the outputs of the register. An ASIC designer can inspect the insides of a fairly complex ASIC. All it takes is a couple of electrical connections on the periphery, and all digital design libraries have these structures. But I am digressing.

In the software world, we lack even this basic test standardization. Every last developer has to build his own test harnesses on top of his test bed instead of leveraging well-tested harnesses written by professionals. The best we have are addons like jmock to help us out. It shouldn’t be like this when we design a new language. We should put this in from the start. I want to be able to go ActorService.enableTesting , which would give me access to all messages being sent.

I wish the day had more hours. I wish I had the time to write this myself.

TDD, evolution and architecture

November 15th, 2009

TDD has had a rollercoaster ride since Kent Beck first coined the term in 2002. This is nothing to wonder about – TDD really has the power to be a significant power for change for a team of developers.

There are basically two camps, who don’t agree 100% on what the last “D” is. Kent Beck originally called it “Test Driven Design”, making the tests an integral part of the design process. More than half the people I talk to today (counting myself) is having a problem not calling it “Test Driven Development”, where the tests are not as strong a design driver.

Today, I am going to write about the “tests provide good design” camp. I am in good company, too. Uncle Bob and James Coplien have both debated this, along with many, many others. I am now going to weigh in on this, too. What arrogance. Oh, well. Here goes:

TDD, in the original meaning of the acronym, is a technique where tests are written first. Since the test is new, it will fail. The minimum amount of code that will make the test run is then written, and the process is repeated. At some point, the developer may decide to clean the code up, in which case the tests must run successfully both before and after the cleanup. This refactoring of the code gradually improves the quality and the architecture of the solution.

Testing the wrong thing

A couple of weeks ago I was told that a team had testing well in hand. They had 97% test coverage. I immediately became very, very skeptical since I know the bug count for this team.

We developers are techies. This is what enables us to create very good applications. It is also one of our biggest blind sides. We tend to treat everything as a technological problem. If this is transferred to testing, we get nonsensical tests with names like “testConstructor”.

This is a really, really big problem. Your customer is never going to care about your constructors. What he cares about is the behavior he expects from the solution. This kind of problem was described by Dan North when he coined the term BDD, where the tests are explicitly modeled on customer-oriented statements. So – tests should test something the customer cares about, not the implementation. We may want to change the implementation in the future.

Your tests should be connected to your business model

A side effect of the above statement is that you, if you are using DDD (Domain-driven design, are going to have tests that nestle up to your domain model very intimately. Your customer is going to use some nouns, those nouns are going to become parts of the model, and your tests will probably express themselves using the same nouns. It is imperative that your tests can handle changes to the “domain language” your customer is using.

The weakness of refactoring

Refactoring is a process where you improve your system by getting your tests running right before and after you make a nonfunctional improvement. The first book I saw on the subject was by Martin Fowler. I recommend this book heartily to anyone who hasn’t heard of the term before. Since TDD makes heavy use of refactoring, it is important to understand the limitations of refactoring.

I really hate the “refactoring” menus in IDEs. This is not refactoring, in the meaning Fowler used. These are automatic transformations of your code, and disconnected from your test cases. This tool is on a lower level than refactoring is.

Refactoring can only help you make changes in situations where you have tests that cover the behavior that is affected by the code you want to touch. There are lots of implications to this simple statement, but the most important one is that you cannot change the interfaces your tests use using refactoring. You would need to change your tests too, which would place the operation you are undertaking squarely outside of the “refactoring” paradigm. Tool support may help reduce the risk of the change, but the change is not a refactoring operation.

I know lots of very competent developers who are trying for tests that have sub-second runtimes, which means they are usually unit tests where large parts of the application has been disabled. The danger inherent in this approach is that your test boundaries should be very carefully chosen – and should not evolve significantly. A sub-second test runtime is nice, but is not a requirement the customer necessarily cares about if it costs him significantly along some other dimension.

Creating APIs is something developers think of as fun – and it is. This does not mean your application should be littered with hundreds of shoddily designed APIs. Keep the interfaces you expose (to tests or other clients) to a minimum. This is necessary since these bits cannot be allowed to change very often.

I think tests should be running against a system that is as close to the finished architecture as possible, and test against fairly large functional areas. These areas will become your “published” APIs.

Evolution of architecture

TDD in its pure form is an evolutionary process where the developer decides which code structure is the “fittest”. He uses the microscope created by his test suite to do this.

As I have already mentioned here, I have just finished reading “The greatest show on earth” by Richard Dawkins. He devotes a whole chapter to the design shortcomings of evolution. In his case it is a rebuttal of creationist theories, but I am going to pinch his argument here to point out the known problems to trying to evolve an architecture.

On page 360, I found the following paragraph:

During the evolution of the mammals, however, the neck stretched (fish don’t have necks) and the gills disappeared, some of them turning into useful things such as the thyroid and parathyroid glands, and the various other bits and pieces that combine to form the larynx. Those other useful things, including the parts of the larynx, received the blood supply and their nerve connections from the evolutionary descendants of the blood vessels and the nerves that, once upon a time, served the gills in orderly sequence. As the ancestors of mammals evolved further and further away from their fish ancestors, nerves and blood vessels found themselves pulled and stretched in puzzling directions, which distorted their spatial relations one to another. The vertebrate chest and neck became a mess, unlike the tidy symmetrical, serial repetitiveness of fish gills. And the recurrent laryngeal nerve became more than ordinarily exaggerated casualties of this distortion.

Again – go read this book!

So – the recurrent laryngeal nerve. What am I on about? Well, this is a nerve that is routed from the brain, down into the chest to hook around a major blood vessel, then to return to the larynx. An engineer would lop it off and reroute it to save several metres of nerve in the case of a giraffe, but this is something that we cannot completely trust evolutionary strategies to do.

Ok, what is the solution?

I believe in self-organizing teams where there are incentives to choose the best designer to go off and design the APIs that need to be good from the beginning. I believe in using tests to help get the rest of the design good enough.

This won’t help much, though. Some teams are not heavy on design expertise, or worse yet, may think they are. I have had colleagues with two years of work experience refer to their extensive experience while talking to people with double-digit career spans. I am sure I did it myself ten years ago, and I don’t think there is a single developer out there who is not overestimating his knowledge in one or more areas. Teams who could use more design expertise should do their design up front, or at least to find the parts of the solution which needs to change.

Other teams are able to get things right straight away because most of the teams are experienced designers. If your team has multiple Kent Becks on board, the best thing for an architect to do is probably to shut up and let them get on with it.

There is absolutely nothing I can say from this end to help find this balance, since the balance is team specific. But if you are a person who in some way or other has an interest in the ability of a team to deliver change and quality fast, I suggest you make sure they get a steady stream of domain model changes – and measure them on their ability to get things into production. That way, you will always know what to expect in terms of speed of change.

If they get it wrong – let them stop and sort it out.

If they suggest not releasing to production as often (it is too costly!), get outside help! Your team is in effect telling you they can’t cope, and the problem may be you.

Agile projects don’t need architects?

November 7th, 2009

The past couple of days, people have talked to me about architects. There are a couple of statements I have heard:

- agilists don’t do design

- we (my architect team) are supposed to work as a team with the developers as a self-organizing team. How can we do that, and still tell the teams that they need to do something differently?

This is all based on a couple of misunderstandings, I think. I will try to straighten them out here.

“Self-organizing teams can decide everything by themselves”. This is obviously wrong. If this is the case, then what is the product owner for? The team decides how to make something, but not what to make.

Mike Cohn wrote someting about this yesterday on his blog. There is obviously at least one proponent of agile – with a proven track record – who also thinks about up-front design, which means the teams around him are not all reactive. YAGNI, but you just might want to think about things for a bit anyway.

The product owner is one thing. Something similar applies to the scrummaster or the agile coach. People seem to think these characters don’t wield any power. This is also bunk. The scrummaster controls the direction of the team, almost in a tayloresque. The major difference is that the team is allowed to set their own pace. The same thing applies to the scrum coach – whoever thinks coaches have no influence is obviously not one.

I have to admit I am not a purist. I think there is a need to have a technical function – something like a product owner – to handle technical alignment and nonfunctional stuff at the system level in large projects. I don’t really care what this character is called as long as both the development teams and whoever is exercising the function are in agreement on what this function can and cannot do. I will continue to call the function “architect”, even if this will lose me some hardcore agilists.

Whoever is in a function like this should take care not to become just another developer. This is the first mistake any developer who is tasked with a different job makes. Once the problems start cropping up, it is very easy to fall back into the job you know how to do. I have seen this happen many times – and I did it myself, back when I started out as an architect/techie. The primary task of an architect is to ensure the teams are aligned. Life will become hard if this task is left unfulfilled.

This is where life gets hard. I don’t believe it is possible to get this technical alignment to happen without being a hardcore techie. I absolutely abhor non-developer architects. Still, I am saying techies who get this kind of mandate need to keep at least partly away from being a developer. Feel conflicted yet? You should. This is a very hard balancing act, which is compounded when teams do not agree and there is a need to be a part of all teams and none to be able to get things moving forward.

A coach achieves this by actively not delivering opinions. There are several ways of trying to handle this conflict, ranging from becoming “an opinionated asshole” who never delivers anything of value, going for the more military imperative style, through a coach-like style. Not everyone can adopt all of these styles, and I am not sure any of these styles are right in all situations either. Most architects lack psychology knowledge, and are ill-equipped to do their task.

I am reading “the greatest show on earth” by Richard Dawkins. On page 217, he muses on the building of cathedrals in the context of evolution:

An architect designs a great cathedral. Then, through a hierarchical chain of command, the building operation is broken down into separate departments, which break it down further into sub-departments, and so on until instructions are finally handed out to individual masons, carpenters and glaziers, who go to work until the cathedral is built, looking pretty much like the architect’s original drawing. That’s top-down design.

Bottom-up design works completely differently. I never believed this, but there used to be a myth that some of the finest medieval cathedrals in Europe had no architect. Nobody designed the cathedral. Each mason and carpenter would busy himself, in his own skilled way, with his own little corner of the building, paying scant attention to what the others were doing and no attention to any overall plan. Somehow, out of such anarchy, a cathedral would emerge. If that really happened, it would be bottom-up architecture. Notwithstanding the myth, it surely didn’t happen like that for cathedrals. (Footnote: My medieval historian colleague Dr. Christopher Tyerman confirms that this was indeed a myth that was invented in victorian times for idealistic reasons, but that there was never a scintilla of truth in it.)

This is a wonderful book – go read it.

So, it seems that our profession is now creating the same myth that was created a long time ago by (parts of?) the building industry. I am guessing whoever built cathedrals in victorian times did not perpetuate this myth, but builders of smaller things – things requiring architecture that could easily be handled by a small handful of people – could have used this myth to cut the architects out of the loop. You can still buy a small house without getting help from an architect. I wonder if there are any failed cathedral projects out there where a bunch of masons went wild?

Our problem is that the development business is very new. We don’t know how to define an architect role in a way that works yet, and we don’t know when we need to have this role around. I have a hunch that most who get this right are doing what they are doing by the seat of their pants.

I would also guess the definition of what a software architect should do is situation-dependent. Some teams produce useful installation instructions without being prompted. Others are completely at a loss and need a bit of guidance. Some teams are whipped into stopping their QA processes in favour of producing more functionality, and are never allowed the time to correct the shortcoming.

My own shortlist of candidate areas for guidance is:
- System-level QA
- Performance
- Versioning
- Deployment
- Technology platform (as in “all subsystems should use the same OS and middleware to reduce operating costs”, not “use junit 4.3.1″)
- Arbitration of interfacing conflicts
- Tracking of delivery by external parties, if applicable
- Moderation of time-to-market demands by showing the cost of skimping on structural quality
- Establishing an expectation of the lifespan of the software

I am writing an architecture mandate right now, where I am trying to find the balance between owning nonfunctional requirements, letting the teams be agile (or not, if that takes their fancy), and still opening the door for support if they need and/or want it. If I go too far in one direction, the mandate turns into a big ball of mud which won’t help anyone. If I go too far in the other direction, there won’t be room for change if there turns out to be a need for it. I am going to try to get reviews of the mandate every three months to try to get the whole thing right.

If you have any thoughts on this, please feel free to leave a comment.

X is dead. Long live X.

October 19th, 2009

The last couple of weeks, there has been a flurry of “X is dead” blog posts. Have your pick: Java, ORM and a lot of other things have been declared dead. Rest in peace. Pushing up the daisies. There is a new technology on the block now.

The inevitable answers are usually not far behind. X is not dead after all. You insensitive clods. And I happen to think the coffin should be black, thank you.

I wish people would take half a step back. Yes, technologies become outdated. Then they die, or linger on, doomed to a Cobol-esque existence in projects without the funding and need to evolve. Get over it. The new technologies will also become obsolete, I guarantee it. Struts is now uncool. I can remember how nice it seemed when it appeared.

The interesting question is how your project can get rid of “dead” technologies, which usually means technologies that are now “uncool” because the in-crowd are not using them any more. Can you do that in your projects? What technologies can you swap out for something different? How often would you want to do that? What would the costs and the benefits be? How can you reduce the costs of getting a project to become dynamic and less technology-bound?

Ave artis nova, simili artis seneca.