As far as Enterprise Architects are concerned, there is no other, single characteristic of a software system more important than the performance and scalability. Or, at least - there should not be. Sadly, in reality, performance is often an afterthought. Inexperienced development teams look at it in disdain and refuse to “waste” time on performance tuning. They amuse themselves with “feature enhancements” and only remember performance when things start to fall apart. That moment in time is usually before the champagne glasses, in celebration of the “successful” launch, get a chance to dry.

When the panic day hits, things don’t get any better. Developers, typically, jump “fixing” code, based on a bunch of wrongful perceptions they have acquired on some i-am-smarter-than-you-if-i-talk-more forums or go begging management for “better hardware”. Benchmarking, profiling and systematic analysis are ignored because nobody has much experience with any of those. Eventually management gets frustrated, hires a highway-robber a.k.a consultant and either gets things temporarily fixed, or in the worst-case scenario - gets a huge, meaningless report of “what’s wrong”.

On the other side of the spectrum are the overly concerned, control-freak architects that design and redesign architecture for months, in anticipation of the performance problems. The typical end-result of such effort is a stack of UML diagrams and little to none production code.

So, what is the solution for achieving Five Nines? Well, actually - the same as in any other industry striving for exceptional quality. How does Car Manufacturing sound?

Kaizen, or the Process of Continuous Improvement was developed at Toyota and is largely responsible for the legendary reliability, durability and efficiency of the Japanese cars.

“Kaizen is a daily activity whose purpose goes beyond improvement. It is also a process that, when done correctly, humanizes the workplace, eliminates hard work (both mental and physical), and teaches people how to do perform experiments using the scientific method and how to learn to spot and eliminate waste in business processes.” (Wikipedia)

Along those lines, the 5 “commandments” of performance and high availability are:
**
**

Performance benchmarking and tuning is an ongoing activity.

It should be a well-defined process.

Bottlenecks should be identified in a “scientific” way - through measurements, not - presumptions.

When done properly, it humanizes the workplace and eliminates crisis situations.

It is learned and perfected only through doing.