Design Engineering: the next step

By Alan Cooper

Software construction is slow, costly, and unpredictable.

Slow is a big problem because markets and technologies are always in rapid flux and our ability to maintain pace is critical to our competitiveness.

Costly haunts business people because every precious dollar spent on software design and construction is typically a dollar that is difficult—if not impossible—to directly attribute to a measurable benefit.

Unpredictable is by far the nastiest of these three horsemen of the software apocalypse. Unpredictable means 1) you don't know what you are going to get; and 2) you won't get what you want. In badness, slow and costly pale in comparison to unpredictable. While well-tempered business people are loath to part with either time or money unnecessarily, exchanging time and money for an asset that generates an offsetting future flow of revenue is the essence of business, and "slow" and "costly" are relative terms. If something costs millions and takes years it can still be considered excellent business if the return is tens of millions annually over dozens of years. However, exchanging time and money for something that doesn't generate an appropriate flow of revenue is bad. Very bad.

The key, it seems, is vanquishing unpredictability, which means determining in advance what the right product design is, determining the resources necessary to build it, and doing so. As the airline pilots say, "Plan your flight, and fly your plan."

Now as many of you know, I'm a big proponent of interaction design. For years, I've been describing how good interaction design done in advance of programming can help to vanquish unpredictable—not to mention slow and costly—better than any other tool. Interaction design determines who the users are, what goals they are trying to achieve, and then creates a tangible, visible plan for how the product will look and behave to get those users to achieve those goals. Over the last few years, more and more companies have adopted interaction design methods. They have flight plans. Why, then, are so many organizations still repeatedly visited by those three ugly horsemen of software apocalypse?

As in all things related to software the answer is complex, obscure, counter-intuitive and shaded with nuance. Throwing money or time at the problem doesn't work. Interaction design is a big part of the solution, but without making structural changes to the programming organizations, interaction design alone can't solve the problem. Over the last 15 years my company, Cooper, has provided hundreds of detailed interaction design solutions to a wide variety of companies. How successful these companies are at building those solutions doesn't appear to have anything to do with the complexity of the design, and it doesn't necessarily depend on the skill of the programmers. I have concluded that software builders struggle to integrate design into their process for two basic causes: 1) programmers have never learned to follow a design, and 2) their day-to-day responsibilities forbid them from doing so. I also believe that these reasons can be understood and vanquished.

Of course you can see how both of these problems would stem from the same root: if a programmer has never learned to follow a written design, then he would structure his daily work to do without. He would attempt to do the necessary design himself, concurrent with the construction effort. And that is exactly what programmers at all levels and in all sub-disciplines of computer programming do: they design code at the same time as they build it. If we could untangle these two parts of the programming job, we could begin to defeat the apocalyptic horsemen.

Software professionals have a long, well-established history of designing as they build. In every university, on every project and every thesis, the two tasks are performed simultaneously by the same person. Certainly there is some rudimentary sketching of class boxes or data flow before coding begins, and this is often thought of as "design" but contrast this to the level of detailed design work that goes into the construction of something much simpler than software, say, a modern office building, where thousands of pages of drawings and notes are completed, checked, distributed, bid-upon, and otherwise vetted long before the first shovelful of dirt is turned.

If we can find a way to motivate programmers to integrate planning into their normal engineering tasks we can make significant progress in the battle against the three nasty horsemen. Planning, not plans, is the key to visibility. We already have clear and detailed plans, but with engineers who can and want to follow them, the software construction process becomes clear even to outsiders (like managers), and the whole process becomes more predictable. With this predictability comes a significant reduction in sunk costs and elapsed time, a better quality product with a longer lifespan, and our market opportunity is maximized.

Programming has always been viewed as just that: programming. Only two distinctions have ever been made about the various characteristics of writing code. Programming has been divided up by its arena of tools and practice, such as COBOL versus VisualBasic versus Java versus XML, or systems programming, or server programming, or user interface programming, or embedded programming. And from the instant the first op code was manually entered into the first ENIAC by plugging in a jumper cable, programming has been divided up by skill level. Programmers are nothing if not competitive, and they share a universal delight in the personal achievement of solving the innumerable complex technical problems served up by any programming project.

Programmers rate themselves almost exclusively by these two rubrics: how challenging is the field and toolset, and how virtuoso are their talents.

I propose that the answer to our software construction predictability problems lies not in these two rubrics (which are excellent as far as they go), but in correcting their inherent exclusion of a vital third rubric: the personal goals of the programmer.

Personal goals are our chief motivators in life, and these motivators drive our behavior in the workplace as well as at home. The personal goals of programmers vary widely, and when these goals are ignored, programmers find themselves working unproductively—and often counter-productively.

Of course each programmer is an individual, and there are as many motivating goals as there are programmers. However, I think that there exist at least two large identifiable categories of programmer-goals, and the population of software writers neatly cleaves to one or the other camp.

The first camp is composed of builders: those programmers who, like the many carpenters and masons who preceded them in history, take a sublime joy in seeing their handiwork take form in the real world where it can be—and is—used and appreciated by others. They may be hammering together packing crates or they may be painstakingly crafting Steinway pianos, but seeing their products of wood and steel assume a tangible form that gets things done in the wider world provides them with a sense of accomplishment and well-being. It achieves their goals.

The second camp is composed of designers: those programmers who, like the many visionaries who preceded them in history, take a sublime joy in seeing complex, apparently intractable problems dissolve in the face of their creative thinking. They may be arranging utensils in their kitchen drawers or they may be painstakingly calculating how to shape a girder to support a mile-long highway bridge, but seeing that the solution they imagine is the best and most efficient one possible provides them with a sense of accomplishment and well-being. It achieves their goals.

Of course, the builders share in the joy of devising clever solutions just as the designers revel in seeing their creations take real form, but if a mutually exclusive choice between the two options ever arises, each will be happy to focus on the type of work that best satisfies his goals.

Generally, in the industrial age, a person who designed industrial tools or mechanisms was called an "engineer" while the men and women who actually built the design were referred to by the name of their trade specialty: carpenter, stonemason, thatcher, pipe-fitter, welder.

In the software world, anyone who writes code is typically referred to as an "engineer" and this word has lost much of its precision and all of its usefulness. However, it is still the name of the profession by which many thousands make their livelihoods, so discarding it or denigrating it is counter-productive. Instead, I will prepend descriptive words to "engineer" to differentiate between those programmers who like to build and those who like to design. I call them "Production Engineers" and "Design Engineers" respectively. Production engineers are good programmers who are personally most highly motivated by seeing their work completed and used for practical purposes by other people. Design engineers are good programmers who are personally most highly motivated by assuring that their solutions are the most elegant and efficient possible.
Sharpen your interaction design skills

Interaction designers' motivations are very similar to those of design engineers, but interaction designers are not programmers. Although most programmers imagine that they are also excellent interaction designers, all you have to do to dissuade them of this mistaken belief is to explain that interaction designers spend much of their time interviewing users.

The common expression "Engineers build bridges" is actually a misnomer. Engineers build mathematical models of bridges and draw little pictures of bridges on paper or inside computers. Ironworkers are the people who really build bridges. This inexact, industrial age metonym has led to much confusion in the post-industrial age, where it's all too easy to confuse software designers with software welders because they both use the same tools and raw materials for their very different work.

Software is the only medium where the construction materials are entirely the same as the design materials: source code. When a design engineer works out a complex problem for how a program will work, she uses source code. When a production engineer later uses that code as a design document and produces his own, release-quality, shippable software, he uses source code. In no other medium is this true and it is this watershed more than any other that defines the post-industrial era.

Virtually every industrial age product is first designed on paper or in some other cheap, disposable, high bandwidth, easy-to-iterate medium. In fact, it is these characteristics of the tools that allow design to exist at all. If designers of 15th century European cathedrals had to use huge blocks of limestone and marble to design with, they simply could not do it, and large or complex structures would not exist. Cathedrals were designed on paper (or parchment or vellum), using scaled-down, two dimensional images of the huge three dimensional buildings. Sometimes scaled-down models—usually made of wood—were used to better communicate the design to illiterate masons.
Design refinement process reduces programming burn rate

Yes, software can be a cheap, disposable, high bandwidth, easy-to-iterate medium. Advocates of Agile methods would have you believe that all software is so. If only it were true. Programs written so as to be cheap, disposable, and easy to change are simply not fit for commercialization. They lack the robust completeness necessary for good behavior. Any business person with the slightest experience in software management (and obviously any programmer) knows this simple truth about software and strives to make his or her code complete enough and robust enough to have value in the marketplace. But a remarkable thing happens here. When a conscientious programmer begins the laborious process of adding robustness to his code, the medium very quickly ceases to be a cheap, disposable, high bandwidth, easy-to-iterate medium. Instead it becomes expensive, cherished, obscure, and ponderous. It transitions almost instantly from a design medium to a production medium. And yet, it is still source code, indistinguishable from its design sibling without lengthy study (and if you are not a programmer, don't even bother trying).

As you might imagine, determining the difference between cheap, disposable, high bandwidth, easy-to-iterate "design code" and expensive, cherished, obscure, and ponderous "production code" is critical to the very existence of successful software. I contend that few people in the software business pay attention to this difference, and this is the primary cause of the familiar litany of software toxicity: programs are expensive to write and maintain, don't please their users, inexplicably fail in known marketplaces, cause dissension and frustration in the programmer ranks, demand expensive technical support, intractably resist business redirection, and are closed off to corrective design efforts.

The solution is elegantly simple in concept, if arduously difficult in implementation.
Collaboration with development is a handshake, not a handoff

First, as stated earlier, we need to segregate engineers who like to design software from engineers who like to build software. Second, we need to give those design engineers a cheap, disposable, high bandwidth, easy-to-iterate design medium that happens to be source code. Third, we need to give those production engineers sufficient design support (both technical and interaction) so that when they begin crafting their masterpiece in the expensive, cherished, obscure, and ponderous medium that is production code, they never have to iterate, back-track, redesign, wonder, or guess at what the correct decision might be as they laboriously erect the huge code ziggurat of which a product of shippable quality consists.

Of course, when you give a design engineer a cheap, disposable, high bandwidth, easy-to-iterate technical medium, you must also provide her with the big picture guidance needed to apply that design medium in the most effective way. Actually, she will demand this from you. Before, when the design engineer was tasked with concurrent design and production coding, she was forced to dive in and solve problems big, small and in-between all by herself with little help from anyone. Always under a tight schedule, she would hunker down and refuse admittance from any others. In her situation, merely listening to someone's suggestions could derail days or weeks of effort. The secret was to pick a direction and go.

As soon as the burden of production programming is removed, the design engineer can code up practically anything in a day or two. The problem then becomes, "What do I code?" Our intrepid design engineer will now come asking that question of the interaction designers (or business and marketing folk). This is what Extreme programming is all about: Getting the chunks of code down to pieces small enough to be useful as design tools.

It is vital that the design engineer not try to solve interaction design issues. Not only do design engineers lack the aptitude, training, tools, and time to do so, but they are already quite busy enough doing their own job. The only way to prevent them from attempting to tackle interaction design questions is to assure that interaction designers are present and available at all times. The great bulk of interaction design can be done in advance of design engineering, but never all of it. There are always many, many human interaction issues that arise during engineering that must be solved by those competent to do so.

Are you an interaction designer?

While interaction designers design program behavior that will allow users to achieve their goals, they are constantly considering technology. They need to know if the software can accomplish presentation style A; can the server deliver performance level B; is the benefit worth the cost of coding up widget C? Interaction designers need the help of design engineers to answer such questions. Freed from the burden of production code, a skilled design engineer can answer such questions typically in a matter of hours. The answers are determined by writing snippets of lean code bereft of error checking, internationalization, multiple browser support, print capability, security measures and the thousand other elements that turn design code into production code. In other words: cheap, disposable, high bandwidth, and easy-to-iterate. "Cheap" is really a benefit to the business, while "disposable" is mostly a state-of-mind, but a state-of-mind that makes "easy-to-iterate" a reality. If the programmer knows that the code won't go out the door and won't be subjected to the quality control necessary for shippable, production code, he knows that he can write it exceedingly spare, lacking any frill or breadth. Just a few lines here and a few lines there are all it typically takes. Short code is easy to discard and replace, which empowers the design engineer to iterate. The essence of design (beyond the epiphanic nature of design vision, which you either have or you don't have) is iteration. You've got to try it out this way and that way and the other way to assure yourself that you've got it right. If, as in the old way, your "try it out" medium is ponderous and opaque, design iteration simply won't happen. Worse yet, when the programmer, not having had the luxury of finding the correct path, sets off down the incorrect path and eventually perceives his error, will be forced to bend the code towards the light. It is this "bending" that causes much of the expense of production coding. It also is the prime cause of recalcitrant software that cannot be grown or adapted to a changing marketplace over its lifetime.

It is vital that the production programmer not have to answer design engineering questions during production coding. The only way to avoid this is to assure that all engineering design questions have been answered before production begins. Interaction designers cannot perform this feat, nor can product managers, marketers, CEOs, or program managers. Only design engineers can do it, and only those engineers with a personal proclivity for determining the correct and most efficient answer can apply the discipline necessary for writing large quantities of code that they know must be 1) correct, and 2) thrown out.
Throwing out disposable code is very good. It's like plowing under a cover crop to enrich the soil while fighting weeds. Conversely, throwing out production code is very bad. It's like throwing out brand new automobiles simply because their gas tanks are empty.

Most business executives believe that writing production code is a good thing. They assume that getting to production coding early is better than getting to it later. This is not correct. Writing production code is extremely expensive and extremely permanent. Once you've written it, it tends to stay written. Any changes you might make to production code are 1) harmful to the conceptual integrity of the code, and 2) distracting and annoying to the programmers. Annoying your programmers is more self-destructive to a company than is annoying the Board of Directors. The annoyed programmers hold your company's operations, products, and morale in the palms of their hands.

All of this is to say that if you are engaged in a productive process of learning what your product is supposed to be doing and how it is supposed to be doing it, then the longer you can delay the beginning of production code the better. The wise executive should spare no effort to ask the most detailed questions and demand the most detailed answers, and demand that these questions and answers be made on cheap, disposable, high bandwidth, easy-to-iterate media, such as paper and disposable code. The more paper design and snippets of illustrative code you can provide to the production engineers the better.

Please note that I am not advocating indecision. Endless wrangling and opinion battles get nowhere. The time you invest in delaying the start of production programming must be time you invest in performing productive design work, both interaction and engineering.

I have deliberately avoided using the highly tainted term "prototype." This word has no useful value in the effort to change and improve our software construction process. A "prototype" has come to mean any piece of code that is not yet considered the final release, even though it has probably already been shipped to the customer. This is why I stubbornly adhere to the clumsy phrase "disposable code." The design engineer must know that this code will never, ever leave the doors of the shop. The design engineer must know that his cheap, disposable, high bandwidth, easy-to-iterate code will be used solely for two purposes: 1) to determine the best way to build the product, and 2) to communicate this method to the production programming team.

The design engineer's role is vital. Any sufficiently skilled software engineer can create a product right, but only the design engineer allows you to create the right product. The irony is that the design engineer is not strictly responsible for determining what the right product is. Rather, that's the job for the interaction designer, who interprets the goals of marketing and business stakeholders along with those of the users. But when design and production engineers' roles are conflated there is simply no effective point for the work of the interaction designer to enter into the process. When difficult, technical design questions are being answered at the same time that big slabs of software concrete are being poured, the software construction process becomes a zero-sum game with everyone fighting for scarce resources.

By disentangling design engineering and production engineering, you create the perfect entry point for interaction design. Design engineers—when not compromised by production—deeply desire to create the correct technical solution, but they are generally clueless about what the human problem is they are solving. Interaction designers provide that guidance, and the design engineers are very eager to hear it. The interaction designer becomes the design engineer's best ally and most valuable resource.

As for production engineers, they must be reassured that they don't have to begin writing any code until they know what that code must do and how it must do it. The interaction designer provides the answer to the former, and the design engineer provides the answer to the latter. Once again, we have an organization whose structure impels collaboration from within, rather than one that imposes it from without (always an ineffective way).

Interaction design is design for humans, design engineering is design for computers, and production engineering is implementation. Recognizing these three separate divisions and organizing the work accordingly is something I call "The Triad." While it cannot exist without interaction designers, it depends utterly on teasing apart the two kinds of engineering which today, in most organizations, are almost inextricably linked. It will take some heroic efforts to segregate them.

Once we solve the predictability problem, we find that the forces work to our benefit in other ways. The more visibility we have into the future merit of our software construction efforts today, the more time and money we will be willing to invest in them. If we are confident that the product will achieve all of its business goals, there is no need to tighten our belts unnecessarily. In fact, if we could map individual efforts to their expected returns, we would aggressively fund those efforts with the brightest future. Instead of imposing draconian, across-the-board spending limits, we'd be able to increase spending on the portions of the product that will drive up its desirability in the marketplace.

The more predictable any construction process becomes, the less time and money it requires simply because you traverse fewer blind alleys. In my 30 years experience designing and building software, I have learned that the most expensive software to build is the wrong software. You might think that I'm describing the situation where you build the wrong software, have to throw it away, and then build the correct software. Actually, that's not a bad thing to do at all, as long as it's disposable code. The most expensive wrong software you can build is the wrong software that is shipped to customers because you promised it to them or because your senior executives or financial backers insist—erroneously—that shipping the wrong software that you do have is better than waiting until you have the correct software.

When you divide your programming team into two squads based on their personal goals you will find that both groups get a lot happier. You will also find that happier programmers are more individually productive and more productively communicative. The design engineers will gladly answer tough technical problems by coding a half-dozen different test modules when they are not simultaneously burdened by the needs to convert that code into something that ships to users. The production programmers will quickly learn to bring design questions to the design engineers to solve collaboratively rather than hunkering down and inventing a quick solution that may or may not be correct.

With a construction team properly organized as a Triad, design engineers will be doing what they love to do: rapidly iterating to solve complex technical problems. Production engineers will be doing what they love: building huge, complex systems that make end users happy. Interaction designers will be doing what they love: determining who the users are, what their goals are, and what the best possible software behavior would be to help those users achieve their goals. Product managers will be doing what they love: supporting and facilitating the team towards their common goal of a successful product. And senior executives will be much happier now that they have visibility into what their software construction teams are up to. By getting the right people doing the right jobs in the right sequence, the whole process becomes more predictable, faster, and less costly.

Currently there is a pitched battle raging in the programmer world between conventional engineering methods and Agile methods. What neither side sees is a path of reconciliation; where Agile and conventional methods can effectively coexist. Providentially, the Triad reconciles them very well. The lean, iterative, problem-solving work of the software design engineer is the archetype of Agile programming. The purposeful, methodical construction work of the production engineer is the quintessence of conventional software engineering, particularly the type espoused by disciples of Grady Booch's Rational Unified Process, or RUP. Both methods are correct, but only when used at the correct time and with the correct medium.

The three horsemen of the software apocalypse—slow, costly, and unpredictable—are largely just a single problem: lack of visibility into the process of software design and construction. If we knew what we were doing, we'd be more willing to pay full price for it, and because the returns on successful software are so generous, full price isn't too much to pay. By creating the role of design engineer we empower both interaction designers and production engineers to fulfill their jobs, and the whole process opens like a flower to the examining eyes of the business folks.