Thursday, March 31, 2011

Solving Problems (with software)

IT project failures are all too common and in my 25+ years of software development I've seen my share. I've personally been involved in projects that were:
  • Never completed
  • Never implemented
  • Without value and/or loaded with additional work for the user

These failed projects all suffered from the same common problem: poor requirements. Those involved were either unable or unwilling to define and communicate not only what was being created, but, just as important, why it was being created. There was no notion of the project's value.

This requirements problem is certainly well know and discussed. Fred Brooks summed it up nicely in his popular essay No Silver Bullet - Essence and Accident in Software Engineering:
...the hard part of building software is the specification, design and testing of the conceptual construct (data sets, relationships, algorithms), not the labor of representing the construct and testing the fidelity of the representation.

Without well defined and communicated requirements we get poorly-implemented, over-engineered solutions to misunderstood problems. And, we get the following types of conversations:

User:How do I save the order?
Developer:You can't save the order, you haven't entered the due date.
User:The due date isn't supplied at the time of order entry ...

User:How do I get to client ABC?
Developer:Use the client drop down.
User:But it's sorted by client number.
User:I never access clients by their number ...

Project Manager:How do you like the new inventory system?
User:It's great but I can't find the low stock report.
Project Manager:That wasn't in scope ...

Why is it so hard to capture and communicate requirements? In a word, complexity. Software is increasingly complex and, since we tend to tackle the "easy" problems first, each new problem is more complex.

I've always been humbled and mindful of my project failures, and I've spent many hours contemplating better ways of defining requirements that would lead to more successes. I've been heavily influenced by Gerald Weinberg, Martin Fowler, Tom DeMarco, Timothy Lister and James Shore.

Communication is the key, of course. But in coming up with what to communicate we must remember that software is a means to an end. We must postpone the creation of the solution until we understand the problem, and we must focus on problem solving, not writing software.

Also, communication cannot be one-dimensional and one-way. You need feedback, because everyone has different knowledge, experience and assumptions, and you need non-verbal themes to guide and facilitate. I've identified five key themes that have been prominent in my successful projects:

Ubiquitous Language
Communicating clearly and accurately, especially regarding complexity, is made difficult because human language is so imprecise. Multiple terms or words can mean the same thing and a single word or term can be understood to mean different things. Now mix in team members at different levels of experience, understanding, skills and responsibilities, and confusion and errors result. A Ubiquitous Language can help deal with this.

Ubiquitous Language (UL) is a simple concept: All members of the team use a common language - the same language - to understand and describe, and it's the language of the problem domain. The UL is used verbally, in written documentation and most importantly, in the code.

Without a UL, precious time is wasted with misunderstandings and translations. With a UL, ambiguity is reduced and communication is clear and accurate. Another important benefit is that all the team activities are connected with the code.

James Shore has talked about UL as a key part of Agile Development. In addition, UL is a key concept in Domain-Driven Design.

Ground Truth
Ever been to a planning/estimating meeting where developers roll their eyes when a go-live date is announced? Ever been to a release demo where user's are focused on what's missing instead of what's presented? These are examples of what happens without ground truth, or shared assumptions and facts that are aligned with reality.

In software we use models - schedules, documents, pictures - to aid decision making regarding scope, dates, quality and risks. But the models are only valuable when they're rooted in reality. And that means they must be constantly reviewed, evaluated and adjusted to reflect the truth.

With ground truth the commitment and involvement of all team members is enhanced because everyone believes in the assumptions and facts used to make decisions.

Solution Abstraction
When first contemplating how a software project will be done it's important to pull back from the mechanics of UI's, frameworks and protocols and think of the project as a value abstraction, focusing on the problem and pain points that are to be addressed/eliminated. I've found it helpful to explore how the problem could be solved without a computer. Client drop-downs and order entry screens are future details to be address after you discover and define the real value is notifying a client when their order cannot be fulfilled.

Constant Context
One of the first books I read when I was transitioning to OO was UML Distilled by Martin Fowler. The first few chapters did a good job of introducing this newbie to OO software development. One of the concepts that made a lasting impression was the use of Perspectives. Fowler described three perspective used with class diagrams:

Conceptual - the concepts in the domain
Specification - the interfaces
Implementation - how the classes are implemented

These perspectives are also commonly used in all forms of communication related to requirements. I've found it helpful to stay away from Implementation perspective so that your focus is on the solution, not the software. Conceptual and specification perspectives are better suited for requirements but a constant context - the consistent use of one or the other - is necessary so that you can focus on the requirements without having to mentally shift between levels. Its much easier to stay at a conceptual level, for instance, than to have to think conceptually and then shift into details and then back to concepts.

Bridge, Not Ferry
One of the parts of Agile development that I really like is the goal of bringing teams together so that collaboration and Q&A are immediate and so that team members are exposed to day-to-day project details. This team togetherness is only possible if the "gap" between the different groups on the team is not an impediment. Historically, there's been a disconnect between business people and developers - described well as a yawning crevasse of doom. You need a bridge to provide direct communication between team members. You don't want team members ferrying information and decisions back and forth across the divide.

No comments: