Interface Design is a Life and Death Matter

Book Review: Mica R. Endsley - Designing for Situation Awareness: An Approach to User-Centered Design

The correspondence between this work and that of Alan Cooper (The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity and About Face 3: The Essentials of Interaction Design) are staggering. Dr. Endsley and her co-writers rail about how a technology focus leads to poor user interfaces, and recommend a user-centered design approach including "Goal-directed Task Analysis." Alan Cooper rails about how a technology focus leads to poor user interfaces and recommends a user-focused approach he calls "Goal-Directed Design." The difference between this book and Cooper's books is the wealth of dramatic examples and underlying research in Designing for Situation Awareness.

Situation Awareness refers to the OO of the OODA loop - getting input from the environment - Observing - and understanding the significance of that input now and in the future - Orienting. It is a concept used widely in aviation, medicine, and the military - areas where life and death decisions are routinely made based on situation awareness.

One study cited in this book identifies flawed situation awareness as the root cause of 88% of aircraft crashes due to human error. In the remaining 12% the wrong decision was made or there was a problem with execution. With these sobering figures, this book lays out design guidelines to enhance situation awareness.

A formal situation awareness design approach would involve realistic prototyping and rigorous testing as you'd expect for anything related to aviation or medicine. This book provides 50 concrete design principles in six different areas to assist this formal design cycle, but as the book says: "These principles can be applied to a wide range of systems from a variety of domains where achieving and maintaining SA [Situation Awareness] is challenging."

Anyone designing interfaces to support situation awareness or quick comprehension - like performance dashboards - can learn from this book. Unlike software design examples, the examples in this book contain flight numbers and phrases like "killing all aboard" that underscore how very critical situation awareness is, and how driven the authors are to help raise the standards of design.

The only minor criticism I can level is a feeling that this book was rushed together; but with the critical importance of the topic I can see why. I look forward to the recently announced second edition.

Organizational Measurement is Hard

Book Review: Robert D. Austin - Measuring and Managing Performance in Organizations

This book is filled with both humorous and chilling examples of measurement dysfunction that make the sometimes academic approach quite palatable. Dr. Austin identifies three different types of performance measurement based on the intent of the measurement - measurement for motivation, process improvement, or process coordination. It is measurement for motivation that causes the dysfunction that this book so convincingly describes.

For example, if we record the fact that 10 widgets are produced on machine A and we are comparing this against the 10 widget benchmark for bonuses, it is very likely that other perspectives like quality will suffer in the drive to make the 10 widget goal. Austin makes the point that the discovery that every time our overall performance is excellent we have produced 10 widgets does not imply that producing 10 widgets will guarantee excellent overall performance.

If we record the fact that 10 widgets are produced on machine A while only eight widgets are produced in the same time using competing technology on machine B, this is measurement for process improvement and can be very useful - provided it is limited in scope and used purely for the stated purpose.

If we record the fact that 10 widgets are produced on machine A and convey this information to the widget packaging department to ensure that enough widget cases are ready, this is measurement for process coordination, and is also potentially useful on its own.

The idea that the intent or goal of the measurement is of paramount importance is one important lesson from this book.

Austin does make some recommendations about developing effective performance measurement systems.

Understanding the costs involved with "perfect" measurements is part of the solution. Substituting a cheaper approximation for a key measurement is bound to cause problems - witness the measurement of nitrogen instead of protein in wheat gluten used in pet food. The incorrect justification for cheaper approaches is a thread surfaces in other areas - reusing financial figures as a proxy for management accounting leads to flawed descision-making emphasizing short term financial gain - reuse of software components leads to products that are hard to use.

One effective technique is using the end customer as the ultimate judge of quality and performance - the kind of approach described 10 years later in Fred Reichheld's The Ultimate Question: Driving Good Profits and True Growth

Excellent Guidelines for Identifying Metrics

Book Review: Kenneth G. McGee - Heads Up: How to Anticipate Business Surprises and Seize Opportunities First

This book was almost great. A great spin on just about every business process monitoring (BPM) approach going - that business surprises can be avoided through predicting the present following his reporting methodology. Some well chosen examples illustrating each of his well-developed steps build to a major case study at GM that unfortunately seems to lie well outside the book's core approach. McGee concludes with inventive speculation about the future role of BPM that taints the well-reasoned logic of his thesis.

The book begins with examples from recent history and business to support the idea that all business surprises actually have early warnings, but these aren't recognized in time to respond. He forms three categories - surprise events, suspected events, and surmounted events. The difference between the three types of events is whether early warnings are ignored, received too late to act, or acted upon. The book focuses on what is required to identify causal events in time to act. I couldn't help but think of this as shortening an OODA Loop to avert a crisis.

Chapter 2, Identifying and Justifying the Right Real-Time Information, provides excellent guidelines for identifying metrics for a departmental operational dashboard. McGee starts with Goals, prioritizes them, then filters the goals that can benefit from real-time information. Once the candidate goals have been identified, early warning metrics are brainstormed. Using corporate priorities, an ROI analysis identifies which metrics will be tracked. The analysis begins and ends with goals - not with what metrics might be readily available.

Following some case studies, McGee moves on to the Real-Time Enterprise. In this case, the objective is to shorten the reaction time to trigger events to surmount surprises regularly. Metrics are chosen with maximum corporate impact in mind. Critical business processes are retooled to allow continuous data collection. The problem I had with this was the example McGee used, reengineering at GM.

GM's recent dramatic improvements in quality and efficiency are used as an example of this Real-Time process, but unlike earlier cases, there is no direct connection to the steps in the text. Some actions are contradictory - the CEO monitoring daily sales figures - while developing strategy. The improvements seem impressive, although recent delays with the Solstice introduction may indicate that some processes are a bit too lean.

The last part of the book is dedicated to the exciting future of the Chief Monitoring Officer. This sounds like an interesting position, a C-level executive with few staff and a wide mandate, reporting directly to the board; but politics and Sarbanes-Oxley will make this a stressful position. I grasp the importance of real-time monitoring and I really like his discussion of selecting metrics, but I didn't make the leap to CMO with the author.

Settling the Settled Data Issue

Bill Inmon concludes a recent article Settled Data is Best for the Data Warehouse with this statement:

The most accepted school of thought should be, that when transactions occur, that those transactions should not be rushed to the data warehouse but should be allowed to "settle" in the operational environment.

I think this conclusion says more about Bill Inmon's view of a data warehouse than it provides practical advice. Data volatility is an issue that needs to be tackled head on and not ignored until the data is settled.

Mr. Inmon suggests that pulling data into a data warehouse in real time leads to messy complications that would be avoided if we wait 24 hours to a month or more to transform each transaction. For some strategic applications this might be reasonable, but for more and more applications, the data is less valuable as time goes by.

Mr. Inmon's example is an incorrect phone bill, where a call to India was identified by the customer weeks after the bill was issued. By waiting a month or more, these issues are supposed to settle. Even if this was a legitimate approach to handling data volatility, how would you decide when data was settled? Perhaps the customer had already paid the bill, then found the error. Perhaps the customer didn't find the error for three months. If the correction came after the settlement period, wouldn't we still have a mess?

Suppose we can wait for transaction information to settle for analyzingng long-term payment patterns. It would be reckless to use the same settlement rules to manage call volumes. To have variable settlement times for different applications of the same call transaction adds another level of complexity.

We need to stand back and assess the data volatility in the context of our business requirements. If our business needs require near real-time analysis, then we must either accept added volatility or added complexity. If our business needs require an absolutely static view of the data, we need to accept that these are historic snapshots and may contain errors that have since been corrected elsewhere. In either case, the business requirements must come first.