You know something’s wrong when a Mars spacecraft lands without any software glitches or crashes, while at the same time here on Earth a company’s financial trading system fails disastrously costing nearly half-a-billion dollars. I expected the opposite, didn’t you? These two separate events demonstrate the brutally obvious outcomes of an organization’s level of commitment to quality. And the differences couldn’t be more stark.
The Mars landing (a.k.a. the 7 minutes of terror) required about 500,000 lines of code to execute with perfect timing, slowing Curiosity’s descent from 13,000 miles per hour to less than three miles per hour in a hovering maneuver to touch down on the surface of Mars. This system’s execution flawlessly triggered 76 explosive charges without damaging the sensitive electronics of Curiosity’s scientific lab equipment.
Alternatively, the high-frequency low-latency trading software used by Knight Capital Group might involve exponentially more code when you consider all of the end-to-end components that comprise the system’s architecture. Knight’s software is responsible for routing trades across the various exchanges with extremely quick response times. There are typically no explosive charges in Knight’s trading system; that is if you leave out the $440 million charge to rescue the company from a financial crash.
With similar levels of system complexity both of these organizations had a common goal: avoiding a crash. But the difference between them is their attitude about quality. Some think that quality is just a characteristic that must be technically validated. But there aren’t just knobs to twist, buttons to press and formulas to compute for quality. Quality is not just a technical aspect of software. Quality is the result of human attention to detail and critical thinking about the outcomes and impacts of our actions. An attitude that supports high quality comes from people who are engaged in and supported in the pursuit of excellence in their work.
The commitment to quality software starts with the attitudes of the people at the very top, through investment in resources and support for culture that rewards the individual’s behavior to prioritize and ensure quality outcomes. There’s an expectation that if individuals are given time and respect to deliver high quality, they will be committed to doing so. As an example, with regard to the engineers working on the Curiosity program NASA Administrator Charles Bolden was quoted as saying: “This is an amazing achievement, made possible by a team of scientists and engineers from around the world and led by the extraordinary men and women of NASA and our Jet Propulsion Laboratory.” Contrast that with this statement from Knight’s CEO Thomas Joyce: “You cannot keep people from doing stupid things…that is what happens when you have a culture of risk.” Joyce’s company lost nearly 70% of its value in 2 days.
The “culture of risk” Joyce refers to is about taking financial risks for gain, which includes hedging against poor quality financial decisions. This attitude of risk-taking is counter-productive to quality. It can permeate the culture of a company even when it comes to decisions about technology. You can see this in the form of extreme cost-cutting on tools and expertise, a lack of training and professional development and also in the prioritization of project completion over quality. When it comes to technology systems the most effective approach to hedging risk is to thoroughly test everything involved with the system and to not simply rationalize impending failure.
As evidence to their attitude towards quality, the engineers at NASA certainly know how to manage technical risk. They test!
About the Author
Mark Tomlinson Mark Tomlinson is a software tester and test engineer. His career began in 1992 with a comprehensive two-year test for a life-critical transportation system, a project which piqued his interest for software testing, quality assurance, and test automation. That first test project sought to prevent trains from running into each other — and Mark has metaphorically been preventing “train wrecks” for his customers for the past 20 years. He has broad experience with real-world scenario testing of large and complex systems and is regarded as a leading expert in software testing automation with a specific emphasis on performance.
Mark worked for six years at Microsoft Corporation as a performance consultant and engineer in the Microsoft Services Labs, in the Enterprise Engineering Center and in the SQL Server labs. His efforts to foster the success of Microsoft’s top-tier Enterprise customers was focused on their early adoption of Microsoft products as part of mission-critical operations. In 2008, as the LoadRunner Product Manager at HP Software Mark led the team to deliver leading innovations for performance testing and engineering as part of HP’s suite of performance validation and management products.
Mark now offers coaching, training and consulting to help customers adopt modern performance testing and engineering strategies, practices and behaviors for better performing technology systems.