DevOps and the Indy 500: Both Driven to Succeed
I was thinking about the Indy 500 the other day and realized there were many similarities with successful DevOps environments.
Both are all about using accelerated laps (or continuous cycles) to quickly get to the finish line (or delivery lane). Both start with a competitive problem: How to win against a field of strong, ambitious competitors? The two need an investment sponsor. Both involve the best technology tools and expertise (or best practices) to operate. Likewise, the two require highly skilled, collaborative support teams (Dev and Ops), plus an expert driver (aka orchestration tools) to drive each lap. To ensure everything is prepped and ready, drivers use a (pre-flight) test track before the next race (or build cycle). The race is won one lap – or cycle – at a time. Each lap is measured (or continuously tested) and the delivery vehicle is tuned, using metrics, as needed. The finish line (deployment goal) is clear but unforeseen circumstances (such as process and product problems) can affect the outcome.
A network equipment manufacturer used these principles to transform its business by implementing DevOps continuous testing for a sophisticated large-scale network product between 2010 and 2014. Business issues with time-to-market, quality, innovation, costs and security forced the transformation. Technical pains including distributed team silos, long serial development/test/release processes, and inconsistent metrics contributed to the overall business discomfort.
Not knowing what to do, but knowing what they wanted, the business leadership team set some challenging goals.
- Monthly releases, on-time
- 50 percent fewer escalations / customer defects per year
- 25 percent more new features / year
- Stabilize CapEx and OpEx spending
- Close intellectual property security holes
The team discovered that getting DevOps right requires much more than fast technology. The journey sometimes took team members outside the boundaries of their technical expertise. People and culture issues stalled progress, caused sidesteps, and required working outside the comfort zone of each organization.
Early on, the organization determined that continuous testing aspects is especially challenging because DevOps is all about speed while testing is all about thoroughness. Getting thorough testing, at speed, is tricky. If not done right you end up driving off the track. The company was forced to re-think testing practices that were designed for thorough testing but not the continuous processes that DevOps requires.
The business formed an implementation steering team, headed by a senior management sponsor, and including technical architects and an infrastructure tools group. A key item was to parse the product, processes, and tools into modular, individually testable packages.
In broad terms there were three major implementation phases.
Phase 1 - Design a successful prototype:
- An isolated test bed of equipment was created in the main engineering lab
- Test and test orchestration tools were installed
- Test suites were refactored into four test cycles: build, integration, nightly, full regression
- Pre-flight tests were implemented to check code changes before commits
- The team implemented metrics to measure the effects of scale and to optimize the entire system: test run times, test environment tests, meta database for tracking test attributes, re-engineered test reports, dashboards, and debug data
Phase 2 - Test and fine-tune technology:
- Parallel processing tactics included virtualized and orchestrated elastic test resource scaling
- Multiple DevOps setups were implemented
- Test-cases were re-engineered for each test cycle
- Implemented a modular package-based test selection mechanism
- Tuned report analytics to highlight the most important results of each test cycle
Phase 3 - Operational Team Excellence:
- Restructured development into customer/market segment teams with full responsibility for quality assurance and training
- Enhanced the development environment with distributed version management systems, static analysis tools and access to pre-flight testing capabilities.
The business results were remarkable.
In the first year, the business accomplished on-time monthly releases; metrics provided visibility for the entire delivery pipeline, and intellectual property security protections were put in place. The second year, it realized 70 percent fewer customer escalations, 60 percent less corrective work, and delivered 25 percent more features. In the third year CapEx spending was stabilized and OpEx spending was reduced. All of the business results can be traced to the underlying technical results.
Along the way the organization learned proactive culture change management reduces staff turnover and implementation could be accelerated using expert consultants familiar with best practices.
It is clear DevOps continuous testing, just like an Indy 500 race, required investment but it was worth it. A business that was struggling is now thriving and on a path for further optimizations. Finish line and victory lap accomplished. Time to break out the milk!
About the Author:
Marc Hornbeek is Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, Infrastructure Test Optimization (ITO) BU. He recently managed DevOps at Spirent. He has performed as the primary architect of test automation tools and champion of test automation for firms ranging from start-ups to large multi-national companies. He published more than 30 articles and has been a speaker at numerous conferences and user forums primarily regarding topics related to continuous automated testing and DevOps. Learn more on Twitter @Spirent.