Organizations Struggle with Continuous Testing Benchmarks

April 25, 2019

Sauce Labs noted the results of its inaugural Continuous Testing Benchmark, a new report that for the first time enables organizations to measure how their continuous testing efforts stack up with those of their peers. Leveraging anonymized data from millions of actual user tests performed on the company’s continuous testing platform, the Sauce Labs Continuous Testing Benchmark identifies the four best practices necessary to achieve continuous testing excellence, and tracks organizations’ collective performance against those benchmarks. The report reveals that room for improvement exists across the entire spectrum of continuous testing excellence, most notably in the areas of test quality and test run time.

A first-of-its-kind for the continuous testing industry, the report clearly defines the four key pillars of continuous testing, along with the associated performance benchmark necessary to achieve excellence in each category, including:

  • Test Quality
    • Benchmark for Excellence: Pass at least 90% of all tests run 
  • Test Run Time
    • Benchmark for Excellence: Test run times averaging 2 minutes or less
  • Test Platform Coverage
    • Benchmark for Excellence: Test against at least 5 platforms on average
  • Test Concurrency
    • Benchmark for Excellence: Leverage at least 75% of available test capacity during peak testing periods

Real-world data shows need for increased focus on key best practices
The report also reveals how organizations performed against those benchmarks across millions of tests run on the Sauce Labs Continuous Testing Cloud. While the majority of organizations excel in the areas of test platform coverage and test concurrency, as continuous testing matures, more work is needed to ensure excellence in test quality and test run time. Of note:

  • Test Quality: Just 18.75% of organizations pass at least 90% of the tests they run
  • Test Run Time: Only 35.94% of organizations complete their tests in an average of 2 minutes or less
  • Test Platform Coverage: Nearly two-thirds (62.53%) of organizations test across 5 or more platforms on average
  • Test Concurrency: A full 70.88% of organizations utilize at least 75% of their available testing capacity during peak testing periods

Collectively, just 6.23% of organizations achieved the benchmark for excellence across all four categories, underscoring the relatively nascent stage of continuous testing as a strategic imperative in the DevOps era.

“Our aim with this report is to provide developers and quality assurance teams with something they’ve never had: a clearly defined set of metrics they can use to measure how they’re performing in their pursuit of continuous testing excellence,” said Charles Ramsey, CEO, Sauce Labs. “As organizations continue to prioritize continuous testing as the foundation of their agile development efforts, we are excited to see how their performance against these benchmarks improves over time, and we look forward to doing our part to help them reach their goals.”

Test implementations at a glance 
In addition to providing organizations with a benchmark for achieving continuous testing excellence, the report also takes a closer look at current testing implementations among Sauce Labs users to get a better understanding of the browsers and devices against which organizations are most commonly testing. Top-line results include:

  • Nearly two-thirds (65.5%) of all desktop browser tests in the Sauce Labs cloud are run on Google Chrome. Internet Explorer was a distant second at 12.4%.
  • Among desktop browser tests, organizations in the technology and banking sectors account for 43.1% and 33.9% of all tests, respectively. Retail, media and healthcare round out the top five, in that order, but significantly lag the technology and banking sectors.
  • Among mobile tests, 77.9% are run on mobile web browsers, compared to just 22.1% on native mobile applications. Whether web or native, more than three-quarters (78.7%) of all mobile tests were run by organizations in the technology sector.

Data and Methodology
The data used in this report comes from the Sauce Labs Continuous Testing Cloud, the largest and most comprehensive testing cloud in the world, with more than 2 billion tests completed to date. This inaugural benchmark report analyses anonymized user data from June 2018 through December 2018.

In related news, Sauce Labs bought Screener, a provider of automated visual testing solutions, as the company continues to evolve to include non-functional testing alongside functional testing to deliver a more comprehensive experience to customers. Screener enables organizations to test their UI across multiple browsers, devices and operating systems to automatically detect visual regressions and spot inconsistencies, ensuring users have the right visual experience regardless of how they view an application.

“As digital leaders increasingly look to deliver apps that are fast, visually appealing, and functionally flawless, their continuous testing needs have evolved to include functional, performance, and visual testing,” said Charles Ramsey, CEO, Sauce Labs. “With the acquisition of Screener, Sauce Labs can now deliver on all three, and empower organizations to shift testing left to support their continuous integration and continuous delivery efforts. We’re thrilled to add Screener to the Sauce Labs family, and we look forward to integrating its many capabilities into our continuous testing platform.”

Sauce Labs today also announced the general availability of Sauce Headless, a first-of-its-kind offering that provides cloud-based headless browser testing on containers to facilitate fast and affordable early pipeline testing. Shifting left is quickly becoming the most important strategy development teams can employ to more effectively implement continuous testing throughout the rapidly evolving software delivery cycle. On the heels of the recently announced Sauce Performance, the acquisition of Screener and availability of Sauce Headless further establishes Sauce Labs as the leading provider of solutions to help developers shift both functional and non-functional testing as far left as possible.

Shifting Visual Testing Left
Many organizations are required to implement visual testing alongside functional and performance testing, and have historically used separate solutions to do so. Adding to this complexity, most visual testing solutions lack the integration points into the DevOps workflow necessary to truly support a shift-left testing strategy. Therefore, most organizations are not effectively testing visual components at the front end of the development process.

With the acquisition of Screener, Sauce Labs can now tightly integrate visual testing into functional testing workflows, enabling users to deploy both visual and functional testing without sacrificing the convenience and simplicity of working with a single vendor. In addition, Screener’s front-end visual component testing solution enables developers to test individual UI components to get the fast feedback they need in the early stages of the development cycle.

“As more code and complexity shifts to the front-end of the development process, visual component testing is quickly becoming a critical part of any comprehensive shift-left testing strategy,” said Loyal Chow, Founder, Screener. “We’re honored to join the Sauce Labs family and we’re eager to work with their world-class development team to continue delivering the solutions customers need to ensure a flawless visual experience for their users.”

End-to-End Plus Front-end Component
For any organization with a digital presence, the visual accuracy of an application has a meaningful impact on how customers interact with their business. With solutions for both end-to-end visual testing as well as front-end visual component testing, Screener enables users to:

  • Build and run their own visual tests
  • Automate test flows in minutes
  • Use DOM snapshots to identify the differences in an app by looking at both visual and programmatic data
  • Ignore areas based on element identification
  • Store baselines and seamlessly manage entire test lifecycle in the cloud

“As a leader in 3D design, engineering and construction management software, our business depends on delivering high-quality web and mobile applications to customers who rely on our software to support their business growth,” said Zohar Liran, Sr. Software Development Manager, BIM 360, Autodesk. “As a long-time customer of both Sauce Labs and Screener, we know full-well the value that comes from conducting both functional and visual testing to uncover issues early in the delivery pipeline. We’re excited to see how Screener’s solutions continue to evolve as part of Sauce Labs, and we look forward to the simplicity of working with a single vendor for both our functional and visual testing needs.”

Early Pipeline Testing with Sauce Headless 
An industry-first offering that supports shift-left functional testing, Sauce Headless enables development teams to get fast feedback on code and increase build efficiency by running atomic tests early in the delivery pipeline. It leverages headless Chrome and Firefox browsers on Linux in a container-based infrastructure so development teams can identify issues early and keep the pipeline moving by testing on every commit. By blending headless testing for quick feedback with cross-browser testing for full platform compatibility testing and debugging, organizations benefit from a single, unified testing experience that can scale instantly while reducing the time and resources needed to support and maintain their test infrastructure.

Terms of Use | Copyright © 2002 - 2019 CONSTITUENTWORKS SM  CORPORATION. All rights reserved. | Privacy Statement