Overview
A leading provider of government sponsored healthcare sought to rapidly expand their IT Teams to enable supporting necessary agile software updates to changing business needs but found that their manual testing capabilities could not keep pace.
Challenges
The company quickly determined that using an Agile Methodology approach as well as automated testing should provide the necessary speed and quality results. However, the company identified multiple challenges related to planning and implementing an automated testing approach including:
- Insufficient knowledge to determine which automated testing tool would best fit their needs
- Lack of experience organizing a detailed automated planning approach
- Limited understanding of the platform needed to reliably execute thousands of automated testing cases nightly in a limited period of time
- Limited test data available to support repetitive test case execution on a daily or more frequent basis
- Insufficient knowledge to estimate the benefits of implementing an automated testing process at this scale
Solution
To help our client meet these challenges, we worked with them to first identify the automated testing tool that would best meet their specific needs. Multiple tools such as Selenium, QTP and Ranorex were evaluated including vendor demos to determine best fit. After taking into account the requirements and constraints including HIPPA and compliance needs, Ranorex was chosen due to its capabilities to automate Web, API and Mobile automation support.
Once the appropriate tool was selected, the team focused on understanding the chosen tool and how to use it to develop the appropriate automated testing approach. Ranorex was introduced to multiple Agile teams with education and support provided. What followed was a six-month introductory period from which a new challenge arose.
Acceptance and adoption of Ranorex was being constrained primarily due to the tool’s reliance on users having at least a limited understanding of C#. Ranorex provides C# wrappers to reduce the programming demands but that support was proving insufficient. To address this constraint, the client created a QCoE (Quality Center of Excellence) and a C# team was pulled together to develop a framework including additional wrappers for the most common automated testing needs to further ease the setup process. Libraries were created and provided as part of Ranorex installation and QCoE members were imbedded for short stints on individual agile teams to assist those teams in understanding and utilizing the new capabilities. Additionally short learning videos were created and uploaded to the internal website to increase acceptance and appropriate usage.
With the additional processes and support mechanisms in place and the barriers to usage reduced, the teams quickly adopted the testing tool and its usage increased rapidly, leading to several additional challenges. First, the adoption of the automated testing approach meant that 100s of new test cases were being produced each week each of which was expected to be executed unattended on a nightly basis leading to overly long testing times. To address the issue,
several options were explored including utilizing the Sauce Labs cloud testing platform but HIPPA and compliance requirements again limited cloud based solutions. Ultimately the QCoE decided to create its own Software and Hardware platform to run the growing batch of test cases unattended and in parallel. This platform has been a success and continues to be extended to meet new and evolving needs including supporting test suites from other tools such as Cypress and Selenium.
The second challenge created by the increased testing efforts involved building and supporting sufficient test data to meet the repetitive testing needs. Specifically, as the teams continued to utilize automated testing, they found that they quickly exhausted the available test data or struggled to find existing test data necessary for the growing set of test cases. These limitations were made visible as teams slowed creation of new test cases and runs of existing test cases, as monitored by the new test platform. To address this challenge, the QCoE adopted a TDM (Test Data Management) strategy utilizing Talend to identify existing test data and subsequently create synthetic test data. The team utilized Talend to create test members, claims and more to ensure appropriate and sufficient test data could continually be synthesized.
As a final challenge, the client continued to acquire health plans with the need to ensure their systems supported their new members at the same high quality as their existing membership. While the automated testing systems continued to support this growth, we worked with the client to assemble an End-to-End test validation team focusing on testing a smaller set of end-to-end suites to ensure both new and existing members were supported. This team provided value in ensuring the systems supported all members and provided leadership with the confidence that rapid changes were appropriately being tested before moving into their Production environment.