Articles about Software Testing

The changing role of test automation engineers Due to the increase in paired programming, the technical skill helix of development and testing is becoming more apparent. This also increases the industry’s need for automated tests and the implementation of scalable Agile teams. And in recent years, the competency and responsibilities of a test automation engineer have shifted, enhancing test automation engineer’s engagement in their Agile team earlier in the Software Development Life Cycle (SDLC). Our scenario: Your company has a customer-facing mobile application, and its revenue is generated from advertisements and subscriptions. When selling ad space, you are contractually obligated to display the correct banner ads to targeted subscribers. So, ensuring a satisfactory level of coverage for various subscriber/ad combinations is extremely important. Often, in this type of situation, organizations will hire an outside consultant who specializes in automating tests at various software layers. Outside of the traditional duties, the hired test automation engineer for your mobile application would also now be responsible for developing and maintaining a test framework. Below, are some software levels that the test automation engineer can create, add to, or maintain for a similar mobile application under test in this scenario. Layer 1: Unit Tests Since they are testing against actual application code, unit tests are the quickest and first software layer in the testing process. These tests are often written by software engineers to ensure that the isolated modules, features, and functions of the mobile application are performing as expected. These may or may not be already implemented, but for the sake of this example, let us just say that they are not. Therefore, the Test Automation Engineer should ask to be included in the planning, design, and execution of the modules under test. This can be done in practice by paired programming, which will enhance inclusivity and camaraderie by working toward that common goal of delivering a quality product as an agile team. Here is a JavaScript code example, showing how unit tests could be set up via Protractor and Jasmine framework (powered by the test runner Karma) for an Angular website. The landing page is where end-users would typically first encounter the website, so the tests below ensure that all elements, headers, and other content is visible to the user. Layer 2: API (Application Programming Interface) Tests This second software layer’s popularity has risen in recent years due to the flexibility, coverage, and overall ROI of tests. The domain layer is where the implementation of the application code, service(s), and database intertwine. These tests usually consist of performing network calls that typically cover which users are authorized/authenticated, what type of calls are allowed, and how data can be passed and returned to the services(s) applicable, typically via REST (Representational State Transfer) API. The variances of how these tests can be created and maintained are vast, depending on the Test Automation Engineer’s competency. The following code example shows an API test written in JavaScript on the Postman application. The data set was read randomly from various collection variables thanks to the switch statement. This test could be run by the Collection Runner in Postman or on a command line interface through Newman. Regardless the data would come from a CSV file. Layer 3: UI (User Interface) Tests The third software layer is where the application’s code is presented. This is where actions can be performed by a human, such as inputting and submitting data through fields and forms, clicking buttons, and engaging with content or other users. The goal of these automated tests at the UI layer is to explore creative and counterintuitive actions that may result in errors. Thus, this layer takes the longest to test, and it is impossible to cover every scenario for many reasons. (On this subject, my colleague Alec Langhammer’s thoughts on “hybrid testing” are worth checking out.) Lastly, some may consider testing at this level as end-to-end (E2E) due to the nature of complete integration required for the interface to perform and present results to the consumer. These tests also can be fully coded, or hybrid coded/record-playback (i.e., cloud-based platforms such as TestProject, Testim, and mabl just to name a few). This last code example, written in Python and powered by a Selenium WebDriver, was created to test the User Interface of a website. It covers interactions with fields by inputting unique information and the functionality to submit the information entered. (Please note what is not listed here is the entire test suite, Document Object Model (DOM), and input data library. this is only one regression script.) Closing thoughts These layers are best optimized when automated with low-level code (easily readable and understandable) utilized within frameworks and libraries that are hosted on cloud platforms that contain an organized source-controlled repository attached to an integrated pipeline controlled by the testers. As a reminder, there are many other common practices and methods that aid the overall goal of testing. Don’t think of this as an exhaustive approach to testing but a coherent and comprehensive testing model that is unique and aligned with the example mentioned in the introductory paragraph. Lastly, don’t overlook the value of exploratory testing, which is still very necessary in ensuring the functionality of the application/platform. This article simply outlines how this three-layered approach can provide more time to balance out covering client requirements and creative edge case scenarios that are not easily replicated with automated testing at any single layer.

In our previous article, Testing in the DevOps Pipeline, we discussed release pipelines and the automated test cases which help make them a reality. We started with these topics because they touch on some of the more exciting and new ways that adopting DevOps methodologies impacts the entire software development lifecycle. Being able to automatically build, test, and deploy code changes at the touch of a button has transformed many historically-more-manual processes, like testing. In this brave new world of automated tests and deployment is there still room for more “traditional” testing practices? And, does having more automated testing lead to more frequent and higher quality releases? Not a Magic Bullet While the ability to rapidly repeat tests for each new build — or on a schedule — provides a high level of confidence in system stability, automated testing should not be seen as a magic bullet. Automated tests provide their greatest return on investment when they are reserved primarily for regression and smoke test suites while leaving most new feature testing manual. When determining which test cases should be automated and which should be left manual, the upfront cost of automation can’t be ignored. In the time it takes to go through the entire process of creating a new automated test for a single new feature, a manual tester could have verified multiple new features. With the upfront cost of time in mind, we recommend automating test cases that are either mission-critical or that are going to need multiple repetitions and keeping one-off feature testing manual. A hybrid approach While many within the industry believe more automation leads to faster releases, we find that a more “hybrid approach” often leads to faster and higher-quality releases. In fact, some of our clients that have managed to make multiple daily deploys have done so by committing the resources needed to fully adopt the hybrid testing approach. These clients have established a separate Scrum team solely dedicated to the development of testing frameworks and automated regression and smoke test suites. Having a team dedicated to this work increases confidence in the overall system stability with each new build, freeing the feature teams to concentrate on new features. As a result, the testers on the feature teams have the overall system stability they need to confidently push out new builds and not worry about attempting to automate incremental releases. In doing so, the teams are able to get new builds out much more quickly, deploying multiple production changes during work hours each day. Additionally, it can lower the cost and technical background needed for testers on the feature teams; instead of an SDET on each team, we can have a few SDETs on the framework team and primarily manual testers on the feature teams. The test automation pyramid As part of evaluating the cost of automation, another model to keep in mind is the test automation pyramid. This model is a way of visualizing which aspects of the system under test should be automated the most. Source: MartinFowler.com Unit tests constitute the base of the pyramid and receive the most automation, while User Interface (UI) tests receive the least. There are three main metrics that collectively determine how much automation should be used for a given area of the system: How long it takes to write the test How long it takes to run the test How much effort is needed to maintain the test Unit tests are relatively quick to write, extremely fast to run, and are reliable tests that should be updated as part of the development process. On the other end of the spectrum, UI automaton can take quite a while to write. Running tests that require WebDriver instances to open browsers and navigate web pages is a slow process, and the nature of UI automation is particularly brittle and requires a lot of maintenance. For these reasons, we encourage an abundance of automation at the unit and service layers while UI automation should be reserved for rarer occasions that justify the upfront and upkeep costs. However, while we discourage being overly zealous with automation at the UI level, that doesn’t mean it shouldn’t be tested; this is where our traditional testers once again have a crucial role to play. Where do testers test? Answer: everywhere In our previous article, we talked about the benefits of release pipelines and the automated testing that gives us the confidence needed to use them. While these practices have been incredibly helpful, automating and streamlining the deployment process, there’s still often a need for manual verification. As a result, most of these pipelines do not push directly to a production environment. A common pattern illustrating the typical flow of a build from its creation to final deployment in production might look something like this: Dev -> Testing -> Staging -> Prod. Each of these stops in new environments serve as manual gates where someone (usually a tester) must sign off on the build before allowing it to continue to flow through to the higher environments. Let’s take a look at how a new build might flow through this process. A build is created by a developer and tested locally in a dev environment. At this stage, the developer is expected to execute unit tests and perform additional cursory testing before pushing to a testing environment. This deployment process should be automated via the help of a build server (e.g. Jenkins) and include automated smoke tests that ensure basic stability of each new build. Once deployed to the test environment, the tester can now run any automation that is not part of the pipeline and begin the process of manual verification. These intermediate environments between the developer’s local environment and production are crucial to the pragmatic approach of blending manual and automated testing. The automated smoke tests in the pipeline during deployment to the intermediate environments gives confidence of general build stability, while the feature-specific manual testing gives confidence to the new feature itself. Should any of the automated tests fail during deployment, or if the tester finds a bug in an intermediate environment, an issue is raised with the developer and the process starts again from the beginning. Once the feature team tester has signed off on the release, the build can continue to a staging environment. Staging environments mirror production and, as the name implies, are used to stage changes before they finally go live. In blue/green deployment environments, the staging environment will be the currently inactive server cluster. The release build is deployed to the inactive cluster, and we are provided with a final opportunity for manual testing before going live. If everything continues to go smoothly in staging, the clusters are flipped, the inactive servers go live, and the previously-active production servers become inactive and ready for use as the next release’s staging environment. Even after the build is live in production, the job of the tester is not yet finished. Final manual testing of the build and monitoring for any errors or outages should be standard post-release procedure. Isn’t this risky? Monitoring your system in production for outages and increased error rates is a critical component to successfully deploying more frequently. While most of this article has explored the benefits of manual testing new features, the potential risks of not automating these tests should be acknowledged. Any features not included in the automated regression tests are features that could possibly break with future builds. Monitoring for outages and increased error rates is a way to mitigate the risk of any issues going unnoticed. This idea of increased risk might make some uncomfortable, but the unfeasibility of full test coverage gets at the heart of what makes a good tester. Testers have long understood that it’s impossible to exhaustively test a non-trivial system, and therefore the role of the tester is to analyze risk and come up with the most pragmatic test plan. There is inevitably a tradeoff between the risks of missing a bug and the time-to-market needs of the organization. Expecting to automate every test will cause unnecessary delays in the process and may even decrease the overall quality of the system. Now more than ever The skills testers have cultivated for decades regarding risk analysis and test coverage are needed now more than ever and should not be ignored in an attempt to automate everything. By extending those skills to the decisions about which tests to automate, it’s possible to create a test plan that pragmatically blends the best of both automated and manual testing and that mitigates potential risk with post-release monitoring.

Are you currently using a DevOps pipeline, or moving that direction? Are you looking to integrate testing into the release pipeline? In this article, we’ll briefly discuss DevOps pipelines and give some hints on how you can test in such an environment. Joining the DevOps moment As you participate in the never-ending dance of software development and releases, you’ve probably come across the term “DevOps.” Though it sounds like an elite team of special operatives, the portmanteau is more accurately described as a movement to blur the lines between the traditionally-siloed worlds of development and operations. In an on-premise software-deployment environment, isolating development and operations has made sense. Operations can focus on releasing the software to the appropriate customer through a medium that users can consume. The developers are 100% dedicated to adding new features and making sure they don’t break the build, and they don’t need to worry about the details and demands of delivering to individual customers. Yet software is increasingly moving “to the cloud.” With cloud apps and services being deployed to locations hosted by software companies (rather than users), the load of operations is eased. Simultaneously, this movement is eroding the argument that development and operations should continue to be siloed. “DevOps pipelines” are a powerful tool when uniting the two functions. With the adoption of continuous integration/continuous deployment (CI/CD) with tools such as Jenkins, Travis CI, Gradle, and Bamboo, you can put together a sturdy, reliable pipeline to handle everything from initial build to release. Developers can now push to a code repository and then watch an entirely automated process release that code into your desired environment. Common Pipeline Components and Corresponding Tools Is quality going down the drain with a pipeline?? If the software-release process is handled by an automated pipeline, is there still the opportunity to test the code before it’s released into the wild? Do I still need testers? Absolutely! Adopting a pipeline does not mean software testing should be sacrificed to streamline the release process. The very basis of DevOps pipelines is to make the release process reliable and easily repeatable while reducing the chances of errors, delays, and miscommunication. All of that contributes to higher quality products, right? New or existing automated tests can be easily and smoothly integrated into the pipeline. Unit tests, smoke tests, and regression suites can all be run as part of a release pipeline. The testing is not limited to API tests, either. CI tools such as Jenkins or third-party testing platforms like Mabl can run your UI tests in “headless mode” to avoid interrupting team members’ work whenever the pipeline runs. “Continuous testing” is a significant opportunity with CI/CD. Let’s walk through a sample scenario. Say several developers have reviewed and approved a code merge to the develop-branch of a code repository. Once the code is merged, the CI/CD tool picks up the changes in the code repository and tells the build server to build the code. If the build is successful, the code is ready to be deployed by another tool. Unit tests (provided by the developers) should be run by the build server as a sanity check. Then, a suite of automated software tests can be kicked off against the successful build as a further check that the code changes have not broken existing features. If any of these steps fail, the pipeline will notify the responsible parties that it’s a breaking build, and deployment is terminated. A successful run will result in deployment to the dev environment. The pipeline is flexible The scenario above describes only one potential setup. The multitude and variety of available tools allow for incredible flexibility! If manual tests are required, the pipeline can be configured to build the code and then wait until the appropriate user approves the deployment. Does this defeat the point of using a pipeline? Not at all! The pipeline keeps all of the steps in order, and any part that should be automated in your specific system can be automated. Steps won’t be forgotten or completed out of order. Logs and results are kept in the pipeline for future reference and easy visibility into the status of releases. So where do I start? You could certainly begin by compiling a pipeline from multiple tools. Many of the individual tools have excellent documentation with notes on how to integrate with other components. If you are new to creating a pipeline, however, we’d recommend checking out GitLab. It is a single tool that can handle source code management, continuous-integration/continuous-delivery, builds, and releases. With GitLab, your pipeline can be built & managed in a single place. Your software can be tested on the pipeline using your automated test suites stored in GitLab, or by running a command to kick off tests living somewhere else. The source code management is based in Git and is very similar to GitHub, with one prominent difference being the use of “merge requests” rather than “pull requests.” Same basic concept, just different terminology. Example of GitLab’s pipeline diagram, displayed while running Another popular CI/CD tool is Jenkins, an established and well-supported automation server. Like GitLab, Jenkins provides a UI for pipeline dashboards and interaction. Jenkins, however, is not an all-in-one CI/CD solution. While GitLab hosts the resources it uses in the pipeline, Jenkins is more of a coordinator that calls different pieces of your pipeline from wherever they are hosted. It is an exceedingly useful and time-tested tool for scenarios where you want to utilize several different platforms in one pipeline. Jenkins can be configured to build, test, and deploy code once it is pushed to a repository. Jenkins dashboards can show multiple projects in one view If the pipeline fits… As support for DevOps continues to grow, so does the number of available tools for adding automated tests into your pipeline. A well-built pipeline is incredibly flexible and should never require teams to reduce software testing, even if some of that testing is manual. Tools such as GitLab and Jenkins provide an excellent starting place for building a sturdy software-release pipeline as you begin your DevOps journey. So, go ahead. Put on those coveralls and grab your wrench. It’s time to build your software release pipeline and join the DevOps movement. And if you’re struggling with where to start or need assistance, don’t hesitate to get in touch. We’re always here to help!