However, as you develop your product and gain customers, you will need to expand your testing infrastructure or risk losing customers and acceleration. You must also consider the context – you will be doing this while also escalating your engineering team. At this moment, End-to-End Testing provides the most value for the money, but it is also one of the most difficult types of testing to master.
This article describes a case study of how one of our clients transitioned from manual, mostly exploratory end-to-end testing to fully automated, stable, per-change testing.
What's the big deal about end-to-end testing?
In that order, the most difficult restrictions on end-to-end testing are stability and time.
End-to-end testing of any system, whether it's a mobile app, an embedded device, or a SaaS web platform, is inherently unstable because you're not testing a closed system. Network connections are highly unreliable, sensors in mobile devices do not always work and are susceptible to interference, and simulated actions occur at a different rate than those of a real end user. For any realistic end-to-end testing environment, these open-system challenges persist.
On top of that, your software will have heisenbugs – those rare, unlikely-to-occur bugs that never seem to occur when you run the same test case manually. When running end-to-end tests at scale, however, these bugs will occur so frequently that your engineers will begin to distrust the entire end-to-end testing infrastructure.
The other major constraint is time – end-to-end testing takes a long time for a variety of reasons. Your platform will be slow – much slower than a mocked-up unit test. The network will be sluggish. It's possible that the hardware you're using is slow or broken. Because of the stability issues, you may need to add a lot of waiting time to your testing code, causing further test setbacks.
All of this can lead you to believe that end-to-end testing is flaky and not worth the effort. It is, however, the only way for you to test the actual end-user behavior of your platform. If you don't do it, you'll have high defect counts, reduced engineering velocity, and dissatisfied customers.
Breaking down the problem
Iterating to excellence is one of our client's core values. That means getting to value quickly. Even if the first release isn't perfect, it's sufficient if it generates real value. In that context, they set out to solve the end-to-end testing problem by first dividing it into several sub-problems:
- Selecting an end-to-end testing framework and transforming critical tests into automated ones that run stable in local and CI settings.
- Creating the real tests and manually running them.
- Executing the tests in a timed CI job against staging.
- Executing the end-to-end tests every PR as part of the CI pipeline.
Below in this post, we walk through how our client got the tests running stable.
Begin with manual tests.
"You're not going to need it" (yet) is one of the well-known extreme programming methods and startup culture adages. Building extensive end-to-end testing infrastructure is pointless unless you know exactly what you're doing.
Our client began developing manual end-to-end test cases early on. They began by developing around 40 test cases, which they manually performed. The engineers and all of us might think this is a bad idea — we need to automate this extra effort! However, there are some compelling reasons to continue to manual tests at the begining:
- You will not have time. Firstly, you should be iterating on the product rather than creating an extensive test framework that would consume hundreds, if not thousands, of engineering hours over many months. Building out test automation will generate profit eventually, but not immediately.
- Manually running tests forces all engineers to become acquainted with the platform. Engineers are not the primary consumers of most software, but when they write code, they get to make minute product decisions every day! As a result, having a deep product awareness throughout the engineering team is critical, and use case-oriented test cases that must be manually executed can aid in this process.
- To create excellent end-to-end tests, you must first create manual tests. You must organize, prioritize, and write effective test cases that cover the essential functionalities of your platform.
Because you can't construct end-to-end tests without the last element, you may as well start with manual testing and define a procedure in which all engineers run the tests on a regular basis. This may seem contradictory – after all, being a software firm, and this activity should be automated – but it's the first step toward iterating to perfection and you'll make it!
The initial manual testing procedure
Here's how we define manual testing and the methodology that goes with them:
- The manual testing process is assigned a single owner, and the product management team identifies the key use cases.
- Engineering creates tests based on primary use cases.
- The tests are classified by application area and prioritized in the same way as bugs are classified: Blocker/Critical/Major/Minor. If a manual test fails, the failure is reported as a defect based on the importance of the test.
- Every deployment runs a portion of the tests. We limit manual test runs to 30 minutes, focusing on the most important non-automated tests.
A manual testing tool such as TestQuality was a great option to define their first tests. There was no need for more sophisticated tools, but it is something that can be scoped out during the original implementation.
What you should not omit is the creation of explicit preconditions, processes and expectations for your tests. When developing a test case, don't omit any steps that the user would need to do — instead, write each click and typing effort as its own line item. This contributes to both clarity and a clear idea of how simple it is for the end user to do an action.
At the very least, each test has a name, priority, anticipated outcome, and preconditions. The processes must be explicit since you want the team to be able to participate in the testing effort and afterwards disseminate test implementation. For these reasons, TestQuality was the right tool to start with not only to work with manual tests but also, with automated testing at a later stage.
Planning ahead of time with manual testing
You'll find yourself creating a lot of repetitive user activities when you define manual test cases and associated processes. It's tempting to reduce complicated user activities to one-liner items, but keep in mind that the user will most likely repeat the actions. If you find yourself repeating a long series of steps to complete an essential task, there is usually opportunity for improvement in the product itself! Our customer was able to uncover opportunities for improvement by recognizing these hidden complications in user processes as we built test cases.
End-to-end test automation that is easy to maintain
Our customer began building up test automation using Cypress once they had sketched out the manual tests they intended to run on a regular basis - roughly a hundred in total. Cypress is an excellent tool for performing end-to-end testing. It automatically records videos of test runs, and its test runner is Google Chrome, which almost all engineers are acquainted with. It also captures the activities it performed on the DOM on video, providing you with clear, visually debuggable recordings of failed test runs. It's a fantastic tool right out of the box.
Integrating Cypress Test Runs and Results with TestQuality
Selenium, a battle-tested browser automation technology, is used elsewhere at our client for process automation. They evaluated it as a way to reduce the number of technologies in the stack, but chose Cypress because of the benefits listed above. It was love at first sight when they first started using Cypress and discovered that integrating TestQuality with GitHub or Jira also worked wonderfully in CI with little configuration!
Their Cypress tests are comprehensive end-to-end tests; nothing is mocked out. They have a staging environment where the tests are conducted, complete with their own clones of the databases and other services required by their platform. The tests were to be written in such a way that they resembled human-readable test stages as much as feasible.
Most of the time, you're reading through a failed exam. That means you have a problem to solve, and understanding the meaning of the test should be the least of your worries. A good test case should be self-documenting and easy to understand. It is also important to consider the reusability of testing code. Assume you have a top bar in your app that allows users to choose between experiences; you'd want to reuse the code that enables those actions.
Finally, certain activities appear easy to the user but are sophisticated on the inside. Consider a search bar: when a user clicks on it, it may need to fetch content recommendations from the backend. An asynchronous loading operation is involved behind this click, and you'd almost likely want to encapsulate that type of sophisticated testing code inside a single function..
Something like this makes a lot of sense for the reasons stated above:
interface TopBar {
SearchBar getSearchBar();
}
interface SearchBar {
void type(text: string);
void selectFirstAutocompletionSuggestion();
void selectAutocompletionSuggestionByIndex(index: number);
}
This method allows you to hide the loading activities behind type and/or selectAutocompletionSuggestionByIndex implementations. When you're finished implementing the test, it may look like this:
Cypress.App.TopBar.getSearchBar().selectFirstAutocompletionSuggestion();
Cypress.App.SideBar.getSelectedDashboard().should('contain.text', 'Adios');
This clearly communicates to the reader what is being clicked on and what the intended response should be!
Waiting mechanism that is unique
Our client required a bespoke waiting solution since their program loads a lot of data on the go and they couldn't predict when these asynchronous operations would be finished. After they finished the first round of Cypress tests, they mainly functioned OK, but every now and then, an awkwardly timed asynchronous request would be delayed, causing problems.
Our customer discovered that in Cypress, waiting for HTTP events is not totally stable. They used cy.route().as() and cy.wait() both, but found them to be unreliable – the waits didn't always happen, causing test execution to fail as things moved past the waits before the app was in the correct state. They explored several solutions but couldn't discover one that genuinely made things steady. To compound matters, cy.route() is also not conceptually correct when designing end-to-end tests. What you want is a UI-based wait condition, much like a real user would. And, in our experience, these UI-based delays are rock reliable in Cypress.
ActiveRequestIndicator has one more surprise up its sleeve: while decrementing, it debounces the changes to the active request count. That is, when the number of active requests approaches zero, the visual signal is deleted only a few seconds later. This allows ample time for any asynchronous activities that may occur after the data fetching is completed.
Finally, Cypress tests can wait for current requests to complete, in addition to any other waits you may have set up. They express it in their Cypress framework code as follows:
export function waitUntilReady() {
cy.get('.Loader:visible', {timeout: 60000}).should('not.exist');
cy.get('.active-request-indicator-item:visible', {timeout: 60000}).should('not.exist'),
}
That's it! Individual tests in our client's Cypress tests utilize only this one waitUntilReady function, so test authors don't have to worry about which background requests will fire. All the test creator has to worry about is when to wait, not what to wait for. Most importantly, this technique is consistent.
Cypress Test Run using a GitHub Cypress Test examples repository
Creating the proper tests
They'd done all the work to create the test cases, prioritize them, and sort out the ones that are difficult to automate by the time we came to creating tests. As a result, developing tests became a matter of turning test cases into Cypress cases.
They indicated which tests were automated and which were not so that they could maintain a single source of truth for all test cases while the test generation activity was continuing. At this point, TestQuality excelent features helped to create and organize test cases in a global test repository - with preconditions, steps, attachments, and more. All this, in a collaborative testing environment seamlessly integrated with their DevOps workflow with powerful live analytics to help them to identify the quality of their testing effort, test coverage, high value tests, unreliable tests, and release readiness.
Sign Up for a Free Trial and add TestQuality to your workflow today!
Some of the manual test cases were found to be unsuitable for automation due to a high initial investment. Our client's chart and map widgets, for example, cannot be readily tested automatically since they rely on pixel-perfect cursor location and clicking on items for an action to occur. There are many better bang-for-buck test cases available, so we swiftly ruled these out as part of the first pass. Prioritizing which tests to write may make a significant impact in terms of both stability and implementation time.
Analyzing Cypress Test Runs with TestQuality's analysis and insights
Establishing standards from the start
For the majority of the first tests we created, we also created new elements for the testing framework. This made the first tests take longer to write, but it also made future tests considerably faster to implement and more stable. For example, when they developed the first test that enabled the user to update the dashboard's filters, they needed to include a reusable helper function like addInclusiveFilterByPasting(). Writing these auxiliary methods in a reusable manner takes some time at initially, but it is vital.In addition to code reuse and maintainability, implementing reusable helper methods to perform user-facing tasks provides predictable structure and consistency. When speaking with others, we all follow pre-existing patterns that we see and hear. This includes coding, which is also a type of communication. It was critical for us to establish a firm baseline of what testing should look like. It has both invisible and powerful impacts, such as lowering the amount of labor required for code reviews later on and boosting test stability over time.
Setting standards for code style, architecture, and patterns becomes even more critical when new components and ways of doing things are introduced into your system. Cypress and end-to-end tests were new additions to our client's software, and carefully and methodically establishing out the test framework helped them set themselves up for success.
What might we have done better?
While we are really pleased with the end outcome, a few things did not go as smoothly as they had intended. The issues they encountered had less to do with technology and more to do with process.The initial effort was inadequately
Our client first used Cypress e2e testing in the summer of 2020. They spent three engineering weeks on this – not a long time, but significant for a company their size. This work was for naught since they never got to the point where the staging CI was up and running and the tests were being executed. They also did not implement a manual method, such as running the tests on the developers' PCs on a regular basis. They might have gotten more value out of the tests a year earlier but failed to do so due to project management issues. Simply simply, they began too soonThe good news is that much of the major technologies mentioned here was part of the first upgrade, which considerably decreased the difficulty of getting the tests running again - there wasn't a lot of code rot. However, they should have completed the task completely. If they could go back in time, they would have instituted a manual method and worked their way up to getting the tests to function.
Instability in tests is caused by a too relaxed approach
Our client enabled the tests in the CI after they had them running pretty smoothly. For the most part, this worked well, although there were a few residual test instabilities that were not entirely resolved. During one week, this caused numerous PR verification builds to fail, resulting in hours of waiting for a green build result for new changes. They handled this by enforcing the rule that all e2e test failures are automatically critical problems if they prevent PR verification builds from running, and by releasing a patch that disables the test until the critical bug is fixed. This performed quite well and aided them in resolving the remaining difficulties.If they could go back in time, our client would have implemented the aforementioned restrictions at the same time they enabled the end-to-end testing on a per-PR basis.
Summary
Here's a rundown of everything our client learnt. If you're starting from scratch with e2e testing in your business, this might be helpful:- Make regular progress toward your goals, and be willing to accept "sub-par" solutions, especially if they require human labor.
- Have a concrete plan that you are executing on that will assist you in reaching the per-PR state.
- Consider the testing procedure to be an important aspect of your journey. Who does manual tests? When are manual tests run? How do you keep track of them? What happens if one of the tests fails? Who reports the bug? What is the top priority? Who's going to fix it? Ideally, you'll have clear answers to these and many more issues right away.
- Request assistance from your product team in prioritizing the test cases; they should understand which aspects of the user experience are crucial.
In terms of technology:
- try to keep the e2e testing as close to user-facing as feasible. This aids with maintenance, readability, and establishes the appropriate standards for the engineering team.
- Don't be scared to incorporate e2e test support right into your API. If you require some UI that is only visible in e2e testing, that's OK — it's a key use case that has to be addressed.
Add TestQuality to your workflow today
TestQuality can simplify test case creation and organization, it offers a very competitive price but it is free when used with GitHub free repositories providing Rich and flexible reporting that can help you to visualize and understand where you and your dev or QA Team are at in your project's quality lifecycle. But also look for analytics that can help identify the quality and effectiveness of your test cases and testing efforts to ensure you're building and executing the most effective tests for your efforts.
