Recently, one of our clients, that has been using WebdriverIO with Selenium WebDriver for frontend automation for a few years as a test engineer, was given the opportunity to move away from Selenium and, as a result, had to compare Selenium alternatives to complete testing (in addition to their usual stack of TypeScript and Node.js).
- Cypress as the most hyped automation tool
- Playwright as a more high-level tool that is close to the browser.
Puppeteer did not make their list since it is merely an API (Chrome DevTools Protocol) for controlling Chromium and requires extra testing framework (Jest, Mocha or something else). WebdriverIO can also be used with the DevTools Protocol, but our client wanted to try something new.
What is Playwright Framework?
The Playwright framework is a Nodejs-based open-source automation platform for end-to-end testing. Microsoft is in charge of its development and upkeep. Its primary purpose is to work on all major browser engines, including Chromium, Webkit, and Firefox.Capabilities Supported by the Playwright testing frameworkThe following functionalities are available with Playwright:
- Make scenarios that span several pages and domains.
- Captures network traffic in order to stub and mimic network requests.
- Mobile devices, geolocation, and permissions are all emulated.
- Used to download and upload files.
- Web automation is possible across browsers.
- Can access native mouse and keyboard input events
Advantages of using Playwright
Our client on his work comparing Cypress and Plywright found the following advantages after using Playwright:- Inspector is a graphical user interface (GUI) tool for debugging tests. Not as excellent as Cypress, but steel is still worthwhile.
- Playwright works quickly in headless mode (our client did not make accurate performance measurements, but it just feels faster than Cypress).
- Element auto-waiting before executing some activities.
- Visual regression testing works right away (in Cypress you have to use a plugin).
- Detailed HTML report that might be beneficial for both local and CI runs.
- Making images, videos, and traces on failures automatically are all useful for debugging failed tests, especially in Trace Viewer.
Playwright Disadvantages
- Documentation that is confusing. Without a question, a Playwright's documentation is excellent, but searching for knowledge takes effort, and examples are sometimes missing. Our client had to use Google and Stackoverflow after the installation procedure to find out some of the specifics.
- Despite the fact that Playwright supports API testing, its request mechanism is not sufficiently sophisticated. For example, it does not allow you to disable follow redirects (but Cypress does).
- Within one test() function, screenshots, videos, and failure traces are created. That is, if you have several tests within test.describe(), you will get videos and traces for each nested test, but not for the entire root test function (steps have the same behavior). It's a shame that so many relics can't be put together in one piece.
- For CI builds, TeamCity is not supported out of the box; you must develop your own reporter.
Playwright Features that are unusual to find:
- A Test Generator is a Graphical User Interface application for recording tests. Of course, the produced results cannot be utilized for real tests: the selected selectors are strange and must be modified, and expected must be manually inserted afterwards. However, it may be useful for developing preliminary drafts for simple tests.
- Custom commands are not supported, like they are in Cypress and WebdriverIO. Instead of constructing bespoke syntactic sugar around your testing infrastructure, Playwright encourages you to leverage Page Object Model.
- The majority of page and locator's methods only operate with the first element on the page. You must look for unique ways that allow you to operate with many items. Playwright forces you to use highly clear selectors, whereas Cypress or Selenium WebdriverIO do not.
- Playwright allows you to send any heading to the test page. In any case, you couldn't pass a single header. Cookies containing several cookies required our client to place each cookie individually. The same unusual thing happened with localStorage: our client couldn't transmit just one localStorage to the new browser's context, because creating a storage state requires the object of both necessary cookies and localStorage (and not just one thing). On the one hand, this is absolutely correct, yet it is an unreasonable restriction.
What exactly is the Cypress Framework?
Cypress Framework is an open-source web app testing automation platform. Cypress is used for end-to-end testing and supports JavaScript. It also works with the Mocha test framework.Cypress takes a distinct approach than other testing software. When you launch a script in the browser, it runs in the same loop as your application. Cypress uses the Node.js server to handle cases that must be run outside of the browser.
In brief, Cypress can provide more consistent results since it understands everything that happens both within and outside the browser. This also allows you to access each item natively without having to deal with over-the-wire protocols or object serialization.
Capabilities Supported by Cypress
Cypress has the following capabilities:
- Provides real-time app debugging by capturing snapshots as your tests run.
- Cypress network control enables you to quickly control, stub, and test edge situations without requiring a server.
- API testing capabilities.
- Capabilities for mocking and stubbing
- Component testing capabilities.
- Because Cypress does not employ Selenium or WebDriver, it delivers quick, consistent, and reliable tests that are not flaky.
- BDD/TDD style testing is supported.
- It has excellent debugging capabilities and developer tools. It generates legible errors and stack traces and speeds up debugging.
- Waiting automatically: Before proceeding, Cypress waits for orders and assertions.
- It is quick since tests are conducted within the browser like an app.
Cypress only supports Chrome, Firefox, and Edge browsers. This makes Cypress unsuitable for running cross-browser testing on browsers other than Chrome and Firefox, such as Internet Explorer and Safari.
As we discussed in a previous blog post, about Maintaining End-to-End Testing using Cypress with TestQuality, where we showed how one of our customer began building up test automation using Cypress once they had sketched out the manual tests they intended to run on a regular basis - roughly a hundred in total. Cypress is an excellent tool for performing end-to-end testing. It automatically records videos of test runs, and its test runner is Google Chrome, which almost all engineers are acquainted with. It also captures the activities it performed on the DOM on video, providing you with clear, visually debuggable recordings of failed test runs. It's a fantastic tool right out of the box.
Integrating Cypress Test Runs and Results with TestQuality
Advantages of using Cypress
- Extensive development experience! Running and debugging tests in the app is really convenient.
- Outstanding documentation! Cypress offers the nicest documentation you may imagine. During the installation and creation of the initial tests, our client just used documentation and examples from it.
- All in one instrument. Cypress includes a test runner, assertion library, HTTP request library, and many more features. There is no need to select several tools for developing testing infrastructure. It also allows plugins for additional functionality.
- Chai as a library of assertions (unlike Playwright with humble Jest expects).
- Getting elements and interactions with them include a built-in "wait till" for the element's actionable state.
- Access to network activities is simple. There is no need for external proxies to stub network requests and responses because it is built into Cypress.
- Custom command support is ready to use (and even types for you custom commands if you use TypeScript).
- Making screenshots on failures automatically.
Cypress Disadvantages
- Cypress has a large number of dependencies. Your node modules directory will become several tens of pounds heavier.
- It is not permitted to debug using console.log. You must use cy.log, which only requires a string.
- During the test run, all of the tests are somehow combined. This implies that variables with the same names cannot be used in various test files.
- Cypress did not start the first time I ran CI.
Unusual Characteristics that you may find in Cypress:
- Electron is the default "browser." Despite being based on Chromium, the Electron environment is not the same as a pure browser. For example, when our client tested web app began acting erratically and throwing console errors, he had no choice but to switch to Chromium – fortunately, Cypress enables cross-browser testing on Firefox and Chrome-like browsers.
- During the test run, it looks for uncaught exceptions from your web app. It could come in handy if your project has a zero tolerance for terminal mistakes.
- Tests may only run in one tab. Cypress does not support multi-tab setups.
- Cypress has its own set of environment variables. If you're used to utilizing Node.js' process.env variables, you'll need to swap them out for Cypress' and include them in the config file.
In conclusion
Playwright is a more sophisticated tool with specific controls that works closer to the browser. Cypress is a more adaptable tool with a predictable API for testers, allowing them to create tests quickly and easily.Before deciding on an automation tool, examine the benefits and drawbacks of Cypress vs. Playwright or Selenium. Spend some time automating some test cases using a cloud-based platform that supports Playwright and Cypress. This allows you to choose which test automation framework will best benefit your team in the long run.
TestQuality plans in its integration roadmap to offer support to Playwright but you may choose Cypress because of the benefits and advantages mentioned above and also it can be love at first sight when you first start using Cypress and discover that integrating TestQuality with Jira or GitHub also works wonderfully in CI with little configuration!
Sign Up for a Free Trial and add TestQuality to your workflow today!
Add TestQuality to your workflow today
TestQuality can simplify test case creation and organization, it offers a very competitive price but it is free when used with GitHub free repositories providing Rich and flexible reporting that can help you to visualize and understand where you and your dev or QA Team are at in your project's quality lifecycle. But also look for analytics that can help identify the quality and effectiveness of your test cases and testing efforts to ensure you're building and executing the most effective tests for your efforts.
