e2e frameworks
WebdriverIO vs Cypress vs Puppeteer
Pros
WebdriverIO
Cypress
Puppeteer
WebdriverIO can be run on the Selenium WebDriver Protocol as well as Chrome DevTools Protocol through Puppeteer
WebdriverIO comes with integrated support for Applitools Eyes allowing you to write seamless visual regression tests
Multiple reporter support such as Video Reporter, Report Portal Reporter and Allure Reporter and mo
Selector API similar to jQuery API is very convenient to use (parent, closest, etc). Probably, Cypress has the best Selector API from all JavaScript E2E tools.
Each command and assertion, when hovered over, restore the Application Under Test (righthand side) to the state it was in when that command executed. This allows you to ‘time-travel’ back to previous states of your application when testing
Puppeteer is based on a protocol that interacts very closely with the browser so that you can take measurements of performance, work with extensions, create any number of browser tabs. Many other tools cannot do this
Cons
WebdriverIO
Cypress
Puppeteer
Running tests through a Selenium WebDriver Protocol takes longer because Selenium adds a layer between tests and the browser. Running tests through a Chrome DevTools Protocol has the same cons as Puppeteer
Cypress is relatively new, and it does not have the vast community that Selenium does
Because it runs inside the browser, you won’t be able to support multiple tabs
Cypress Command API a bit unobvious because syntax looks like synchronous code but it asynchronous in fact. We can not use async/await
Selector API not very convenient because you can find nodes by test only with XPath. Also, most of the helpful CSS pseudo-selectors do not work. Have to use XPath a lot.
How it works
WebdriverIO
Cypress
Puppeteer
Selenium WebDriver Protocol or Chrome DevTools Protocol through Puppeteer
WebdriverIO is always up to date with the latest automation frameworks and therefor supports not only capabilities of the WebDriver but also commands of the Chrome DevTools protocol using tools like Puppeteer. The framework allows you to freely switch between running remote WebDriver commands as well stubbing and mocking features of Puppeteer. Have a look into the examples directory.
Node.js based client-server architecture
Cypress runs test code written by the user in the Node.js environment on the "server-side". Scripts that emulate user activity (click, focus...) run inside the tested page in the browser on the same device. This is the "client-side".
Puppeteer is a Node library that provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default but can be configured to run full (non-headless) Chrome or Chromium.
Desktop browser support
(It can be helpful to run tests in various browsers to be sure that functionality is working the same way for any browsers)
Maybe we do not need to run E2E tests in all browsers. Maybe we want to create visual regression tests instead of cross-browser testing by E2E tests. E2E tests check only functionality, but cross-browser issues relate to markup or CSS problems in general. Even if our E2E tests passed on all browsers that do not mean that the user will not meet some bugs in markup because E2E tests tied to some DOM attributes like `data-e2e`. But if tests found these attributes in DOM that does not mean that the user will see the same picture. Maybe we should split tests into E2E (for functionality checking) and visual regression tests (for cross-browser issues checking)
WebdriverIO
Cypress
Puppeteer
Chrome
Internet Explorer
Safari
Opera
Firefox
Edge
All browsers which supported by Selenium are available
But if you use Chrome DevTools Protocol through Puppeteer then you will have the same browsers as described in Puppeteer column
Chrome
Edge
Firefox
Support for Sarafi and Internet Explorer is on the roadmap
Chrome
Firefox (Beta support)
As soon as this library use Dev Tools Protocol then all browsers that have the support of this protocol can be potentially be supported. But at this moment only Firefox and Chrome are ready to use.
Mobile browsers support
(We can test web apps inside the mobile browser with that feature)
WebdriverIO
Cypress
Puppeteer
Webdriverio has integration with Appium through Appium Service.
Appium is an open-source test automation framework for use with native, hybrid, and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. That means that we can test through all browsers available in mobile OSes.
Cypress will never be able to run on a native mobile app, but we can test some functionality of mobile web browsers.
Currently, we can control the viewport with the `cy.viewport()` command to test responsive, mobile views on a website or web application.
Puppeteer can only make Chrome emulate mobile devices. This can be done with DeviceDescriptors. We can have a more clear view about DeviceDescriptors from here.
We need to choose the preferred device name from DeviceDescriptors and include device names in the script.
Selector API complexity
(A selector is a function that identifies a webpage element in the test. The selector API provides methods and properties to select elements on the page and get their state.)
WebdriverIO
Cypress
Puppeteer
Cypress has a pretty convenient Selector API similar to jQuery. Also, Cypress supports selectors for finding elements by text. All CSS pseudo-selectors are supported too.
Puppeteer support such selectors as document.querySelector and document.querySelectorAll. The problem is that if you need to find elements by text then you have to use XPath selectors. Also, if you want to find parent element or you want to make non-trivial traversing you have to use XPath too.
Continuous Integration guides
WebdriverIO
Cypress
Puppeteer
Cypress has own docker images:
cypress/base - All system dependencies, no browsers.
cypress/browsers - All system dependencies and browser(s).
cypress/included - All system dependencies and the Cypress test runner installed globally.
Also, Puppeteer has guides how to run it in Docker and in cloud services like Heroku, AWS Lambda and other
Can mock HTTP requests
(This will allow substituting data for specific requests in order not to make a real request or in order to test the behavior of the application with a specific server response. Also useful for integration tests)
WebdriverIO
Cypress
Puppeteer
No
As soon as WebdriverIO uses Selenium WebDriver Protocol by default then it cannot intercept requests. You can try to create own mock server and route web application to it. Also, some packages like `nock` can help probably.
You can try to use Chrome DevTools Protocol through Puppeteer. You can get Puppeteer instance and use its API to mock requests.
Yes
It has great support for this feature out of the box. But it has one limitation: Cypress only currently supports intercepting XMLHttpRequests. Requests using the Fetch API and other types of network requests like page loads and script tags will not be intercepted or visible in the Command Log. You can delete global Fetch API and polyfill it with the Ajax version.
Yes
It has support for this feature. But it has not a very convenient interface for that as Cypress has. You can enable requests intercepted and subscribe to the 'request' event. You will need to create conditions for a particular method and URL in this event callback in order to mock different requests (or you can create some abstraction and use it like in Cypress).
Mocking responses for dataURL requests are not supported. Calling `request.response` for a dataURL request is a no-op.
Has Built-in waiting mechanism
(It does not require dedicated API to wait for redirects or page elements to appear)
WebdriverIO
Cypress
Puppeteer
No
Yes
No
Has Slowmo mode
(Tests are executed at full speed with minimum delays between actions and assertions by default, which can make it hard to identify problems when the test is running. Useful so that you can see what is going on.)
WebdriverIO
Cypress
Puppeteer
No
Only explicit wait commands
No
Only explicit wait commands. Also, it can be achieved with hack
Yes
Cloud services for cross-browser testing support
(SauceLabs, Browserstack, Browserless, etc)
WebdriverIO
Cypress
Puppeteer
SauceLabs
BrowserStack
LambdaTest
CrossBrowserTesting
TestingBot
None
But it is on the roadmap
Cypress’ API is written to be completely compatible for integration with Sauce Labs. It is our goal to offer full integration with Sauce Labs in the future; however, complete integration is not yet available.
Browserless
Can handle Drag-and-Drop actions
WebdriverIO
Cypress
Puppeteer
Yes
It has its own method `#dragAndDrop`. But if test with Puppeteer as a core then you will have problems which described in the same criteria of the Puppeteer column.
Yes
As soon as Cypress can create a native DOM event then you can emulate any actions, for example, drag and drop. You should use the `#trigger` method for that purpose.
No
At this moment puppeteer does not support drag and drop actions properly. You can try to use a combination of `mouse.move` > `mouse.down` → `mouse.move` → `mouse.up` but this does not work for all cases. For example, at this moment it does not work for `react-dnd` for some reason. Puppeteer team says that drag-and-drop support is in proposal state in DevTools Protocol.
Has built-in feature for tests parallelization
(This feature can reduce the time of tests execution)
WebdriverIO
Cypress
Puppeteer
Yes
Once your tests have several spec files, you should start running your tests concurrently. To do so, adjust the `maxInstances` property in your config file. WebdriverIO allows you to run your tests with maximum concurrency—meaning that no matter how many files and tests you have, they can all run in parallel. (This is still subject to certain limits, like your computer’s CPU, concurrency restrictions, etc.)
Yes
But requires a paid plan
Cypress can run recorded tests in parallel across multiple machines since version 3.1.0. While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.
Execution of test files can be parallelized on a per group basis, where test files can be grouped by the browser under test. This versatility enables the ability to allocate the desired amount of CI resources towards a browser to either improve test duration or to minimize CI costs.
No
Puppeteer has no built-in feature for parallelization. Only with 3-party libraries can help.
Has built-in video recording feature
(This is useful when tests run in CI. If the test fails then we can check what happened by video)
WebdriverIO
Cypress
Puppeteer
Yes
Video Reporter is a reporter for Webdriver IO v5 that generates videos of your test executions. If you use allure, then the test cases automatically get decorated with the videos as well.
Yes
After tests complete, Cypress automatically compresses the video in order to save on file size. By default, it compresses to a 32 CRF, but this is configurable with the `videoCompression` property.
No
The Puppeteer can not record video of user actions due to DevTools Protocol capabilities.
Has built-in capabilities to collect code coverage
(Technically, collecting code coverage does not fall for e2e tests. This is a separate feature. To do this, you need to instrument the code using the `istanbul (nyc)` package, for example, then run the test, and then take the result stored is in the global variable `__coverage__`. See https://github.com/istanbuljs/nyc). But some of the e2e tools have special capabilities for that feature.
Pro tip: we can combine unit tests and e2e tests code coverage reports to get the whole picture
WebdriverIO
Cypress
Puppeteer
No
But technically you can collect code coverage from global variable `__coverage__` (you need to instrument code with `nyc` first). If you want to generate some static HTML report then you need additional setup
Yes
Cypress has a detailed description of how to collect code coverage in documentation
To handle code coverage collected during each test, there is a `@cypress/code-coverage` Cypress plugin. It merges coverage from each test and saves the combined result. It also calls `nyc` (its peer dependency) to generate static HTML reports for human consumption.
In the future, Cypress will collect code coverage by the Chrome browser’s V8 engine native capabilities.
Yes
The Puppeteer can collect coverage not only for JavaScript files. but for CSS files too. Puppeteer uses the API of the DevTools Protocol.
To output coverage in a form consumable by Istanbul, see puppeteer-to-istanbul.
Quality of documentation
(The importance of documentation is difficult to overestimate. The better the documentation, the less complexity with support)
WebdriverIO
Cypress
Puppeteer
Very well done documentation with many Reporters integration guides and with many Services integrations guides.
Cypress has the most detailed documentation and it is clear and well-structured. In many articles, there is a video that tells about the topic.
Also, on the Youtube channel, we can find a lot of interesting videos about various aspects of Cypress.
Puppeteer only have API documentation. No guides or best practices and recipes. But API documentation is pretty detailed. Also, there is good troubleshooting documentation.
Has built-in capabilities to use the BDD approach (with Cucumber and similar tools)
(This feature is useful because we can write user cases in plain English (or another language) instead TDD where we just make assertions. Gherkin syntax help people who are not a programmer to read it and to edit it)
WebdriverIO
Cypress
Puppeteer
Yes
No
But it has a great 3rd-party preprocessor library for that purpose.
No
But it is pretty easy to integrate cucumber library and puppeteer.
Built-in TypeScript support
(Using TypeScript brings you all the advantages of strongly typed languages: rich coding assistance, painless scalability, check-as-you-type code verification and much more.)
WebdriverIO
Cypress
Puppeteer
Yes
Yes
Cypress ships with official type declarations for TypeScript. This allows you to write your tests in TypeScript.
Yes
Puppeteer has a separate typings package.
Has built-in capabilities for page object pattern support
(The goal of using page objects is to abstract any page information away from the actual tests. Ideally, you should store all selectors or specific instructions that are unique for a certain page in a page object, so that you still can run your test after you've completely redesigned your page.)
WebdriverIO
Cypress
Puppeteer
No
But Webdriverio has a very detailed guide on how to use simple javascript classes for that pattern
No
And this pattern is discouraged in Cypress and this is an anti-pattern.
According to Cypress, there are some problems with that approach:
Page objects are hard to maintain and take away time from actual application development. I have never seen PageObjects documented well enough to actually help one write tests.
Page objects introduce additional state into the tests, which is separate from the application’s internal state. This makes understanding the tests and failures harder.
Page objects try to fit multiple cases into a uniform interface, falling back to conditional logic - a huge anti-pattern in our opinion.
Page objects make tests slow because they force the tests to always go through the application user interface.
No
But we can create own javascript classes for pages.
Has paid options
WebdriverIO
Cypress
Puppeteer
Yes Training courses - Learn Webdriver Test Automation with WebdriverIO
Starts from 20$ per access
Yes
Cypress Dashboard - It provides timely, simple, and powerful insights on all your tests run at a glance. With automatic parallelization and load balancing, you can optimize CI and run significantly faster tests.
Starts from 99$ per month per 5 user
No
Last updated