by  Ostap Elyashevskyy
Quality Management

Cloud Based Automation Testing with Selenium WebDriver: How Not to Get Lost

clock-icon-white  5 min read

When talking about Web UI Automation testing, we usually mean testing based on WebDriver which is the de facto standard test automation tool. In most cases, Web testing is performed on the local environment and a relatively small number of machines. But what if Web Automation testing needs to be performed on more than 20-30 different configurations? How can you establish and support testing in this case?

In this article, I will analyze the existing cloud systems and infrastructures, as well as compare them to find the best solution for hosting 30+ machines with different test configurations. In our case, “test configuration” presupposes a machine with the pre-installed operating system (OS) and Web Browser which will be used for Web application(s) testing. Let’s clarify the requirements:

  • Test lab should include 50 machines
  • Each virtual machine should support WebDriver and allow execution of Web UI functional testing
  • Typical configuration of machine: CPU 1 core 2-3 GHz, RAM 4Gb, HDD 30Gb
  • A mix of the following OSs: Windows 7, Windows XP, Windows 8, Ubuntu, CentOS
  • Predefined browsers, namely Firefox, IE, Chrome

Automation Testing Solutions Overview

Talking about our specific situation, having a farm of 50 physical machines is not an option. Can you imagine how many resources are needed to buy, support, and store such an amount? The best fitting choice would be virtualization servers or cloud solutions. Let’s go along with the following list for further analysis:

WebDriver Based Cloud Services

BrowserStack

BrowserStack provides instant access to all desktop and mobile browsers without the need to configure and prepare the test environment. It supports pre-installed 700+ browsers, 40+ mobile devices, and 9 Desktop operating systems, so it is possible to select any OS-browser combination to launch Web Driver-based tests. BrowserStack supports different types of licenses at a cost ranging from 49$ to 999$+ per month depending on the number of parallel executions and duration of usage.

It’s recommended to opt for BrowserStack if:

  • Web UI application should be tested using WebDriver
  • A range of different configurations are to be verified
  • Applied code does not use any 3rd-party libs like Sikuli, AutoIt, etc.
  • Applied code is wri tten in supported languages: C#, Python, Java, Ruby, etc.

SauceLab

SauceLab is a similar service to BrowserStack, which also allows running tests on the remote machines. Just like BrowserStack, it supports Mobile and Desktop browsers. The main advantage of SauceLab is an ability to test both Web and UI native applications using WebDriver and Appium.

When to go for SauceLab? Here are the main reasons:

  • WebDriver or Appium-based tests
  • Web or native or hybrid applications, both Web and UI, have to be tested
  • No other software besides WebDriver or Appium is used
  • A range of different configurations are to be verified
  • Applied code is written in a supported language: C#, Python, Java, Ruby, etc.
  • Manual testing is also required

To size up, in our case we cannot use these solutions because other tools and libraries besides WebDriver, would need to be installed (namely Sikuli, AutoIt, etc).

Scalable Cloud Solutions: Amazon EC2, RackSpace, & Azure

The most popular cloud solutions are Amazon EC2, RackSpace and Azure. Generally these providers have common features which fit our situation, namely:

  • Dynamic change of the hardware system configuration: add/remove CPU/RAM/HDD/etc.
  • Import VM with one of the supported OS (mostly server OSs: MS Windows Server, Ubuntu, CentOS, etc.)
  • Start/Stop instances
  • Pay per use
  • Machines scale to any extent
  • Tracking and analysis of usage statistics

These solutions appear to be the most suitable option when:

  • Hardware requirements change from time to time
  • There is a need to scale hardware parameters (e.g., 8/16/32/64 Gb of RAM is required)
  • Control of mac hines via API is required

This solution is good for flexible and dynamic environments. Since we have a static environment, there’s no need to scale it. Also the open question is licensing and import of VM images. These services provide import of images but there are some tricky moments: before import the VM image should be converted, which is not always easy. Also, Windows, WMware, etc. require additional payment for licenses. This approach can be used only for servers where the application under test is hosted.

VMWare vSphere ESX/ESXi Servers

VMWare ESX(i) Servers or Virtual Machine appliance consists of hardware and software parts (firmware), and doesn’t require the OS to be installed in advance. That’s why VMWare is often called bare machine or bare metal.

Hosting VMs on ESX(i) servers makes sense when there’s a static environment (e.g. 50 machines) and the parameters of the machines don’t have to be changed often. ESX(i) Servers are hosted mostly inside the company, so support/configuration effort should be taken into consideration.

This solution is a perfect fit when:

  • Environment configuration changes rarely
  • Scaling of environment is not needed
  • Hardware requirements are known
  • There is a possibility to host and support ESX appliances

To have a clearer vision, let’s compare all of the solutions in one table:

selenium

Conclusions

In our case, the best option for hosting 50 machines is to use local VMWare vSphere ESX(i) Servers because of the price, availability, low support cost and static environment criteria, as well as the ability to manage and control VM Ware ESX machines through Jenkins CI (through vSphere plugin), Web UI, or desktop application. Today’s market is bursting with dozens of cloud systems and solutions. Choosing a proper service requires time for analysis and what’s more important – understanding what your business needs and why.