TestRay: Simplify Your Automation and Pipelines

One of the many tasks of a QA automation engineer is to automate UI interfaces as well as APIs. However, it also involves making integrations with the development pipeline, creating reports, and then trying to find the right frameworks to make it scalable so that it does not get stuck when support for a new platform is added, or when new SDKs or APIs are developed.

That is why at TestDevLab we have developed TestRay. With this CLI tool, we are attempting to unify most of the everyday struggles of an automation engineer. This tool allows you to create automation pipelines, test browser UIs (Chrome and Firefox among others), desktop applications for macOS and Windows, as well as mobile tests for the two major players—Android and iOS. Additionally, it also includes support for API calls, and soon we are planning to add support for more types of assertions and validations.

How does TestRay work?

This CLI tool works thanks to easy-to-read YAML files, which are on their way to becoming our “development” language for such cases. With these YAML files, you will be able to create cases/steps and sets, and they can run any kind of workflow that you could think of.

These are some of the great features of the TestRay automation tool:

  • Easy to create automation pipelines based on command lines, API calls, and UI automation
  • Simple setup for your automation needs
  • Fast implementation of multiple drivers running in parallel (Android/iOS/Desktop running in parallel for the same test case)
  • Built-in Cucumber reporting 
  • Clear logging in a terminal and log files
  • Easy installation

The main focus of this tool is to make things easier for any kind of automation pipeline. Even though there are some advanced features that are not yet included, such as some custom asserts, this limitation can be circumvented by running your own scripts within the TestRay pipeline. To help you understand the scope of this tool, here is a comparison of supported platforms and different kinds of similar testing tools:

iOS Android Desktop Browsers Desktop App API calls Shell/Bash Commands Open Source Codeless
TestRay Yes Yes Yes Yes Yes Yes Yes Yes
Robot Framework Yes Yes Yes Yes Yes Yes Yes No
Leapwork No No Yes No Yes No No Yes
Smartbear Yes Yes Yes Yes Yes Yes No Yes

TestRay is based on Selenium and Appium for the UI interactions over most of the platforms, and we have added several methods to be able to get the most out of the available interactions from these two frameworks. TestRay also works with remote testing, so you can use most of the cloud providers that support Selenium and/or Appium servers or you can set your own grid.

Technical details

If you have ever written any automated tests on any of the most common languages (Java, Python, JavaScript, etc.) you might have faced the struggles of setting everything up, making the drivers work correctly, and/or parallelizing multiple device interactions. You are only given a set of utilities and “hooks” with which you can achieve this result, but it takes time and effort to set everything up correctly.

With TestRay your life is made easier and it only leaves you with one task, which is creating the actual test itself. First, we need to create a “cases” folder on your project folder which will contain all the cases and configuration files. Let’s start with the configuration file “config.yaml”. Here is where you define the configuration of your roles or the drivers that will control the automation pipelines:

Devices:
 - role: desktopChrome
   browser: chrome
 - role: desktopFirefox
   browser: firefox
# COMMAND LINE ROLES
 - role: commandLine,commandLine2

We can observe four roles here. The first two roles are for Chrome and Firefox browsers which will both run on your local computer separately. The last two roles defined (commandLine and commandLine2) will run in parallel, will not start any driver, and are used only for command-line execution. This can also be done for any other role supported by this tool.

Now that we have configured these roles, we can start creating a simple case using them. In this example, test case configuration will go under a file called “case_chrome.yaml”. When defining your own test cases make sure all your case file names have a prefix “case_” so the TestRay CLI can recognize them. In this example scenario, we will just open a window and navigate to the Google homepage:

OpenChromeGoogle:
 Step: I open chrome and navigate to google
 Roles:
   - Role: desktopChrome
     App: desktop
 Actions:
   - Type: navigate
     Role: desktopChrome
     Value: https://google.com

On the first line we can define a Step Definition, which essentially is a small piece of code with a pattern attached to it that enables defining executable steps in human-friendly language. Then we choose which roles will be used by the case. For this one, we only use the Chrome role defined in the config.yaml file. Using the “Actions” keyword we can define a series of UI interactions and other types of tasks such as command lines, navigation, or API calls. In this case, we are just navigating to the Google homepage for browsing.

We can now create a second case that uses Firefox instead of Chrome, and put it in a new case file called “case_firefox.yaml”

OpenFirefoxGoogle:
 Step: I open firefox and navigate to google
 Roles:
   - Role: desktopFirefox
     App: desktop
 Actions:
   - Type: navigate
     Role: desktopFirefox
     Value: https://google.com

Moreover, if we want to run a case that runs both steps with the two browsers, we can do so in a separate scenario like this:

OpenGoogle:
 Roles:
   - Role: desktopFirefox,desktopChrome
     App: desktop
 Actions:
   - Given: I open chrome and navigate to google
   - And: I open firefox and navigate to google

Now you have a full case written with Gherkin syntax and the only thing you need to do is execute it. This can be done by opening a command prompt or a terminal, navigating to the project folder and executing the following command:

testray execute OpenGoogle

Once the test is finished, a “Reports” folder will be made with a cucumber formatted JSON report, and a text file with all the logs with what happened during the execution.

Let us try something a bit more complicated. In the example below we will open Chrome browser, navigate to a webcam testing website and start a test using dummy input.

TestBrowserChromeAV:
  Roles:
    - Role: desktopChrome
      Capabilities:
        chromeOptions:
          args:
            - use-fake-ui-for-media-stream
            - use-fake-device-for-media-stream
            - no-sandbox
            # - use-file-for-fake-audio-capture=/home/user/silence.wav
            # - use-file-for-fake-video-capture=/home/user/1080p-5fps.mjpeg
      App: desktop
  Actions:
    - Type: navigate
    Role: desktopChrome
    Value: https://webcamtests.com/
    - Type: click
    Strategy: id
    Role: desktopChrome
    Id: webcam-launcher
    - Type: sleep
   Time: 10

In this example we are specifying some browser-specific capabilities using the keyword “args” which passes command-line arguments directly to the browser program. Specifically in this case, the first three arguments are used when running the browser and are directly passed, while the other two are commented out and are ignored when parsing the file. The three arguments that are used allow us to simulate browser input but the browser itself controls what dummy data will be fed to the website while the other two allow us to specify what specific sound and video data will be used.

These are only some of the examples of how TestRay can be used to automate the testing process and how simple it is to use the framework and the syntax. There are many more Action types, roles, and configurations that can be used to customize your test cases, but the examples shown should give you a general idea of this tool and how it works.

If you want to check out more examples please visit this GitHub repository.

Key takeaways

The TestRay CLI was created with the idea of making UI interactions among different devices under the same test plus command-line execution easy to handle, without having to deal with the threading, variable handling, and other struggles that arise from that kind of automation. We are still improving it and making it more accessible to more scenarios and projects.

In general, TestRay can be used for many different purposes, but it’s mostly designed to get your automation pipelines tidier and cleaner under a single framework and syntax. It has its limitations for more complex validations, but the TestRay team is working towards enabling additional integrations so you can automate more, better and faster!

Click here to check out the project. We would be happy to hear your feedback, which you can send to us at: [email protected].

Einārs Netlis-Galejs
Einārs Netlis-Galejs
Articles: 2