Automate a Web/Desktop application by using Sikuli library

Introduction

Have you ever worked for a combined Web-Desktop project and your boss said “Make automation tests to test the Web application but also to test integration between the Desktop and the Web application and take into consideration to use open-source tools because out budget is limited?”. Well, there is a solution for everything!

First, let’s better understand what a Web-Desktop architecture means: some companies can offer a diverse amount of software packages that must work together. It is important not only to test individual packages but also the chain between them. The problem may be that individual applications can have different bases. In the image below we can observe that any change done in the Desktop application should be integrated in the Web application.

Because we are dealing with both testing a Web application and also with the integration Web-Desktop, a solution would be to use:

•           Selenium WebDriver for testing the Web application;
•           Selenium WebDriver and Sikuli for the integration.

Integrate Selenium WebDriver and Sikuli in the Same IDE

It doesn't make sense to say too much about Selenium WebDriver because this a free tool already well known and commonly used.

Sikuli[1] is an open-source research project that automates anything we see on the screen. It uses image recognition to identify and control GUI components. It is useful when there is no easy access to a GUI's internal or source code, this means that the only requirement here is that the objects that are tested to be on the screen.

You’ve probably already guessed that in our testing framework we are going to use Sikuli to simulate actions in the Desktop application and then with Selenium, we will check the integration of the data in the Web application. In order to achieve this we do not want to use an IDE for Selenium and another one for Sikuli. How can we obtain this result?

•      Install Java;
•      Use Intellij IDEA + Maven for Selenium;
•      The core of SikuliX is written in Java, which means we can use the SikuliX API as a standard     JAVA library in our program. Download sikulixapi.jar from Sikuli site;
•      In the Maven pom.xml file add the following dependency;         

•  Import classes we need in our applications: import org.sikuli.script.*;
•   Because we will have to capture images on the screen to identify object, installing the Sikuli    software is anyway required. The reason we want to use it is because Target Offset        functionality will help us to indicate where exactly we want to execute an action in the Sikuli  object. Let’s imagine a text with a label near it. If we want to click on the textbox and not on  the label, the functionality will help us to click on the specified coordinates;

•           The objects (image) should be saved on the disk and used in Intellij IDEA.

Basic Framework for Web-Desktop Integration

Having integrated sikulixapi.jar dependency in Maven, we are now able to access Sikuli methods by declaring objects of type Screen, Pattern and App:

The most important features about a screen object are: click, double click, type. We can create functions to simulate these actions like described below. It is better to keep all these helpers in a separate class:

Click:

DoubleClick:

Type:

What is the meaning of SikuliWaitElement method? To perform operations on an image, we need to make sure that the image is visible on the screen. Therefore we can expect a number of seconds until the picture appears on the screen. This is not the safest solution because we cannot be sure that indeed the image will be available after the number of seconds indicated (maybe the desktop application is slow). An alternative would be to wait in a loop one second until reading a desired result (while image does not exist, wait one second). But this is also not a complete solution because it may happen that the object does not even exist, which means that the execution of the test would not stop. The best way is to wait a maximum number of seconds. If during this period we find the object, then we can go to the next step, otherwise we will stop the test. Here is an example:

IsImageAvailable function should verify if the image is present or not on the screen.

For every image we should declare a constant. We should also have a constant that gives us the path to the images folder. It is wise to keep these constants in a separate class.

Then, we can use our image like below (the example clicks on the desired image):

If we want to minimize the run time of our scripts it is recommended to simulate keyboard functionalities by using TypeImage function previously exemplified instead of using a lot of images. Below we can see an example of how we can use this function. Suppose in the Remote Assistance Settings dialog we want to change the settings in order to have

•        Allow this computer to be controlled remotely à unchecked
•         Set the maximum amount of time invitations can remain open à 6 Minutes
•        Create invitations that can only be used from computers running Windows Vista or later à checked

Initially the dialog looks like this:

We can write this code in order to obtain the desired result after Allow this computer to be controlled remotely image was captured:

Other Important Functionalities. Screen Class [2]

Until now we discussed about the most used functions: Click, DoubleClick, Type, Wait and Exists. But besides these, there are other interesting functions in sikulixapi.jar library, like:

FindAll(image) searches the entire screen for all the matching visual patterns and returns a list of locations of those similar patterns. We can use this function for example for unchecking all check-boxes on a page.

DragDrop(image1, image2) takes two images as arguments. The first image describes the source GUI object to drag and the second image describes the appearance of the destination location where the GUI object should be dragged to and dropped.

Drag(image) starts a drag-and-drop operation by dragging at the given click point.

Hover(image) makes the mouse cursor hover over an object.

RightClick(image) performs a mouse click on the click point using the right button.

MouseDown(button) presses and holds left mouse. We can simulate a combination of the button constants Button.LEFT, Button.MIDDLE, Button.RIGHT.

MouseMove(image) moves the mouse to a new location indicated by the image.

MouseUp(button) releases mouse button.

Wheel(image, WHEEL_DOWN | WHEEL_UP, steps) moves the mouse pointer to a location indicated by the image and turn the mouse wheel in the specified direction by the specified number of steps.

Paste(image, text) pastes the text at a click point.

Highlight(second) highlights the region for some period of time.

Other Important Functionalities. App Class [3]

Opening an application à App.open(application name).

Switching to a different application à App.focus(application name)

Closing an application à App.close(application name).

Other Important Functionalities. Pattern Class [4]

GetFilename gets the filename of the image contained in the Pattern object.

Conclusion

We succeeded, in this article, to find a solution to automatically test the integration between two totally different systems in a single IDE.

Even Sikuli is not an automation test tool that depends on the code, it is a very usable and user-friendly tool. Testers that haven’t worked with automation tools before will notice in a short time that they will be able, by adding only a few lines of code, to perform very easily the main actions like clicking, double-clicking and typing on an object.

Sikuli completely depends on what’s shown on the screen, can be sensitive to resolution differences and also the tests are content-sensitive and can be affected by the object layout changes. However, Sikuli included in Java is an alternative for integrating two different systems and, also, can be used to compare objects in the web page that are difficult to catch with Selenium.

References

[1] http://www.sikuli.org/

[2] http://sikulix-2014.readthedocs.org/en/latest/screen.html

[3] http://sikulix-2014.readthedocs.org/en/latest/globals.html

[4] http://sikulix-2014.readthedocs.org/en/latest/pattern.html