Jenkins - Run Visual Studio Performance and Load Tests

Introduction

Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013, gives you the possibility to test the performance of your application. If you don't want to run tests in a CI you can use Jenkins.

What do you need?

1. Jenkins with MSTest Plugin installed is required. Don't forget to set the path to mstest.exe in Manage Jenking --> Global Tool Configuration:

2. Then, you have to create a new Jenkins job, set a Source Code Management (e.g. git, tfs, etc.).
Delete work-space before build starts.

3. Create batch command step that should
- restore nuget packages (if any)
- build the solution

..nuget location\nuget.exe restore "..performance project location\yourproject.sln"
..msbuild location\msbuild.exe "..performance project location\yourproject.sln " /p:Configuration=Debug

4. Create a Run unit tests with MSTest step that should contain:
- MSTest version defined in Global Tool Configuration
- performance/load tests + path
- result name to store .trx results 

5. Add a post build step of type Publish MSTest test result report like in the image below (exclude this step for a load test, included only for performance tests):

6. Finally, attach as artifacts, all .webtestResult files (for performance tests) or .trx files for load tests.

Visualize test results

For a performance test, the test results should look like this

You can notice, failure results and all .webtestResult files attached as artifacts. If you have a failed test, e.g. in this example MAINPortfolioDuplicate, just click on MAINPortfolioDuplicate.webtestResult. Open this with Visual Studio and you'll see the cause of the error.

For a load test, open .trx artifact in Visual Studio to see test results display in Test Results window.

Once you'll click on a test result, you'll see details with graphs

and errors:

Happy testing and... make it green, becomes a dream :)

Web Performance Testing with Visual Studio - JSON Extraction rules

Introduction

Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013, gives you the possibility to test the performance of your application. In this article we'll show you how to use extraction rules in order to take information from a JSON response that is later used in a performance recorded test. It is required that you already know how to create a new performance test/record it, etc. If you don't, you can inspect MSDN documentation here.

Access token

In our application, we have a login form. After submitting, we receive a JSON response that contains variable access token that is used to authenticate in the application. In order to use this information, we need to use an extraction rule, to later use it in other requests.

Visual studio comes with some predefined rules, but, none of them is capable to extract information from a JSON. That is why we need to create a custom extraction rule.

What are the steps for creating a custom extraction rule:

1. Open a Test Project that contains your Web performance test
2. Create a separate class library project, e.g. ExtractionRule
3. In the Class library project, add a reference to the Microsoft.VisualStudio.QualityTools.WebTestFramework dll
4. Create a class that derives from the ExtractionRule class. Implement the Extract and RuleName members, e.g. JSONExtractionRuleClass (see below, the code for this)
5. Build the new Class library project
6. In the Test Project, where your performance test is, add a reference to the Class library project that contains the custom extraction rule  
   
7. In the Test Project, open a Web performance test in the Web Performance Test Editor
8. To add the custom extraction rule, right-click a Web performance test request and select Add Extraction Rule.
   The Add Extraction Rule dialog box appears. You will see your custom validation rule in the Select a rule list, together with the predefined validation rules. Select your custom extraction rule and then choose OK   
   
9. Use this context parameter extracted from a JSON to later authenticate your requests. To do this, add a header to your request with name = Authorization and value = Bearer {{loginToken}}  
   
10. Run your Web performance test

JSONExtractionRuleClass

The code for extracting the rule, necessary to pass step 4 is the following

Happy testing and... make it green, becomes a dream :)

1 Way to Integrate Applitools Eyes with Jenkins

Introduction

In an older post we talked about Applitools Eyes. What is it? It is an automation testing tool that will check all visual aspects of the application from a given screen, meaning that we don't need to write separate tests for each UI element on the screen. The tool provides easy integration with different test tools and frameworks like Selenium, CodedUI, HP QC, etc

Jenkins Integration

You can create a job that is running only tests having category = "Applitool". If a test fails, you will notice in Jenkins an error with a link to Applitools eyes site, that will load applications's UI, where you will be able to evaluate the test result, accept/reject changes, etc.

But, if there are several failures in the application, clicking on every link will be annoying. That is why, we can use Jenkins plugin, we will be able the integrate applitools ui directly in Jenkins.

Applitools Eyes Plugin

Go to Jenkins --> Manage Jenkins --> Manage plugins and install this plugin.
Go to Applitools job configuration and Build Environment check Applitools Support

Go to your project solution and add this:

and, before instantiating Eyes driver, do this:

Then run your job. Applitools Eyes will be integrated inside Jenkins. There you be able to accept/reject changes, etc, directly in Jenkins.

If you want to read more about this, please watch this video.

I hope you enjoyed this article!

Happy testing and... make it green, becomes a dream :).