Guide to get started with Visual Studio Web Load Testing and Automation

[Microsoft_Testing_Team]

First published on MSDN on Feb 23, 2017

Authored by Edwin Hernandez

Hello everyone! We are MS Testing Services team and we are restarting the TestingSpot blog at msdn.com. It has been a few years of inactivity but our first post of the year is going to be a guide for Web Load Testing with Visual Studio, we hope you like it and start coming back for more!

This article provides consolidated information in a concise presentation for anyone who wants to start a web load test project using Visual Studio. You may have noticed that there is plenty of information on MSDN and visualstudio.com that tells you a lot of information about how things work like cloud testing or on-premises requirements, agents, controllers, webtests, plugins, etc.

However, the amount of information can be easily overwhelming and is not in a single and concise location, this article can be used as a starting guide for newcomers to understand the basic setup and to find the best route into achieving the results desired.

1. Software Requirements. Versions and licenses

Visual Studio

The first thing you will need is Visual Studio. The Web Load & Performance Testing features of Visual studio are only available on versions: Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013.

Other type of tests such as Coded UI, Manual tests and Unit testing are also available on Visual Studio Test Professional 2015 however these are all functional tests. We’ll focus on performance web tests for now, this is NOT available on VS Test Professional. You can find a feature comparison among VS versions here: Compare Visual Studio 2015 Offerings .

Virtual User Licenses

On premises. No virtual user licenses are needed. You can execute load tests with any number of virtual users (as far as your hardware resources allow you) as long as you have any of the following:

– Visual Studio Enterprise 2015 with MSDN

– Visual Studio Enterprise 2015 annual and monthly subscribers (check Visual Studio 2015 Licensing White Paper )

– Visual Studio Ultimate 2013 with MSDN (check Visual Studio 2013 Licensing White Paper )

* In Visual Studio Enterprise (or Ultimate) trial version, the virtual user count is limited to 250.

Cloud-Based testing. No extra software installation needed. You do need:

– A Visual Studio Online (VSO) account ( get one here ).

– That gets you 20,000 virtual user minutes every month to load test at no extra charge (check how virtual users minutes work and test duration limitations here: VSO Virtual users minutes ).

Controller and Agents Software

On premises. If you are planning to load test an application in an on-premises environment, you will need a Controller and at least 1 agent machine (check next section for hardware requirements). You can find the installable software for these two components based on your Visual studio version here:

– Download Agents for Microsoft Visual Studio 2013 Update 5 . This is the correct version of the Agents and Controllers software for load testing. Don’t use the Test Agents for VS 2015 for load testing. It is meant for continuous tests on build scenarios, it doesn’t have a test controller or a configuration tool.

Cloud-Based Load testing. No additional software installation needed.

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Visual Studio is used to record webtests, kick off Load Tests and visualize results. It is not recommended to install it on the Controller machine or Test Agents. The system requirements for the machine to host either Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013 are:

• Windows 7 SP1 or above, or
• Windows Server 2008 R2 SP1 or above.
• A minimum of 1.6GHz, 1.0 GB of RAM (1.5 if working on a VM).
• 10 GB of available HD space (20 GB for VS 2013).
• Check details here:
  • Visual Studio Enterprise 2015 with Update 1 
  • Visual Studio Ultimate 2013 with Update 5

Controller and Agents Hardware Requirements

On premises. If you are planning a load test project in a on premises environment, you will need a controller and at least 1 test agent, to set this up you will need the following:

• OS/Framework requirements
  • Controller: Windows 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2
  • Test Agents: Windows XP SP3 / 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2.
  • Both: .NET Framework 4.5
  • More information here: System requirements for Controller and Agents

You can also execute local test runs (without controller or test agents) directly from your Visual Studio host, this is done normally for debugging purposes since you are limited by resources.

• Hardware sizing

The table below shows the recommended hardware requirements. You can use this table to plan how many test agents and what size of controller suit your load testing needs:

Configuration	Component	CPU	HD	Memory (RAM)
< 500 virtual users	Test agent	2.6 GHz	10 GB	2 GB
< 1000 virtual users	Test agent	Dual processor 2.6 GHz	10 GB	2 GB
N x 1000 virtual users	Test agent	Scale out to N agents each with Dual 2.6 Ghz	10 GB	2 GB
< 30 computers in the test environment. This includes agents and servers under test.	Test Controller	2.6 GHz	Read load test repository section
N x 30 computers in the test environment. This includes agents and servers under test.	Test Controller	N 2.6 GHz processors	Read load test repository section

Tips:

• A rule of thumb that I use is that a Test Agent in a VM can hold from 500 to 1000 webtest virtual users on a typical load test. If you have many extraction plugins or external dependencies(CSV files, database connections) that number will decrease since it takes more processing to do that.

• Test Agents save the load test results in a temp folder before sending everything to the Controller. You may want to monitor the free space in the HD and periodically clean the temporary folders. It's usually in this path: \\<TESTAGENTVM>\c$\Users\<USERPROFILE>\AppData\Local\VSEQT\QTAgent\<TESTRESULTSESSION>\AGENT01\

• Keep in mind other Server resources aside from CPU and RAM Memory. It's always a good practice to add the Test Agents to your Scenario for monitoring.
  
  • In one of my test engagements I assumed that 3 BIG Test Agent VMs would be equivalent to 9 SMALL Test Agent VMs because the specs CPU and RAM specs matched up. However during the load test I found that this caused a bottleneck on the Test Agents because my virtual users were in queue waiting for the Agents to free threads.

• Load Test Repository

Load tests results may be stored in the Load Test Results Repository, which is a SQL database. The Results Repository database is created by setup for controllers (or created automatically on the first local run of a load test), a the database will be created automatically if the load test schema is not present.

SQL Server 2012 Express LocalDB, which is installed with Visual Studio is the default database server for load tests. If SQL Server Express is detected with an existing load test database, Visual Studio Enterprise will try to connect to it and use it.

However, for heavier database needs you should consider upgrading to a full SQL Server to provide further scaling potential. Also, SQL Express is limited to using a maximum of 4 GB of disk space. If you will run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product if available. More on Load Test results repository .

Tip: In my experience, a typical single 1 hour load test can take from 100MB to 500MB of storage depending on the number of webtests, performance counters, VMs you are monitoring, etc. Plan storage needs accordingly and take in consideration that this information is temporarily stored in the Test Agents. When the test ends the results are collected in the Controller's RAM memory before finally being placed in the SQL Repository, so also plan RAM Memory needs accordingly.

• Alternative: Testing using VSO or TFS instead of Controller

For test scenarios (still on premises) using Visual Studio Online (VSO) or Team Foundation Server (TFS) 2015, you won’t need a test controller because Agents for Microsoft Visual Studio 2015 handle orchestration by communicating with VSO or TFS 2015. For example, you’re running automated tests with your build and release workflows in VSO or TFS 2015.

Cloud-Based Load testing. You can use cloud-based load testing to avoid having to use your own resources and machines. In this case you don’t need a Controller or Test Agents. The cloud-service provides virtual machines in the cloud that generate the load needed to test your target application/website.

All you need is a Visual Studio Online (VSO) account. Check ‘Getting started with Load Tests’ further down in this document.

3. Getting started with Webtests

Basic components

At the most basic level you will need to do the following steps to get started:

• Create a web performance and load test project

• Record a web performance test.

• Create a load test.

• Run and analyze your load test

Follow this article to learn how to create the most basic components.

Validation

Validation rules help verify that a Web application is working correctly by validating the existence of text, tags, or attributes on the page returned by a Web request. Validation rules can also verify the time that it takes a request to finish, and the existence of form fields and their values.

Check this article to find out how to add predefined Validation rules to your webtest. You can also create your own Validation rules, check Customization section further down on this page.

Parametrization

• Constant parameters. You can create a Context Parameter for any value in a request of a WebTest. The constant parameters are displayed at the bottom of your webtest. A simple case where you would want to do this is for host or website URLs. Read this article for more information.
• Dynamic parameters. Web sites and applications use dynamic values, this is a value that is generated every time that a user runs the application. A dynamic parameter can cause your Web performance test playback to fail because it has a different value every the test is run. Therefore, you cannot play back recorded values. (E.g. a session ID). To solve this situation you have a couple of options (follow link to articles):
  • Auto-detect dynamic values and promote them to WebTest parameters . Use VS auto-detect feature to identify and parameterize dynamic values (it may not catch all values).
  • Add an Extraction Rule . Find dynamic values by using ‘quick find’ on recorded results then add an extraction rule:
• Extraction rules extract data from the responses to Web requests and store the results in the test context as name/value pairs. Extraction rules are predefined in Visual Studio (you can also create your own, check Customization section), you can use them to extract form fields, text, attributes, headers, regular expressions, and hidden fields, Check full article here .

Data Binding

You can bind parameters (e.g. form post parameters) on the requests of your WebTest to data sources to provide different values to the same test. This makes your load test more solid and realistic. You have several options for the data source type:

– SQL database (several versions permitted, check FAQ)

– CSV text files

– XML files (or SOAP XMLs)

Check this article for a step by step guide.

4. Getting started with Load Tests

Load Test setup

Load Test can be created in a Load Test projects. If you followed the article at the start of Section 3 of this page you must already have a simple Load test. Some of the Load test settings and scenarios can be setup on the initial Wizard or directly on the load test afterwards. Below are descriptions for the main sections:

• Scenarios

Load tests contain scenarios, which contain Web performance tests. Scenarios are important because they give you flexibility in configuring test characteristics that allow for simulation of complex, realistic workloads. You can have more than one scenario per load test.

Use the Load pattern, test mix model and test mix to specify what webtests to use, the manner in which the load will be applied to your application and pacing.

You can also use Browser mix, Network mix to make your workload more realistic. Check MSDN article on editing Load tests scenarios .

• Counter Sets, Counters and Mappings

Counter sets that are useful when you analyze performance counter data. The counter sets are organized by technology such as Application, .NET Application, IIS, or SQL. You can create counter sets during the initial New Load Test Wizard or later on the load test file itself.

There is a set of predefined counter sets or you can create your own. You can also add counters to the default Counter Sets or to your own. The counters are added from Perfmon.

On the Run settings section of a load test you can specify Counter Set mappings where you can add computers (by DNS or IP) and map them to Counter sets. Visual studio will collect data for every counter on a set for each computer where it is mapped to.

To learn more about Counter Sets, thresholds, sampling and other tasks, check this article on MSDN .

• Run Settings

Run settings are properties that organize the manner in which a load test runs. You can have more than one run setting in a load test but only one can be active at a time while the other settings provide a quick way to select an alternative settings.

Settings are used to specify things like test duration, Logging, SQL server collection, warm up duration, sampling rate or validation setting between others. You can also add Counter Set Mappings to map computers to Counter Sets.

Check this article for full description of tasks available.

Running Load Tests On Premises

• Test Settings file

Make sure your Load test is configure to run On premises by checking the General section of your active .testsetting file on your Solution. Test Settings are not the same as Run Settings.

Other items to configure on this window are Roles and Timeouts. On the Roles section, test execution method should be set to ‘Remote execution’ if you want to use your Controller and Test Agents. Check this couple of article to learn more about Test settings: Managing Test Controllers and Test Agents with Visual Studio , Edit Test Settings .

• Controller-Agents connectivity

Also make sure the Controller can communicate with the Test Agents by checking the Load Test > Manage Test Controllers. That dialog should show if the controller has any Test Agents connected to it. More information here: Manage Test Controllers.

• Running the Load Test

From that point forward you should be ready to run a Load Test. During or after run time you have access to Graphs and Tables with throughput, response times information, performance counters for the computers you configured and details on vusers activity. Check this tutorial to learn more: Run and analyze your load test .

Running Load Tests using the Cloud Service

There are a couple of options for running a Load test with the Cloud Service:

• You can run a Load test from Visual Studio (Enterprise 2015 or Ultimate 2013) and Visual Studio Team Services.
  
  
  • As mentioned above you will need a VS Team Services account.
  
  
  • Your solution needs to be connect to TFS.
  
  
  • In your solution; if you open your active .testsettings file, on the General section select ‘Run tests using Visual Studio Online’:

• From there you can specify the Location you want your load to come from on the Load Test file and run the Load test. Check the full tutorial here , specifically the ‘Run and analyze your load test’ section.

• Your second alternative is to run a basic (and much more limited) load test directly in Visual Studio Team Services. The up side is that you don’t need a load test project (or Visual Studio). You do need a Visual Studio Enterprise monthly or annual subscription or a MSDN subscription.
  
  
  • To do this you need to go to the Load Test Hub on your Visual Studio Team Services account page.
  
  
  • Setup is simple, just specify URL, Load location and some simple Test settings. Check the com article here.

5. Customization

Visual Studio offers extendibility options to increase the built-in functionality. Follow the articles below to learn more about the different options:

• Custom Validation Rules . You can create your own validation rules. To do this, you derive your own rule class from a validation rule class. Validation rules derive from the ValidationRule base class.
• Custom Extraction Rules . You can create your own extraction rules. To do this, you derive your own rules from an extraction rule class. Extraction rules derive from the ExtractionRule base class.
• Custom Plug-in’s. You can create custom plug-in to write code and attach it to a load test or a Web performance test. You can use the load test API and the Web Performance test API to create custom plug-ins for tests to expand to the built-in functionality.
  • Create a custom plug-in for your Web Performance test: You can use Web performance test API to create a custom plug-in to add more testing functionality to your Web performance test, including at the request level. You can also create a Web Service test.
    
    • How to: Use the Web Performance Test API
    
    
    • How to: Create a Web Performance Test Plug-In
    
    
    • How to: Create a Request-Level Plug-In
    
    
    • How to: Create a Web Service Test
    
    
    • How to: Create a Recorder Plug-In

• Create a custom plug-in for your load test: You can use load test API to create a custom plug-in to add more testing functionality to your load test.
  
  • How to: Use the Load Test API
  
  
  • How to: Create a Load Test Plug-In
  
  
  

FAQ

Please feel free to ask questions in the Comments section of this blog post. I can try to answer them and add them to this FAQ.

How many vusers can I run per agent?

As mentioned above the recommendation is up to 500 for a 2.6 GHz processor and up to 1000 for a Dual 2.6 GHz processor. This can change depending on the length of the webtests, the payloads of the application responses and the use of custom plug-ins.

What version of SQL do I need?

SQL Express or SQL Express 2012 Localdb for local test runs and debugging. Full SQL for heavier needs such as an Enterprise level application performance test project, for example the standard version of SQL Server 2014, 2012, or 2008 R2.

Can load tests use other test types in their test mix besides web performance tests?

Yes, you can include unit tests and coded UI tests.

Can virtual users simulate pausing between test steps?

Yes, you can specify think times to simulate the time spent by a user on a web page.

Can I mix versions of TFS, Microsoft Test Manager, the test controller, and test agent?

Yes, here are the compatible and supported combinations:

TFS	Microsoft Test Manager, with Lab Center	Controller	Agent
2015: upgrade from 2013	2013	2013	2013
2015: new install	2013	2013	2013
2015: upgrade from 2015 or new install	2015	2013	2013
2013	2015	2013	2013

https://msdn.microsoft.com/en-us/library/dd648127.aspx#Anchor_2

What firewall permissions do I need?

The default port used by the test controller is 6901 and the test agent’s default port is 6910. The client (Visual Studio machine) uses a random port by default which is used to receive the test results from the test controller. For all incoming connections, the test controller authenticates the calling party and verifies that it belongs to specific security group. This is true regardless of the test rig configuration you are using: on premises or cloud based testing:

You should create exceptions for the mentioned ports on the firewall, you can also change the port. Check this msdn article for more information: Configuring Ports for Test Controllers and Test Agents

What databases can I use as a data source?

You can use the following:

• Microsoft SQL Azure.

• Any version of Microsoft SQL Server 2005 or later.

• Microsoft SQL Server database file (including SQL Express).

• Microsoft ODBC.

• Microsoft Access file using the .NET Framework provider for OLE DB.

• Oracle 7.3, 8i, 9i, or 10g.

What is a good place to learn more?

Aside from MSDN you can go to Channel 9 videos for a more visual learning experience: https://blogs.msdn.microsoft.com/visualstudioalm/2015/12/10/visual-studio-2015-test-tools-getting-s...