TestingSpot Blog articles

How AI Is Transforming Performance Testing

padma_sathuluri — Wed, 04 Feb 2026 14:59:17 GMT

Performance testing has always been a cornerstone of software quality engineering. Yet, in today’s world of distributed microservices, unpredictable user behaviour, and global-scale cloud environments, traditional performance testing methods are struggling to keep up.

Enter Artificial Intelligence (AI) — not as another industry buzzword, but as a real enabler of smarter, faster, and more predictive performance testing.

Why Traditional Performance Testing Is No Longer Enough

Modern systems are complex, elastic, and constantly evolving. Key challenges include:

Microservices-based architectures
Cloud-native and containerized deployments
Dynamic scaling and highly event-driven systems
Rapidly shifting user patterns

This complexity introduces variability in metrics and results:

Bursty traffic and nonlinear workloads
Frequent resource pattern shifts
Hidden performance bottlenecks deep within distributed components

Traditional tools depend on fixed test scripts and manual bottleneck identification, which are slower, reactive, and often incomplete.

When systems behave in unscripted ways, AI-driven performance testing offers adaptability and foresight.

How AI Elevates Performance Testing

AI enhances performance testing in five major dimensions:

1.AI-Driven Workload Modelling

Instead of guessing load patterns, AI learns real-world user behaviours from production data:

Detects actual peak-hour usage patterns
Classifies user journeys dynamically
Generates synthetic workloads that mirror true behaviour

Results:

More realistic test coverage
Better scalability predictions
Improved reliability for production scenarios

Example:
Instead of a generic “add 100 users per minute” approach, AI can simulate lunch-hour bursts or regional traffic spikes with precision.

Intelligent Anomaly Detection

AI systems can automatically detect performance deviations by learning what "normal" looks like.

Key techniques:

Unsupervised learning (Isolation Forest, DBSCAN)
Deep learning models (LSTMs, Autoencoders)
Real-time correlation with upstream metrics
prioritized, actionable recommendations and code-fix suggestions aligned with best practices

Example:
An AI model can flag a microservice’s 5% latency spike — even when it recurs every 18 minutes — long before a human would notice.

Predictive Performance Modelling

AI enables you to anticipate performance issues before load tests reveal them.

Capabilities:

Forecasting resource saturation points
Estimating optimal concurrency limits
Running “what-if” simulations with ML or reinforcement learning

Example:
AI predicts system failure thresholds (e.g., CPU maxing out at 22K concurrent users) before that load is ever applied.

AI-Powered Root-Cause Analysis

When performance degrades, finding the “why” can be challenging. AI shortens this phase by:

Mapping cross-service dependencies
Correlating metrics and logs automatically
Highlighting the most probable root causes

Example:
AI uncovers that a spike in Service D was due to cache misses in Service B — a connection buried across multiple log streams.

Automated Insights and Reporting

With the help of Large Language Models (LLMs) like ChatGPT or open-source equivalents:

Summarize long performance reports
Suggest optimization strategies
Highlight anomalies automatically within dashboards

This enables faster, data-driven decision-making across engineering and management teams.

The Difference Between AIOps and AI-Driven Performance Testing

Aspect	AIOps	AI-Enhanced Performance Testing
Primary Focus	IT operations automation	Performance engineering
Objective	Detect and resolve incidents	Predict and optimize system behaviour
Data Sources	Logs, infrastructure metrics	Testing results, workload data
Outcome	Self-healing IT systems	Pre-validated, performance-optimized code before release

Key takeaway: AIOps acts in production; AI-driven testing acts pre-production.

Real Tools Adopting AI in Performance Testing

Category	Tools	Capabilities
Performance Testing Tools	JMeter, LoadRunner, Neoload, Locust (ML Plugins), k6 (AI extensions)	Intelligent test design, smart correlation, anomaly detection
AIOps & Observability Platforms	Dynatrace (Davis AI), New Relic AI, Datadog Watchdog, Elastic ML	Metric correlation, predictive analytics, auto-baselining

These tools improve log analysis, metric correlation, predictive forecasting, and test script generation.

Key Benefits of AI Integration

✅ Faster test design — Intelligent load generation automates script creation
✅ Proactive analytics — Predict failures before release
✅ Higher test accuracy — Real-world traffic reconstruction
✅ Reduced triage effort — Automated root-cause identification
✅ Great scalability — Run leaner, smarter tests

Challenges and Key Considerations

⚠ Data quality — Poor or biased input leads to faulty AI insights
⚠ Overfitting — AI assumes repetitive patterns without variability
⚠ Opaque models — Black-box decisions can hinder trust
⚠ Skill gaps — Teams require ML understanding
⚠ Compute costs — ML training adds overhead

A balanced adoption strategy mitigates these risks.

Practical Roadmap: Implementing AI in Performance Testing

Step 1: Capture High-Quality Data
Logs, traces, metrics, and user journeys from real environments.

Step 2: Select a Use Case
Start small — e.g., anomaly detection or predictive capacity modelling.

Step 3: Integrate AI-Ready Tools
Adopt AI-enabled load testing and observability platforms.

Step 4: Create Foundational Models
Use Python ML, built-in analytics, or open-source tools to generate forecasts or regressions.

Step 5: Automate in CI/CD
Integrate AI-triggered insights into continuous testing pipelines.

Step 6: Validate Continuously
Always align AI predictions with real-world performance measurements.

Future Outlook: The Next 5–10 Years

AI will redefine performance testing as we know it:

Fully autonomous test orchestration
Self-healing systems that tune themselves dynamically
Real-time feedback loops across CI/CD pipelines
AI-powered capacity planning for cloud scalability

Performance engineers will evolve from test executors to system intelligence strategists — interpreting, validating, and steering AI-driven insights.

Final Thoughts

AI is not replacing performance testing — it’s revolutionizing it.

From smarter workload generation to advanced anomaly detection and predictive modelling, AI shifts testing from reactive validation to proactive optimization.

Organizations that embrace AI-driven performance testing today will lead in speed, stability, and scalability tomorrow.

AI‑Powered Performance Test Analysis using GHCP

AnilKumarGolla — Mon, 02 Feb 2026 15:39:48 GMT

Problem Statement
Performance testing teams often face significant challenges in comparing JMeter test results across environments or test runs.
Manual comparison and analysis of multiple result files is time-consuming, error-prone, and lacks actionable insights.

Solution
An AI-powered solution leveraging GHCP has been developed to address the identified challenge. This solution is designed to deliver

Seamlessly compare JMeter performance results across environments (e.g., On-Prem vs. Azure) or test runs.
Provide AI-driven insights to highlight endpoints with significant performance changes.
Deliver clear, prioritized recommendations for faster issue resolution.
Reduce analysis time by up to 80%, minimizing resource utilization and enabling cost savings.

Business Outcomes

Automated Performance Comparison: Seamlessly compare JMeter performance test results across two environments (e.g., On-Prem vs. Azure) or between different test runs, reducing manual effort and accelerating analysis.
AI-Driven Insights for Decision Making: Leverage AI to identify endpoints with the most significant performance improvements or degradations.
Actionable Observations and Recommendations: Generate clear, prioritized recommendations based on key performance trends, ensuring faster resolution of bottlenecks and improved application reliability.
Enhanced Efficiency and Cost Savings: Minimize analysis time and resource utilization through automation, contributing to measurable effort savings and improved operational efficiency. manual comparison of multiple JMeter result files is time-consuming and automation can reduce analysis time by up to 80%

Pre-Requisites

Visual Studio code with GHCP Enabled.

Usage Guidelines

Start GitHub Copilot Chat from within Visual Studio Code.
Attach follow files in GHCP Chat

Azure_PerfTestResults.json
OnPrem_PerfTestResults.json
PerfResultsAnalysis_Instructions.md

Note: *.json files are statistics.json files generated as part of JMeter Html Reports.

Execute below UserPrompt

UserPrompt: “Follow the steps in #file:PerfResultsAnalysis_Instructions.md and compare the two JMeter result files uploaded.”

File Structure Validation is performed to ensure both files conform to the expected test results format.
Upon successful validation, select the performance metric for comparison. Ex: AverageResponseTime

Test results are analyzed, and Response Time Comparison between Baseline (On-Prem) and Azure is presented, including deviation and performance status. Results are also exported to a CSV file for easy reference.
AI-Driven Performance Insights are generated to provide actionable recommendations.

Use the prompts below to perform a more detailed analysis of your test results.

UserPrompt: “Get me the Average Response time of GetProducts API between Azure and On-Prem”

User Prompt: “Expected SLA on Azure is 150 ms, Get me the APIs whos Average Response Time is > 150 ms on Azure”

GitHub Repository for project is available at https://github.com/AnilKumarGolla/PerfAnalysisUsingGHCP

Java Test with Visual Studio Code

Microsoft_Testing_Team — Wed, 04 May 2022 14:44:30 GMT

By Torian Harris

This tutorial will guide you through how to write and test a Java program using Visual Studio Code. It will also cover how to push the program to a Repository and automatically run tests by building a pipeline.

The first step is to install Visual Studio Code which you can download here: Download Visual Studio Code - Mac, Linux, Windows. If you do not have Java already installed, you can find it here: Download Java for Windows

Once you open the application, click the Extensions icon (Ctrl + Shift + X) on the left then type in "Extension Pack of Java" and install it by clicking the install button.

Open View > Command Palette (Ctrl + Shift + P) and type in ">Maven: Create Maven Project”, click it then click “maven-archetype-quickstart” and version “1.4." Afterward, name the group ID and artifact ID and save the project to a folder. Press Enter to proceed through the Terminal prompts then, open your newly created project.

Replace the code in App.Java with the following code:

package com.example; public class App { public static void main(String[] args) { System.out.println(sum(1,2)); System.out.println(evaluate("2+3+4")); } // adds two ints public static int sum(int a, int b) { int sum = a + b; return sum; } // calculates an additional expression public static int evaluate(String expression) { int sum = 0; for (String summand: expression.split("\\+")) sum += Integer.valueOf(summand); return sum; } }

Next, click Run on the App class. This will open a Terminal that will display the results of the sum and evaluate functions.

Now that we have a working project, let us set up some tests to validate the functions. Replace the code in AppTest.Java with the following code:

package com.example; import static org.junit.Assert.assertEquals; import org.junit.Test; public class AppTest { @Test public void sumTest() { int sum = App.sum(1,2); assertEquals(3, sum); } @Test public void evaluateTest() { int sum = App.evaluate("2+3+4"); assertEquals(9, sum); } }

Click the play button next to the AppTest class to run the tests. The Testing Window will open showing the progress and results of the tests.

Now, we have functioning testing for our Java program. The next thing to do is to create a repository and build our pipeline. Navigate to your existing organization at https://azure.microsoft.com/en-us/services/devops/ or create a new organization if you don’t already have one and click New Project. On the left, click Repos to find your repo URL. Copy it, then in Visual Studio Code, open the Command Palette, type in “>Git: Clone” and confirm. From here paste in the URL from your repo and save it locally. Copy all files from your previous project into the new project structure. Then in the Source Control window, commit and push your changes to the server.

Lastly, it is time to build our pipeline. Head to the Pipelines tab in Azure DevOps and click “New Pipeline.” Select Azure Repos Git, select your repository, then “starter pipeline.” In the YAML file, change the value of “mavenPomFile:” to “’pom.xml’” and on the right side clear out the Solution name. Click “Save and Run” and confirm to start the build process. If you click on the queued Job, you will get to see the build process in action. Once that is done, head back to your pipeline and click “Tests” to see the results of your run.

Congratulations! You have built a Java program and a test pipeline using Visual Studio Code.

If you are interested in learning more about Java testing features using Visual Studio Code, check out this post:

Visual Studio Code + Java Feb 2022 updates

What is BDD? How to use SpecFlow in Visual Studio 2022 - SpecFlow Series - Part 1

Microsoft_Testing_Team — Fri, 11 Mar 2022 19:03:05 GMT

By Edwin Hernandez

Hello everyone. This time we would like to talk about BDD and SpecFlow. We have featured many types of test automation in this blog, from UI Tests for regression to Performance Testing with JMeter. We have also highlighted some models to organize your tests such as POM (Page Object Model).

In this post, we will first review from a concept standpoint, what is the BDD model and how to use it. Then we will review what Cucumber is just like a leg up into a tutorial/intro of SpecFlow for .NET.:

1. CONCEPTS

What is BDD?

Behavior-Driven Development (BDD) is an agile software development process that encourages collaboration among developers, quality assurance testers, and customer representatives in a software project. It encourages teams to use conversation and concrete examples to formalize a shared understanding of how the application should behave (Wikipedia).

What this means is that BDD tries to link business requirements with technical specifications by writing requirements in plain English through something called Domain-Specific-Language (DSL).

A related concept is TDD, which is a methodology in which developers first write a Test, and only then do they write the code to make the test pass. It's an iterative process in which more tests are created which then results in more code or improvement to code.

BDD is an evolution of TDD. In BDD, behavioral specifications are written in plain English (DSL), and these specifications can be used to write test code, and the test code can be used to create app code.

This doesn't always happen like this. As a matter of fact, on most of the projects of which I've been part, the BDD Feature is created first, then a developer writes some code to implement it, and only then is a test written to validate the Feature. This is because I've mostly worked implementing API or UI testing, which is difficult to implement without a working UI or an API Swagger (but not impossible, you can just create stubs until endpoints are available).

Below is a sample Feature specification (also from Wikipedia). BDD practices tell us that a feature must have a Title, a Narrative, and Acceptance Criteria. You can have multiple scenarios for the same feature since you need to cover several angles:

Title: Returns and exchanges go to inventory.

As a store owner,
I want to add items back to inventory when they are returned or exchanged,
so that I can track inventory.

Scenario 1: Items returned for a refund should be added to inventory.
Given that a customer previously bought a black sweater from me
and I have three black sweaters in inventory,
when they return the black sweater for a refund,
then I should have four black sweaters in inventory.

Scenario 2: Exchanged items should be returned to inventory.
Given that a customer previously bought a blue garment from me
and I have two blue garments in inventory
and three black garments in inventory,
when they exchange the blue garment for a black garment,
then I should have three blue garments in inventory
and two black garments in inventory.

Once a Feature is written, tests can be created. Each sentence on the specification is supposed to correspond to a unit of test code. These sentences can pass arguments down to the test code too (this is mostly done with a test tool, more on this later).

It's important to point out that BDD does not replace TDD, it's an evolution of it. Also, BDD can have some overlap with other test design patterns/models, such as the ones we have covered in this blog like Page Object Model (POM), or Fluent Design Pattern. But BDD works at a slightly higher level than those design patterns, it works at the requirement specification layer of the solution. This means that you can (and maybe should) implement BDD on top of a design model like POM.

BDD has been around for some time (at least 2008-9), but it seems to have picked up in the early 2010s. BDD is however only a methodology, some tools make it easier to implement it in a Programming IDE. We will mention two of them.

Cucumber

We will review what Cucumber is, only from a concept standpoint since there is already a tutorial in this blog that takes you step-by-step on how to use it. You can find it here: Get Started with Cucumber and Azure DevOps!

Having said that, Cucumber is an open-source tool created to implement BDD. It was created in Ruby and meant for Ruby testing. Nowadays it supports other languages such as Java and JavaScript. The open-source version of Cucumber is called Cucumber Open and can be found on GitHub. It seems to be supported by SmartBear who also owns a proprietary version called Cucumber Studio.

Cucumber implements Features files according to the BDD standard and uses a line-oriented design to link the sentences (lines) in the Feature file to test code. This approach is called Gherkin Language, a language used in Cucumber to define test cases (feature files).

Please check out the Cucumber Tutorial that is mentioned above for a step-by-step description on how to implement BDD with Cucumber using Eclipse for your IDE and even integrating your test solution with Azure DevOps.

SpecFlow

As you read this Microsoft Community blog, you may be here to learn how to implement all of this for .NET and on Microsoft's stack. Well, SpecFlow is the most popular (maybe only one?) open-source BDD framework for .NET.

SpecFlow started as a GitHub project in 2009 when a group of developers decided to build a native .NET framework to automate Gherkin feature files. This then evolved into SpecFlow, which was funded by Tricentis in 2019.

Now, SpecFlow has great documentation. If you are interested in this technology, you should check out their Step by Step Getting Started Guide and their Blog.

In this series of posts, we will provide a similar installation/use guide but try to provide added value by updating the steps for Visual Studio 2022 (official documentation still shows 2019) and by focusing a bit more on Microsoft's stack and Azure DevOps integration. We will also try to show you how to avoid the normal bumps along the way and investigate a few C# examples of usual cases where BDD is used in the .NET world.

In the next post, we will also explore Spex, another tool that makes it easier to sync your Feature files with Azure DevOps Test Cases.

Please read ahead!

2. TUTORIAL

Installing SpecFlow for Visual Studio 2022

Step 1. Installing Extension
- The setup starts by installing the SpecFlow Extension which will then install several item templates. For this, start Visual Studio 2022 without opening any project or code. Then from the top menu, go to Extensions>Manage Extensions, then search the Marketplace for SpecFlow for Visual Studio 2022:
- Hit download and restart VS. When it comes back again, go through the dialog to install the extension:

Step 2: SpecFlow Project Template
- Create a new project. You will notice that new templates have been installed. Create a new SpecFlow Project (C#, Windows).
- Use Framework .NET 6.0, choose MSTest for Test Framework, and make sure to add the FluentAssertions Library.
- The template will create a starter SpecFlow solution that should look like this:

You should now have everything you need to start writing Feature files. Let's review the methodology first.

Writing Feature following the BDD guidelines

Let's try to follow the complete BDD process, which as we said, is an evolution of TDD. We will try to now anchor it on Agile planning and later we will integrate it with Azure DevOps.

- To simplify things, let's work with a simple Class Library project that we will use to simulate buying/selling stock.

- For this let's assume the following user story (tracked in e.g., Azure DevOps):

As a StockApp User

I want to purchase a given amount of stock at the latest value

So that I can increase the value of my portfolio

Step 1. Let's write a SpecFlow Feature file with a single scenario to test that story:

Please note that this example shows that you can use parameters in line, which will then be replaced by the values in the Examples table. This is a quick and smart way to implement Data-Driven Testing. Also, note that we are using the BDD keywords for Given, When, and Then to organize our narrative (And takes the value of the previous line). We are "translating" the user story into a BDD style Feature file with statements and validations to accomplish the acceptance criteria.

Step 2. Now comes the fun part, you can right-click each of the lines on the Feature file individually or right-click any space to scope all lines, and "Define Steps". This will create step definitions for each line.

- This process should create arguments for your methods or every parameter you are using in your Feature file.

- You can choose to copy the step definitions to the clipboard or create a new class (.cs) already containing them

If you generate steps for all lines, it should look like this:

Step 3. Now, following the principles of TDD, we have a test, but we don't have the code to test. Let's add a Class Library project into the solution and reference it in the test project:

- At the Solution level, right-click then Add>New Project. Select a C# Class Library project, name it and make sure to use .NET 6.0:

- It should end up looking something like this:

- Then make sure to reference the Class Library project in the Test Project. Right-click References>Add New Reference, then:

Step 4. The next step would be to create the stubs of some empty classes, empty methods, and in general, just reorganize the arguments in the step definitions:

- Please note the use of SpecFlow Fluent Assertions in this line:

newPortfolio.Value.Should().BeGreaterThan(initialPortfolioValue);

This is an assertion, even if we don't use the Assert class. If the comparison fails, it should generate an Exception and fail the test. These fluent assertions are very useful and more efficient at validating collections and objects with many properties for such cases as it is common for API Test solutions.

- Now, the classes, methods, and properties mentioned in this test code don't exist in the Class Library project yet. If you used Visual Studio's feature that lets you create empty stubs with Non-Implemented Exceptions, then if you run your test, they will fail of course. Let's move on to the next step.

Step 5. Following the TDD principles, let's write some app code in the Class Library so that we have something to test:

That's just some simple app code that fits our purpose. Make sure your solution is built, then let's move on to execute the tests.

Running SpecFlow Tests in Visual Studio

If you have followed this tutorial down to this point, you should be good to go and open the Test Explorer from the top menu (Test>Test Explorer) and let it discover all tests. It should look like this:

- Please note that each line of Test Data (Examples) is treated as a different test. You can execute a single data set, the whole test method, even the whole test class. Feel free to Debug and run line by line if you are interested in seeing how the execution progresses.

- If you run all the tests, they should pass successfully:

That's it, you have your first passing SpecFlow tests. Not much data is provided, other than the fact that they passed and the duration. Now let's review how to improve the reporting capabilities.

SpecFlow Reporting Capabilities

Reporting is a very useful capability of SpecFlow. This feature is similar to the Advanced Logger for UI Tests NuGet Package that we described how to implement in this blog but goes a few steps forward in that it provides a dashboard for your test run.

**Note: The way reporting was implemented before SpecFlow 3 was that Reporting was included within the SpecFlow framework and package. However, these reports have been deprecated along with the whole SpecFlow-Runner (check the announcement here). Those reports were also helpful and If you are interested in how these SpecFlow 2 reports used to work, check the documentation here. However, I think that the new reports are just as good, if not better.

The new way to go, starting on SpecFlow 3, is to use SpecFlow LivingDoc. This is a separate NuGet package that you should already have installed in your solution since you are using the SpecFlow Project Template:

SpecFlow LivingDoc can help you generate reports for Azure DevOps or locally. For now, let's review how to create a report locally:

Step 1. Build (and clean) your project.

Step 2. Execute all tests.

A file named TestExecution.json should have been created in this location:

<SolutionDirectory>\<TestProject>\bin\Debug\net6.0

Step 3. The HTML report can only be generated through Command Line (CLI). You will need SpecFlow LivingDoc CLI, but do NOT install the one from NuGet Packages. Instead, open the Package Manager Console and execute this:

dotnet tool install --global SpecFlow.Plus.LivingDoc.CLI

That should install the version that is compatible with your SDK (.NET 6.0).

Step 4. From the same Package Manager Console, execute LivingDoc by providing the folder of your test project and the location of the TestExecution.json file:

LivingDoc feature-folder C:\users\<yourUser>\source\repos\BlogDemo\BlogDemo -t C:\users\<yourUser>\source\repos\BlogDemo\BlogDemo\bin\debug\net6.0\TestExecution.json

That should generate an HTML file like this one:

- I find this report very useful because it includes every data set, all step definitions, scenario, and Feature descriptions and it is easy to filter. You can run this report for full execution of all the features of your application and it should include all in a single report/dashboard.

- Make sure to check out the Analytics tab, to find the summarization (i.e., dashboard) view:

This should be enough for a simple example that includes all the basic components of BDD and SpecFlow in Visual Studio for .NET 6.0.

This post will be the first in a SpecFlow series, please stay tuned for the next ones:

How to integrate SpecFlow BDD Tests with Azure DevOps using LivingDoc and Spex
How to create API Tests with SpecFlow BDD in Visual Studio
How to create Selenium UI Tests with SpecFlow BDD in Visual Studio

Thanks for reading!

References

Continuous Testing with Selenium and Azure DevOps

Microsoft_Testing_Team — Wed, 09 Feb 2022 15:36:47 GMT

By Travis Ray

There are many articles and blogs on how to use the open-source tool, Selenium, to automate testing at the UI layer. This article assumes the reader has found a plethora of materials detailing the creation of Selenium tests and may be wondering what the use of automation is if someone must manually execute it. Creating stable automated tests is most of the battle. However, the real value of automation is realized when the execution and reporting is automatic as well.

If you are using Azure DevOps pipelines, the task is relatively simple to implement. The first step is to include the test project in the Git repo of the Azure DevOps project of the application being tested. In this example, I have created a separate project in the application solution:

Configure the build pipeline with a Copy Files task to push all the artifacts of the build to the build.artifactstagingdirectory.

The next build task will be to publish the contents of the build.artifactstagingdirectory. This will make the dll’s and dependencies of the Selenium tests available to the build agent.

The next step will be to configure the Release pipeline. The UI tests will be executed only after a successful deployment of the application code. Add a new Visual Studio Test task.

For the Test Files property of the VsTest task, specify the path to the dll of the Selenium test project.

If you are unsure about the path to the test binaries, you can check the log from Copy Files task of the of the latest successful Build Pipeline. “D:\a\1\a” will be replaced by “**” in the Test files property of the VsTest Task. This specifies the root directory of the drop.

Check the boxes titled: “Test mix contains UI tests” and “Upload test attachments”

Save the pipeline and create a release run. The tests will now be automatically executed at the end of a successful deployment (Continuous Testing).

The test results can be further explored by clicking on the flask icon of the Release stage.

By default, only the aborted and failed tests are shown. Explore the passed test results by including them in the filter.

No Filters:

If you have implemented the GetScreenshot() method of the Selenium webdriver and AddResultFile() method of the TestContext object into your tests, attachments will be included in the results:

[TestMethod] public void PUL_Categories() { Helper.InitializeChromeDriver(out _driver, "http://<partsunlimited>.azurewebsites.net"); PULPages pulPages = new PULPages(_driver); try { Assert.IsTrue(pulPages.SelectMore()); _driver.GetScreenshot().SaveAsFile(Directory.GetCurrentDirectory() + "/SelectMore.png"); TestContext.AddResultFile(Directory.GetCurrentDirectory() + "/SelectMore.png"); } catch { _driver.GetScreenshot().SaveAsFile(Directory.GetCurrentDirectory() + "/SelectMore.png"); TestContext.AddResultFile(Directory.GetCurrentDirectory() + "/SelectMore.png"); Assert.Fail(); _driver.Quit(); } }

Attachments* tab:

Thank you for reading! If you are interested in more information on this subject, we encourage you to check out or previous posts below:

References:

Monitoring Application Performance with Application Insights | Azure DevOps Hands-on-Labs (azuredevopslabs.com)

Helpful SQL Tools and Tips for Test Engineers - Part 1

Microsoft_Testing_Team — Mon, 22 Nov 2021 18:32:37 GMT

By Cayce Chismar,

Hello everyone, this time we start a new series where we will post some more of the great content from our college in Microsoft Test Team (Principal Customer Engineer) boB 'the ToolMan' Taylor, who is an industry expert in all topics SQL. The theme of this series will be "Helpful SQL Tools and Tips to make your life as Test Engineer" (or Developer, QA, etc.), really anyone who spends time working with SQL.

As some of you may know, we have featured content from boB before, such as his "Learning Justo Query Language" video tutorials. If you are interested in that, please don't miss our post.

Today we will review a tool he created to make the life easier for anyone who spends a lot of time working on SQL Server Management Studio (SSMS). This tool will allow you a quicker and more efficient way to connect to several SQL Servers through SSMS.

When working on SSMS, connecting many different SQL Server instances can get repetitive, having to enter information, Server Name, credentials, etc. each time. If you know a little C# and SQL Management Objects then boB has constructed a Connect To SQL class to help 'connect to any SQL Server instance along with any SQL Azure Database server instance’. The Connect to SQL base class is imported and your class will be derived from this base class. This is meant to be a simpler way to connect to a database, using an UI that looks familiar rather than going through C# config files.

Let’s do a quick walk through of this tool:

Connect to SQL Base

In boB's method, the user will have access to open a connection and any Connection Information needed to perform tasks. boB created a 'Connect to SQL Base' class to be added into your project that lessens inputting data repeatedly, such as Server Name and credentials. This is also useful if a person does not know the name of which database they are needing to connect to; this class will go out and search through all available databases and allow you to pick which database you would like to use. Within boB's ConnectToSQLBase class, there are two standard resources: a default icon for use by the dialog and a default banner for use by the dialog.

Let's dive into the code that connects our database and determines the correct Authentication method!

Cursor current = Cursor.Current; Cursor.Current = Cursors.WaitCursor; // Create our connection information for use by the Results Form ServerConnection sc = newServerConnection(ServerName); // Create a ConnectToSQLBase object to simplify management SqlConnectionInfo sci = newSqlConnectionInfo(sc, ConnectionType.Sql);

First, capture the current cursor and set wait cursor. Then create a ServerConnection object based on the ServerName in the cbServerName ComboBox.

sci.Authentication = SqlConnectionInfo.AuthenticationMethod.NotSpecified; switch(AuthMethod) { case0: // Windows Auth sci.UseIntegratedSecurity = true; break; case1: // SQL Server Login sci.UserName = txtUserName.Text; sci.Password = txtPassword.Text; break; case2: // Active Directory Password Authentication sci.Authentication = SqlConnectionInfo.AuthenticationMethod.ActiveDirectoryPassword; sci.UserName = txtUserName.Text; sci.Password = txtPassword.Text; sci.EncryptConnection = true; break; case3: // Active Directory Integrated Authentication sci.Authentication = SqlConnectionInfo.AuthenticationMethod.ActiveDirectoryIntegrated; sci.UseIntegratedSecurity = true; sci.UserName = Id.Name; sci.EncryptConnection = true; break; }

Based on the AuthMethod selected in the cbAuthentication ComboBox, modify the ServerConnectionInfo object with the appropriate information.

// Use TCP connection sci.ConnectionProtocol = NetworkProtocol.TcpIp; // Set user requested timeout sci.ConnectionTimeout = ConnectionTimeout; // Finally, we can create our SqlConnection SqlConnection con = newSqlConnection(sci.ConnectionString); try { // Now, make sure we can open a connection (to ensure user has rights) // if they don't have rights, it will throw an exception DoWork(con, sci); } catch(Exception ex) { MessageBox.Show(ex.Message, ex.Source, MessageBoxButtons.OK, MessageBoxIcon.Error); } finally { Cursor.Current = current; con.Close(); }

Line 11 calls the DoWork() method, which needs to be implemented in your derived class.

booluseLocalServers = false; DataTable dt = SmoApplication.EnumAvailableSqlServers(useLocalServers); cbServer.Items.Clear(); cbServer.Items.Add("(local)"); // Work item 311 // Only add servers that are at the minimum version or greater foreach(DataRow r indt.Rows) { string[] verParts = r["Version"].ToString().Split('.'); // Only add servers that are at the minimum version or greater if(Convert.ToInt32(verParts[0]) >= MinimumVersion) { cbServer.Items.Add(r[0].ToString()); } } cbServer.Items.Add("<Browse for more...>");

"Note: you must set the minimum SQL Server version number that you want prior to the following call which fills the cbServer drop-down list. You can set this value in your constructor. It currently defaults to 12."

That is all, we hope this information is useful, please follow links below for the source code and original posts and let us know of any questions in the comments section below. Thanks!

The original Archive code is on boB Taylor's GitHub

Other interesting resources in boB's blog::

Thank you all for reading, we hope this information is helpful. Please leave any questions or comments in the section below!

Advanced Logger for UI Tests - now on NuGet Package!

Microsoft_Testing_Team — Tue, 21 Sep 2021 14:48:47 GMT

By Edwin Hernandez

Hello everyone,

We published on this blog some time ago a post about an Advanced Logger for UI tests. You can follow this link if you are interested in reading that post. For an improved version of that Logger that is now on a NuGet package, please continue reading for features, how to install it and a example:

Features:

This logger can be added to a Selenium Test Project running on Visual Studio as a NuGet Package.
The logger is compatible with Selenium, WinAppDriver and Xamarin iOS.
The package makes available an AdvLogger class that can help you log messages into an integrated HTML file.
The messages can contain screenshots (which expand on hover) and optional messages.
Screenshots are embedded into single HTML for easier management.
You can categorize your messages in 4 levels (info, warning, passed & error) and filter in/out as needed.
The HTML log file is added to the TextContext as an attachment. This makes it part of the results file which you can see in Visual Studio and gets uploaded automatically to Azure DevOps if you are running your UI-test project from Azure DevOps.
Adding an attachment to the TestContext was initially supported only for MSTest. Starting on version 1.0.0.9, this is now supported for NUnit.

If you want to know more about setting up a Azure Pipeline that contains a Selenium UI test project, please follow the link posted above, it has some intro into this topic. We also have a upcoming article about this that will be publish in the coming weeks.

Walkthrough

Now that we have seen the features, let's do a walkthrough of how to install it and use it on a sample Selenium C# project.

We are going to use a sample project that has a single test that validates that a "User is able to create a product review". The target app is a demo website of OpenCart.

The following would be our starting code:

The code has a Test Class and a single Test Method. All libraries related to Selenium and Test Libraries have already been imported into a Visual Studio MSTest Project. This project implements a Page Object Model for the design (if you want to learn more about selenium designs, please check this article).

The files on the solution would look like this:

The page classes contain the actual selenium code that interacts with the browser. This model allows for these classes to be reused and helps with maintenance.

Here is one of the page classes:

public class StoreHome { public static void SearchItem(IWebDriver driver, string searchStr) { driver.FindElement(By.Name("search")).SendKeys(searchStr + Keys.Enter); } public static void LoadStoreHome(IWebDriver driver) { driver.Manage().Window.Maximize(); driver.Url = "https://demo.opencart.com/"; } }

The test class is as follows:

[TestClass] public class OpenCartTests { IWebDriver driver; [TestInitialize] public void Initialize() { driver = new ChromeDriver(@"C:\Users\edwinh\source\repos\BlogDemoNuget\BlogDemoNuget"); } [TestMethod] public void OpenCart_ValidateReviewSubmision() { StoreHome.LoadStoreHome(driver); StoreHome.SearchItem(driver, "iphone"); SearchResults.SortItemsBy(driver, 4); SearchResults.OpenItem(driver, 1); ItemPage.GotoReviews(driver); ItemPage.PostAReview(driver , "UITester", "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam eget nisi a nunc varius dictum at sit amet diam. Donec posuere gravida dui vitae elementum. In efficitur sodales erat a consectetu" , 4); string msg = ItemPage.GetReviewMessage(driver); Assert.AreEqual("Thank you for your review. It has been submitted to the webmaster for approval.", msg); } [TestCleanup] public void CleanUp() { driver.Quit(); }

You can get this whole project by cloning it from this public GitHub repo here, which would include all the other page classes.

For now, please look at the OpenCart_ValidateReviewSubmision Test Method, as you can see, since this we are implementing the Page Object model, the test is relatively easy to follow.

First load the store page > then search for an item > sort the list > open the first item > go to reviews > and post a review. We additionally have a validation.

The test works as it is but Selenium and MSTest do not produce any kind of report other than passed/fail.

This is where the Advance Logger for UI-tests comes into play.

To add it first search for it from the NuGet Package Manager:

Install it and solve any dependency incompatibilities with your project.

Now that the library is available for your project, the following changes are needed:

1) Create a Test Context in your test class (but outside of your test method):

private TestContext testContextInstance; public TestContext TestContext { get { return testContextInstance; } set { testContextInstance = value; } }

2) Initialize the AdvLogger

AdvLogger.imageSize = 400; //In Pixels. Default is 300 AdvLogger.includeSnapshootsForPassAndError = true; //Only Pass and Error levels include snapshots. Default is true. AdvLogger.onHoverImageSizeAsPercentageOfWindow = 50; //How big you want your screenshots to expand to. Default is 100% of browser screen width. AdvLogger.Current.LOGLEVEL = 1; //Logger will include only messages at specified level or above. E.g. Level=1 means no filters //levels: 1=info, 2-passed, 3-warning, 4-error

3) Add some Log Messages in your test method and page object classes

One of the Page Object classes:

public class StoreHome { public static void SearchItem(IWebDriver driver, string searchStr) { driver.FindElement(By.Name("search")).SendKeys(searchStr + Keys.Enter); AdvLogger.Current.LogMessage("Passed", driver); } public static void LoadStoreHome(IWebDriver driver) { driver.Manage().Window.Maximize(); driver.Url = "https://demo.opencart.com/"; AdvLogger.Current.LogMessage("Passed", driver); } }

For a better report, you will need to add a LogMessage to each Page Object method, so that there is a sequence of screenshots or messages in your HTML report. LogMessage uses reflection to get the class and method name and uses this to build the report.

Test Method:

[TestMethod] public void OpenCart_ValidateReviewSubmision() { AdvLogger.Current.InsertDividerForNewTest(); StoreHome.LoadStoreHome(driver); StoreHome.SearchItem(driver, "iphone"); SearchResults.SortItemsBy(driver, 4); SearchResults.OpenItem(driver, 1); ItemPage.GotoReviews(driver); ItemPage.PostAReview(driver , "UITester", "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam eget nisi a nunc varius dictum at sit amet diam. Donec posuere gravida dui vitae elementum. In efficitur sodales erat a consectetu" , 4); string msg = ItemPage.GetReviewMessage(driver); Assert.AreEqual("Thank you for your review. It has been submitted to the webmaster for approval.", msg); AdvLogger.Current.LogMessage("Info", driver, "Validation passed"); AdvLogger.Current.LogMessage("Info", driver, "REACHED END OF TEST CASE"); }

You need to insert a New Test Divider at the start of your test method. The other two "Info" messages are optional but I think it gives the report a nice wrap up.

4) EndLogging to generate html

[TestCleanup] public void CleanUp() { driver.Quit(); var fileName = AdvLogger.Current.EndLogging(TestContext); System.Diagnostics.Process.Start("explorer.exe", fileName); }

EndLogging processes all the messages and screenshots into an HTML report and attaches it to the TestContext. Once it's in the TestContext it's part of the Results File in Visual Studio and it gets uploaded into Azure DevOps if you run this project as part of an Azure Pipeline.

Process.Start just opens up the report at the end of each test, this is optional.

The end result is an HTML report that would look like this:

I hope this Logger continues to help anyone that needs it. You can find these resources in the following locations:

NuGet Package URL: https://www.nuget.org/packages/AdvUITestLogger

GitHub Repository of the Project described above: https://github.com/edwin-hernandez-maynez/BlogDemoNuget

Please leave any questions in the comment are below. Thanks for reading!

P.S. We added support for NUnit:

NUnit Example

Starting with NuGet Package 1.0.0.9, the Logger supports NUnit (in addition to MSTest). The dependency on a test framework is only used when calling the EndLogging method to generate the report and attach it to the TestContext.

You can find in this branch of the repo, an example implementing NUnit and the AdvUITestLogger:

https://github.com/edwin-hernandez-maynez/BlogDemoNuget/tree/UsingNUnit

Learning Kusto Query Language - A tool for performance test engineers

Microsoft_Testing_Team — Mon, 24 May 2021 19:06:01 GMT

Video series by boB Taylor, intro by Edwin Hernandez

Hello everyone! This time we bring you a series of video tutorials by boB Taylor, who is a member of the Microsoft's Performance & Quality Services Team. He recently completed this video series about Kusto Query Language, and this is a great opportunity for us to compile it into a single place for you along with some introductory information.

What is Kusto Query Language (KQL)?

You may be asking yourself: What is Kusto? What is KQL? And why would this matter for a Quality and Performance Test Blog? Well, Kusto itself was the internal code name for Azure Data Explorer, and Kusto Query Language (KQL) is the primary means of interaction with it. KQL allows you to send data queries, process data, and return the results of this processing without modifying the data or metadata. Now, why would this matter for Test Engineers? Let's first define what Azure Data Explorer can do for you:

Azure Data Explorer

Azure Data Explorer is a service that allows you to store log and telemetry data. It can handle many data streams emitted by several entities concurrently (websites, LOB apps, CRMs, Social data sources, IoT devices, Cloud components, etc.). This data is collected and stored. Data Explorer then provides a way for you to analyze this large volume of data, perform complex queries and drill down into specific events.

Therefore, Azure Data Explorer provides a great way for you to perform diagnostics, monitoring, and reporting of all components of your application and environment. This kind of analysis is key for performance testing and application optimization.

Kusto Query Language(KQL)

KQL is a read-only query language. The syntax is similar to SQL, but it was created specifically to work with large datasets in Azure. Since it's read-only there are no update or delete clauses. It is based on relational management systems, which use schema entities, and is organized into a hierarchy like SQL's databases tables and columns.

Other Azure Services

KQL is the primary mean to query data from Azure Data Explorer; however, it is also used to interact with the following inter-related Azure services:

Application Insights
Log Analytics
Azure Monitor, and again:
Azure Data Explorer

Microsoft documentation

If you want to learn more from the official Microsoft documentation, make sure to check out the following:

What is Azure Data Explorer? Check this one to learn about the possible data sources and workflows.
What are Application Insights? Fundamentals, how to implement the telemetry, do analysis, etc. We hope to bring you a detailed guide on this blog in the future.
Azure Monitor Overview. Solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
Overview of Log Analytics. Tool in the Azure portal used to edit and run log queries with data in Azure Monitor Logs.
Getting started with Kusto. Check this one to learn the basics about KQL, where there is a tutorial and samples.
For a more hands-on kind of learning, check out this lab in Microsoft Learn: Monitor performance of virtual machines using Azure Monitor for VMs

boB Taylor's Video Series

If you are done leafing through the official documentation above and want a much more detailed training. boB created a series of video tutorials where he goes into detail on KQL, from the fundamentals to complex functions, passing through the most common operators:

1	What is KQL and why should I care?	Link to video
2	The Editors or how do I write and execute KQL queries	Link to video
3	Our first KQL operators	Link to video
4	Wait where can I consume Kusto data?	Link to video
5	Summarize	Link to video
6	Project, extend, and explain	Link to video
7	Distinct, sample-distinct, top, and top-nested	Link to video
8	Scalar functions part 1	Link to video
9	More scalar operators!	Link to video
10	Fun with datetime, timespan and date_part	Link to video
11	Conditional logic and strings	Link to video
12	Advanced aggregation operators	Link to video
13	Let, join, and union	Link to video
14	Datatable, prev, next, row_cumsum, and materialize	Link to video
15	Range, make-series	Link to video
16	Series_decompose family of functions	Link to video
17	The remaining time series functions	Link to video
18	Machine Learning Plugins	Link to video
19	User Analytics Plug-ins	Link to video
20	Last on KQL - charting	Link to video

In Conclusion…

If you want to monitor and diagnose the performance of your application in Azure, KQL is a key resource. It is a good skill to have in your portfolio, especially if you are involved in load testing. Please make sure to check our other article about a Collection of Useful Tool for Performance Test Engineers, and please leave any questions in the comments section.

Thanks!

Test Automation and EasyRepro: 05 - Adding EasyRepro Tests to Azure DevOps

Microsoft_Testing_Team — Mon, 26 Apr 2021 16:42:33 GMT

The following is the fifth and final on a series of articles by @Ali Youssefi that we have been cross-posting into this Test Community Blog for a couple of months now. These articles were first published by Ali in the Dynamics community but since the topic related with Testing, Quality and Selenium, we though it would makes sense to publish here as well.

If you didn't get a chance to catch the previous parts of these series, please have a look the links below:

Otherwise, please read ahead!

Summary

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. This article will cover incorporating EasyRepro into a DevOps Build pipeline, allowing us to begin the journey toward automated testing and quality. We will cover the necessary settings for using the VSTest task and configuring the pipeline for continuous integration and dynamic variables. Finally we will review the test result artifacts providing detailed information about each unit test and test run.

Getting Started

If you haven't already, please review the previous articles showing how to create, debug and extend EasyRepro. This article assumes that the steps detailed in the first article titled Cloning locally from Azure DevOps have been followed. This approach will allow our test design team to craft and share tests for quality assurance across our DevOps process.

The Run Settings File

The run settings file for Visual Studio unit tests allow variables to be passed in similar to the app.config file. However this file is specific to Visual Studio tests and can be used in the command line and through Azure DevOps pipelines and test plans. Here is a sample to create a runsettings file.

The image above shows how to implement the TestRunParameters needed for EasyRepro unit tests. You can also find an example in the Microsoft.Dynamics365.UIAutomation.Sample project called easyrepro.runsettings. The runsettings file can be used to set the framework version, set paths for adapters, where the result artifacts will be stored, etc. In the Settings section below we will point to a runsettings file for use within our pipeline.

The ClassInitialize Data Attribute

The ClassInitialize data attribute is used by Visual Studio unit tests to invoke the constructor of a test class. This decoration coupled with the runsettings file will allow us to pass in a TestContext object containing the run parameters.

Properties

The configuration values from the runsettings file are included in the Properties object similar to the app.config file. For usage with EasyRepro we will want to leverage the .ToSecureString extension method which will help us when logging into the platform. Below is an example using this extension method.

Setting up the Build Pipeline

In the first article Test Automation and EasyRepro: 01 - Getting Started, we discussed how to clone from GitHub to a DevOps Git repository which we can then clone locally. The examples below follow this approach and assume you have cloned locally from Azure DevOps Git repo.

The first thing to setting up the Build Pipeline is to navigate to the DevOps organization and project. Once inside the project, click on the Pipelines button to create a Build Pipeline. Tasks here require resolving the NuGet package dependencies, building a solution with MS Build and running unit tests using the VS Test task as shown in the image below.

The core task is the VsTest which will run our Visual Studio unit tests and allow us to dynamically pass values from the Build pipeline variables or from files in source control. The section below goes into the VsTest task, specifically version 2.0.

Reviewing the VsTest task

Test Files

The test file field needs to point to a working directory, dictated by the Search folder field, to locate the compiled unit test assemblies. When using the default task this field looks for assemblies with the word test. If you're starting with the Sample project from EasyRepro you will need to change this to look for the word Sample as shown above. When this task is run you can confirm if the correct assembly is found in the log for the task.

Test Filter Criteria

The test filter criteria field is used to help limit the scope of the unit tests run within the unit test assembly. Depending on the data attribute decorations you can restrict the unit test run task to only run specific tests. The criteria can be somewhat challenging if you haven't worked with them before so I'd suggest using the Visual Studio Command Prompt to test locally to better understand how this will work in Azure DevOps Pipelines.

The above image shows an example of using the TestCaseFilter argument to run a specific test class. This argument can be used to run specific classes, priorities, test categories, etc. For instance

More information on the test filter criteria can be found here.

Settings File

The settings file field works with the vstest.console.exe utilizing the "/Settings" argument but allows the ability to pick a file from the repository directly. This field is customizable to also work with Build Pipeline variables which I'll describe next.

Override Test Run Parameters

Overriding test run parameters is useful when we want to reuse the same test settings but pass in variables from the Build Pipeline. In the example below I'm replacing the parameters from the runsettings file on the left with Build Pipeline variables on the right.

Below are my pipeline variables I've defined. This allows me to check in a runsettings file but not have to modify parameters before committing. The values can be plain or secure strings which you will have to take into account if you plan to use one or the other. These variables can also be provided at run time when we queue the build.

Enabling the Continuous Integration Trigger

Enabling the continuous integration trigger allows developers to craft their unit tests and have our build pipeline run on push of a commit. This is configured from the Triggers tab on the pipeline which will bring up the above image. To enable, check the box for 'Enable continuous integration' and set which branch you want to have this fire off on. This doesn't restrict a tester from queuing a build on demand but does help us move forward towards automation!

Running the Build Pipeline

To kick off the build pipeline, commit and push changes to your unit tests as you would any Git commit. Once the push is finished you can navigate to the Azure DevOps org and watch the pipeline in action. Below is a recording of a sample run.

Exploring the Results File

The results of the unit tests can be found in the build pipeline build along with the logs and artifacts. The test results and artifacts are also found in the Test Runs section of the Azure Tests area. The retention of these results are configurable within the Azure DevOps project settings. Each build can be reviewed at a high level for various test result statuses as shown below:

The summary screen includes the unit tests ran and information about failed tests that can be used to track when a regression began. Columns shown include the last time a test ran and what build it began to fail on.

When investigating a failed unit test a link to a new or existing bug can be added. This is useful to help track regressions and assign to the appropriate team. Bugs can be associated from the test run or specific unit test and include links to the build, test run and test plan. The exception message and stack trace are automatically added if linked from the failed unit test.

Each test run includes a test results file that can be downloaded and viewed with Visual Studio. The test artifacts can also be retained locally for archiving or reporting purposes. The contents of the test can be extracted and transformed to be used by platforms such as PowerBI or Azure Monitor.

Its key to point out that when used with a Azure Test Run these results can be retrieved via the API and reported on directly. Below is an image of the response from the Test Run.

Next Steps

Including Unit Tests as a Continuous Delivery Quality Gate

Building and running our EasyRepro tests with Build Pipelines represent an important first step into your journey into DevOps as well as what has been called Continuous Quality. Key benefits here include immediate feedback and deep insights into high value business transactions. Including these types of tests as part of the release of a Dynamics solution into an environment is paramount to understanding impact and providing insight.

Including Code Analysis as a Continuous Delivery Quality Gate

One thing I'd like to point out is that UI testing can help ensure quality but this should be coupled with other types of testing such as security testing or code analysis. The PowerApps Product Group has a tremendously valuable tool called the PowerApps Project Checker for code analysis. Project Checker can help identify well documented concerns that may come up as part of our deployment. This tool can be used via PowerShell or from the PowerApps Build Tasks within the Visual Studio Marketplace. It can also be run from the PowerApps portal in a manual fashion if desired.

This code quality step can be included as part of the extraction of modifications from your development or sandbox environments or as a pre step before packaging a managed solution for deployment. For additional detail there is a wonderful post by the Premier Field Engineer team documenting how to include this task into your pipelines. Special thanks to Premier Field Engineers' Paul Breuler and Tyler Hogsett for documenting and detailing the journey into DevOps.

I highly recommend incorporating this important step into any of your solution migration strategies even if you are still manually deploying solutions to better understand and track potential issues.

Scheduling Automated Tests

Scheduling tests can be done in various ways, from build and release pipelines to Azure Test Plans. For pipelines triggers can be used to schedule based off a predetermined schedule. Azure Test Plans allow for more flexibility to run a specific set of tests based off of test cases linked to unit tests. To find out more about setting this up, refer to this article.

Conclusion

This article covers working from a local version of EasyRepro tests and how to incorporate into a Azure DevOps Build Pipeline. It demonstrates how to configure the VsTest task, how to setup triggers for the Build Pipeline and how to review Test Results. This should be a good starting point into your journey into DevOps and look forward to hearing about it and any question you have in the comments below.

At the Microsoft Test Team, we have followed this tutorial and often use EasyRepro for our UI-Test related needs for Dynamics 365, please stay tune for a Tips and Trick related to EasyRepro, until next time!

Simulating Targeted Throughput for Load testing with JMeter

Microsoft_Testing_Team — Mon, 22 Mar 2021 18:15:25 GMT

By Tejas Shinde

JMeter is quite a versatile tool when it comes to Load Testing. Developers can use it to simulate various usage patterns; it also has various plugins that enable it to manipulate these usage patterns.

When we want to simulate the application load by using APIs, we think about targeted throughput. In such scenarios, typically, we want to load the system with specific throughput (requests per second) for a specified duration.

In this post, we will take a look at some JMeter plugins that we can use for controlling throughput and ways to use these plugins more efficiently.

We will use the following JMeter plugins for this example:

Dummy sampler - To simulate request samplers
3 Basic Graphs - To monitor the active threads and transaction rate (throughput)
Custom Thread Groups - For Concurrency Thread Group

In the following example, we will try to simulate the load of ~20 req/sec. There are various ways you can control the rate at which you can generate the requests; the most commonly used ways are either introducing think time in between the requests or using any of the throughput controllers. The JMX solution can be found at the bottom of this article

Every thread group uses the same two samplers that are under the Test Fragment "Request Samples" using the module controllers.

For "First Sampler", response time is set to "${__Random(3000,5000)}" and for "Second Sampler" it is set to "${__Random(5000,10000)}"

Throughout this test plan, all the thread groups are set to use 500 threads. And will be executed for ~5 minutes. Typically, the total number of threads needed to achieve a throughput can be calculated by using the following formula Throughput * Avg. Response Time / 1000. To be on the safer side, instead of Avg. Response Time, we use Maximum Response Time in the formula. However, for demo purpose, we are going to use 500 as thread count in the below examples

We will discuss the following cases:

Without any throughput controlling measures
With sleep time (Using Constant Timer)
With Constant Throughput Timer
With Throughput Shaping Timer
With Concurrency Thread Group with Throughput Shaping Timer

Case 1: Without any throughput controlling measures

In this case, no throughput controlling is introduced. This gives us a rough idea about how much throughput we can achieve with 500 threads without having any throughput control.

Results:

When we run this thread group, the following are the results of the Aggregate Report:

In my case, JMeter was able to generate 81.2 requests/sec. (It may differ in your case based on the various factors)

Throughout the execution, 500 threads were active and sending the requests.	Throughput graph looks consistently distributed

As we can see, without using any throughput controlling measures, 81.2 req/sec were generated using 500 threads.

Case 2: With sleep time (Using Constant Timer)

In this case, throughput will be controlled by introducing sleep time in between threads. In this example, a Constant Timer with 15000 milliseconds is used for sleep time. You may need to try it out in a few runs to find the optimum value of sleep time to get the desired throughput.

Results:

Throughout the execution, 500 threads were active and sending the requests.	As we used sleep times, request generation happens in bursts where throughput will be balanced as a result of sleep times

Case 3: With Constant Throughput Timer

In this scenario, we will use Constant Throughput Timer. This timer will also work similarly to that of Constant Timer. It will pause the threads based on periodically based on the current value of throughput to match up the target throughput value. In this case, an advantage will be sleep time will be calculated automatically based on the target value.

*Note that the throughput value is in terms of req/min

Results:

Throughout the execution, 500 threads were active and sending the requests.	Throughput graph looks very much similar to the graph from the previous case, where we manually paused the threads

Case 4: With Throughput Shaping Timer

In this case, we are using a JMeter plugin - Throughput Shaping timer. This plugin also works similarly to Constant Throughput Timer, which pauses the threads based on the target Throughput. However, a couple of advantages are the request bursts are more consistent in nature, and the samples lie very close to the median value, as can be seen in the throughput graph. Also, this plugin will allow us to simulate various levels of throughput values over a period of time which is not possible in the case of Constant Throughput Timer as it can target only a specific value.

Results:

Throughout the execution, 500 threads were active and sending the requests.	With Throughput Shaping Timer, the graph looks more consistent, and the sample's distribution is close to the median value of the throughput, which has been set as a target

Case 5: With Concurrency Thread Group with Throughput Shaping Timer

So far, in all of the previous cases, we have used 500 threads to generate the throughput of 20 req/sec. Depending upon the total number of requests and response time of those requests, it will be quite a calculation to identify the optimum number of thread count that is needed to generate the target output. In cases where the response time is quite low, we will need much less than 500 threads to generate that throughput. And similarly, in situations where response time is high, we may need more than 500 threads to generate the desired throughput.

To calculate the number of threads that are needed to generate the target throughput, we will use another plugin from JMeter in conjunction with Throughput Shaping Timer. Also, to generate the minimum amount of threads, we will use the Schedule Feedback function from JMeter, which will define the number of threads that should be running concurrently to generate the target throughput.

Following is the Concurrency Thread Group, which is from the "Custom Thread Groups" plugin. Target concurrency is defined using the feedback function that will work based on the feedback from Throughput Shaping Timer to provide the latest count of threads that are needed to achieve the target of the Throughput Shaping Timer.

Feedback function parameters can be explained as follows:

ThroughputShapingTimer - Name of the element. In this case, the name of the Throughput Shaping Timer
5 - Number of threads at the start
500 - Maximum allowed threads
10 - Number of spare threads in the pool

Results:

So as you can see, by using less than 180 threads, we were able to generate a throughput of 19.1 (It’s less than 20 as a result of throughput ramp up and ramp down steps defined in Throughput Shaping Controller).

Maximum threads used during this run were less than 180.	Maximum threads used during this run were less than 180.

Caution:

While using the concurrency thread group with Throughput Shaping Timer, makes sure that you specify the sufficient number of threads as if the number is not high, the plugin will show a warning message in the logs stating that there are not enough threads available in the pool to generate the target throughput
Similarly, make sure that the specified thread count is not too high as it may result in a spike in throughput at the end of the test (as seen in the above throughput graph) while JMeter is shutting all the threads

Conclusion

Hopefully, this post has helped you learn how to control the throughput in your Load Test project by using the mentioned plugins. Please find references to the plugins and manual from JMeter below. You can also find an attachment containing the scenario that was used to write this article (JMX file).

Please leave us any questions or comments too. Thanks for reading!

References:

JMeter plugins wiki - https://jmeter-plugins.org/wiki/Start/
JMeter user manual - https://jmeter.apache.org/usermanual/

Attachment:

Test Automation and EasyRepro: 04 - Monitoring and Insights with EasyRepro

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:31:02 GMT

The following is the fourth on a series of articles by @Ali Youssefi that we have been cross-posting into this Test Community Blog for a couple of months now. These articles were first published by Ali in the Dynamics community but since the topic related with Testing, Quality and Selenium, we though it would makes sense to publish here as well.

If you didn't get a chance to catch the previous parts of these series, please have a look the links below:

Otherwise, please read ahead!

Summary

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. In this article we will discuss monitoring execution timings and delivering these to Azure Application Insights. We will look into the various timings within an execution, how to add tracing statements and finally to push events to Application Insights.

Getting Started

If you haven't already, please review the previous articles showing how to create, debug and extend EasyRepro. This article assumes that the reader has this knowledge and will focus specifically on monitoring.

Command Timings

Commands in EasyRepro unit tests represent an interaction with the browser to perform an action. Each command has a name, a start time, end time and other properties such as how many times it was attempted and was it successful. What's returned to the unit test is a list of command results represented by the ICommandResult object.

Going to the definition for ICommandResult we see the following properties:

These properties represent pretty much everything we need to know about a command result. However since this is a interface, it can be extended to include any other metric needed.

Now that we have the definition we need to dig deeper to better understand how these properties are populated. But before we get into that let's review the object that implements the ICommandResult interface: the BrowserCommandResult class.

Exploring the BrowserCommandResult class

The image above shows the constructor for the BrowserCommandResult class and three important attributes that represent collecting the start and stop time stamp as well as the value returned.

Exploring the BrowserCommand.cs file

The BrowserCommand object is where the actual command gets executed. It contains options describing tracing, retry policies, exception policy, etc. What's most important is the Execute method which performs the action from the unit test.

When a command from a unit test is run (i.e. GetValue), it ultimately comes to this method to be performed. In the image above you can see the retry attempts are set as well as the tracing and timing (result.Start()). Once the command has finished the result and timings are returned in the BrowserCommandResult object.

Execution Times

The execution time for a command represents how long the command took to execute as shown in the Execute method in the BrowserCommand class result.Start and result.Stop methods. Since we are focusing on timings these two methods are key. They are used to calculate the execution time we will use to monitor the performance of our unit test. These methods are shown in the BrowserCommandResult image above and return a simple timestamp.

The execution time calculation takes the StopTime property and substracts from the StartTime to provide a number in milliseconds. This number is then set to the return value of our unit test command.

Think and Transition Times

You may have noticed above a couple of values for Think and Transition Times.

Think times are built in mechanisms to put a thread to sleep. They can be used to allow actions to complete or represent a user wait in the UI. In the image above we see that the ThinkTime value is set to 2000 which is because inside of the OpenApp method the Browser.ThinkTime method is run.

The Transition time value represents the time from the previous command end to the current command's start time and removing think times in between.

How is all of this calculated?

Inside of the InteractiveBrowser class is a method called CalculateResults which is called from the Execute method described above. In the image above you can see think time is straight forward, it comes from the CommandThinkTimes property.

The transition time is a calculation adds the seconds and millisecond differences between the previous command and current command then subtracts the think time. All three of these timings can help us better understand where long running commands are coming from and where we could possibly tune unit tests by reducing think times between unit test commands.

Logging and Monitoring

Debug and Trace

The ability to provide output statements is essentially for any type of unit testing including ones built using the EasyRepro framework. Typically you see statements such as Console.WriteLine or Debug.WriteLine used in the unit test link in the image below.

Its important to note that these same lines can be added to the EasyRepro framework if you choose. Here are two images showing the output from the Debug window and the output from the test result itself.

Debug window:

Test Result Window:

You can see the test output shows only the WriteLine methods I added while removing the debug information from Visual Studio. Utilizing this technique is key to providing detailed information to the developer and testing teams to understand how a unit test is performing. This output will also be included in the test results file which will help us once we begin automating unit testing.

Application Insights

Application Insights is an extensible Application Performance Management (APM) service that can be used to monitor applications, tests, etc. Source: Application Insights Overview

In this section we will cover how to setup the Telemetry Client to be used by your unit tests and how to push command timings to Application Insights. Its assumed you have already setup an Application Insights resource and have the instrumentation key. For help on setting this up refer to this article detailing enabling and configuring Application Insights.

Setup Application Insights Telemetry Client

Begin by ensuring you have the Microsoft.ApplicationInsights NuGet package installed. As of this writing I'm using the v2.4.0 which is included in the Microsoft.Dynamics365.UIAutomation.Sample unit test project from GitHub.

Next, navigate to your app.settings file and add the Instrumentation Key to the AzureKey key.

Once that's done we need to setup the TelemetryClient and give it our key. Once complete we are now ready to send messages to Application Insights.

Push Timings to Application Insights

Once the unit test has completed all of the EasyRepro actions we are going to send the CommandResults collection to Application Insights. From the previous section we detailed that each command result includes the Start and End times as well as calculated timing such as execution, think and transition times.

In the image above you can see how to loop through each command result and set the property and metric bags for the CustomEvent table in Application Insights. The telemetry client includes methods for each type of table in Application Insights so you can also include items like exceptions, navigation timings, etc in the appropriate table. Review this document for additional information.

Next Steps

Expanding messages sent to Application Insights

The example above shows how to collect and send timings for each command to Application Insights which is useful to understand how the test and platform are performing. In addition you may want to think about adding additional contextual details about each unit test run such as who is running the test, where its running and information about the org that the test is running on. As you can see the property and metric bags are dictionaries that can be added to to include these detail points. There are also table columns native to each specific table that can be leveraged as well to house these data points.

One area of the platform that gets overlooked is the Performance Center which is extremely useful for understanding page load timings. These markers are accurate representations of a page load. They can be extracted and sent to Application Insights as well. Check out the SharedTestUploadTelemetry sample unit test for more information. This test uses the legacy UI however EasyRepro can be extended to utilize the Performance Center in the unified interface as well. Please refer to the previous section for extending EasyRepro for techniques to accommodate this.

Conclusion

In this article we have explored EasyRepro commands and how the timings for each command are gathered. We discussed logging techniques within our unit tests. We setup Application Insights within an EasyRepro unit test and pushed messages from our test. Finally we discussed adding additional data points to our messages.

Please refer to the the previous articles for extending and debugging EasyRepro unit tests. For the next article we will work to incorporate changes to EasyRepro unit tests into an Azure Build Pipeline using Continuous Integration. This will be our first step into adding EasyRepro as a quality gate for our DevOps process!

Get Started with Cucumber and Azure DevOps!

Microsoft_Testing_Team — Thu, 03 Jun 2021 12:31:01 GMT

By Anusha Ammaluru

This time we bring you a blog article about Cucumber, Selenium and Integration with Azure DevOps, let's get started and welcome to the journey to learn Cucumber.

The blog post will cover the following topics:

Cucumber Introduction
Setup Cucumber with Selenium in Eclipse
Cucumber Basics
Eclipse Integration with Azure DevOps

Cucumber Introduction

Cucumber is a tool that supports Behaviour-Driven Development(BDD). It lets us define application behavior in plain meaningful English text using a simple grammar defined by a language called Gherkin. Cucumber itself is written in Ruby, but it can be used to “test” code written in Ruby or other languages.

Cucumber is one of the most powerful tools. It offers us the real communication layer on top of a robust testing framework. The tool can help run automation tests on wide-ranging testing needs from the backend to the frontend. Moreover, Cucumber creates deep connections among members of the testing team, which we hardly found in other testing frameworks.

What is Gherkin? It is a simple, lightweight, and structured language, which uses regular spoken language to describe user requirements and scenarios. Examples of regular spoken language are English, French, and around 30 more languages. Gherkin contains a set of syntax/keywords

Feature: Defines the feature (generally a user story)

Given: Specifies the pre-condition of the test

And: Defines additional conditions of the test

Then: States the post-condition/expected result of the test

Key points to note:

The test is written in plain English, which is common to all the domains of your project team.
This test is structured that makes it capable of being read in an automated way. Thereby creating automation tests at the same time while describing the scenario.

Setup Cucumber with Selenium in Eclipse

Setup java
- Download the latest version of Java from https://www.oracle.com/technetwork/java/javase/downloads/index.html
  - Download .exe file
- Install Java by double-clicking on the .exe file and proceed with default options
- Set up Java Environment Path
  - Type Env in the windows search and click on the ‘Environment Variables’ option
  - Click on Path in the System variables and click the ‘Edit’ button
  - Add folder path where the JDK is installed and click the 'OK' button
  - Go to Command Prompt and type java -version If you see a screen like below, Java is installed
Setup Eclipse
- Download Eclipse Photon from https://www.eclipse.org/downloads/packages/release/photon/rc3, if you are going to maintain the repo in Azure DevOps, I had faced issues earlier with other bundles.
- Select Eclipse IDE for Java Developers and Install
- Launch the eclipse and select a workspace folder to save your repo

Install Cucumber Plugin for Eclipse
- June, 2021 UPDATE: The recommended way to install cucumber is now through the Eclipse Marketplace, although installing it through the Eclipse Update Site is still an option (with the new URL).
- Check here for all the supported ways to install Cucumber for Eclipse (https://github.com/cucumber/cucumber-eclipse-update-site).
- To install through the Eclipse Marketplace:
  - Directly from the Eclipse marketplace. Go to Help > Eclipse Marketplace
  - Search Cucumber Eclipse Plugin.
  - Or, go to this url: https://github.com/cucumber/cucumber-eclipse-update-site, and drag and drop the Install button in your running Eclipse workspace.
- To install through the Eclipse Update Site:
  - From Eclipse, go to menu Help > Install New software
  - Work with: https://cucumber.github.io/cucumber-eclipse/update-site
  - Select the check-box for Cucumber Eclipse Plugin
  - Select Next as per the instruction shown during installation.
  - Restart your Eclipse after completion of instruction.
  - Now latest version of cucumber-eclipse plugin is installed successfully in your Eclipse.

Download Cucumber JARS from Maven Repo
- Go to https://search.maven.org.
- Search for cucumber-core in the Central Maven Repository.
- Download jar file
- Similarly, search for all the below libs in the Maven repo and download JAR’s

Download Selenium
- Download WebDriver Java client from Selenium website
- Extract the files and save it in your local folder

Configure Eclipse with Cucumber and Selenium WebDriver libs
- Go to Eclipse : File-> New -> Project -> Select Maven
- Click the 'Next' button
- Filter Cucumber and select cucumber-archetype
- Enter Group Id and Artifact Id and click the ‘Finish’ button
- Add Selenium Jars
  - Right-click on Project ‘CucumberTest > Select Properties > Java build path. Then navigate to the Libraries tab and click Add External JARs.
  - Browse to the local folder where the selenium jars are saved and select all the jars and add
  - Go to the lib folder and add all the remaining Jars
  - Click ok
- Add Cucumber Jars
  - Right-click on Project ‘CucumberTest > Select Properties > Java build path. Then navigate to the Libraries tab and click Add External JARs.
  - Browse to the local folder where the cucumber jars are saved and select all the jars and add
  - Click the 'Ok' button
  - Now we are all set 😊

Cucumber Basics

Cucumber Feature File
- An entry point to the Cucumber tests. It contains a list of scenarios/test cases written in natural English language using Gherkin keywords
- Create a package in eclipse project -> name it as ‘Features’
- Right-click on the ‘Features’ package and create a file -> name it as ‘DemoTest.feature’
- Add a Feature: Name and description and a scenario underneath like the below example
- Step Definitions
  - Add selenium java code in the step definition methods corresponding to feature files.
  - Sample code:

@Given("user is on home page")

public void user_is_on_home_page() {

driver.get("https://pul-ai-anu.azurewebsites.net/");

driver.manage().timeouts().pageLoadTimeout(100, TimeUnit.SECONDS);

}

@When("I click on login link")

public void i_click_on_login_link() {

driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

}

@When("I enter username {string}")

public void i_enter_username(String email) {

driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

}

@@When("I enter password {string}"

public void i_enter_password(String password) {

driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

}

@When("I click on login button")

public void i_click_on_login_button() {

driver.manage().timeouts().pageLoadTimeout(100, TimeUnit.SECONDS);

}

@Then("I verify the login is successful")

public void i_verify_the_login_is_successful() {

assert(login.lnk_profile.isEnabled());

}

Junit Test Runner Class

package runner;

import io.cucumber.junit.Cucumber;

import io.cucumber.junit.CucumberOptions;

import org.junit.runner.RunWith;

@RunWith(Cucumber.class)

@CucumberOptions(plugin = {"pretty"},

features= "src\\test\\resources\\features" ,

glue= "stepDefinitions")

public class RunCucumberTest {

}

Right-click on TestRunner class and Click Run As > JUnit Test Application

Test Run Report will be shown like this

Eclipse Integration with Azure DevOps

Reference link: https://docs.microsoft.com/en-us/azure/devops/java/labs/eclipsegit/?view=azure-devops
If you face issues in accessing the Teams Explorer, follow the below workaround
- Credits: https://github.com/Microsoft/team-explorer-everywhere/issues/285#issuecomment-474944707
- Download jaxb-api-osgi jar from Maven Central (https://search.maven.org/search?q=g:javax.xml.bind%20AND%20a:jaxb-api-osgi&core=gav)
- Copy/move this jar to your eclipse/dropins directory
- Edit eclipse/plugins/com.microsoft.tfs.core_14.134.0.201804261732/META-INF/MANIFEST.MF
- append ",javax.xml.bind" to the "Import-Packages" line. It should look like this when complete:
  - Import-Package: org.eclipse.equinox.security.storage,javax.xml.bind
restart eclipse with -clean option so OSGi bundles are recomputed
Right-click on the project -> Team and you will have the rest of the git operations

Conclusion

I have created a sample Java Maven project using the Selenium Page factory and Cucumber in GitHub https://github.com/anu-01/CucumberSelenliumPageFactory. This is a great place to get started with a Cucumber-based framework.

Test Automation and EasyRepro: 03 - Extending the EasyRepro Framework

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:30:39 GMT

The following is the third on a series of articles by @Ali Youssefi that we will be cross-posting into this Test Community Blog. These articles were first published by Ali in the Dynamics community but since the topic is very related with Testing it makes sense to publish here as well.

If you didn't get a chance to catch the first one of the series, please have a look here:

Otherwise, please read ahead!

Summary

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. This article will focus on reviewing how EasyRepro works with the Document Object Model otherwise known as the DOM. This will help us understand how we can extend the EasyRepro code for use with custom objects or areas of the platform not included in the framework at this time. We will wrap with a brief look at XPath and referencing elements from the DOM.

Getting Started

If you haven't already, check out the first article which covers getting familiar with EasyRepro and working with the source code locally. It covers cloning from GitHub to Azure DevOps then from Azure DevOps Git repository to a development machine. and reviews setting up dependencies and test settings to run a simple test. The second article goes into common actions performed by users when interacting with Dynamics 365 and the correlating EasyRepro commands. It wraps with covering common troubleshooting and debugging scenarios.

The EasyRepro framework contains an abundance of commands that should suit most of your needs. However you'll find that at times either these commands will need to be modified per your customizations or you may need to create your own custom commands for reuse among tests.

Some example scenarios for modifying or creating custom commands may include:

Using an ADFS Redirect Login
Working with custom components and resources
Working with strongly typed entity attributes
Including navigation and functionality not present natively in the framework

These scenarios require an understanding of how EasyRepro interacts with Selenium to simulate user actions in a browser. This article will focus on a typical use case for extension: the Login process. We will describe common Selenium functionality and how to reference elements on the Document Object Model in a way to increase maintainability.

Sample Scenarios for Extending

Working with the Browser Driver

As we discussed earlier, your organization's login process may have a sign on page that a user will get redirected to once they input their username. This is where we begin our journey into extending the framework and utilizing Selenium.

Typical Single Sign On Page:

Custom method for a Single Sign On Page:

The image above shows a method designed to search through a sign in page DOM for input fields for credentials and submission. You will notice that we are working with the browser driver natively (args.Driver) and using methods not found in the sample EasyRepro unit tests. These include FindElement which is part of Selenium and WaitForPageToLoad which is part of EasyRepro's extension of Selenium. I'll touch on these at a high level now.

FindElement

FindElement is part of the Selenium.WebDriver assembly and is used to find an element in the DOM. I'll cover how to search through the DOM in the next section Extending and Working with XPath but I wanted to take this time to show how we will begin extending unit tests using Selenium natively.

The method returns an IWebElement object that represents the element you want to work with. For instance if you want to work with the username textbox you can use FindElement to locate and return the username textbox. Once returned the framework can interact with it and perform actions such as input text or click.

Reference:

https://www.toolsqa.com/selenium-webdriver/c-sharp/findelement-and-findelements-commands-in-c/

WaitForPageToLoad

WaitForPageToLoad is part of the EasyRepro framework as a Selenium extension. This is key to point out that we are leveraging both EasyRepro and Selenium natively to achieve our desired result to login. This method waits a specific amount of time and checks for the document ready state. The time interval used to check can be adjusted if needed.

SendKeys and Clear

SendKeys is used to send keystrokes to an element on the page. This can be a complete string such as an email or a single keypress. This can be used to send the username and password to your sign in page. It also can be used to send the Enter or Tab keypress to move to the next field or submit.

Clear as it sounds is used to remove any sort of input that may already exist in an element. This is useful if your sign in page attempts to automatically input credentials.

Both of these methods are part of the IWebElement shown above. Other useful properties of IWebElement include Text, Selected and GetAttribute.

Understanding how to extend element references

Considerations need to be made when designing unit tests to help reduce the maintenance work needed if something changes. For instance when referencing custom HTML web resources or even the ADFS Redirect from above, think how a change to an element in the DOM could propagate across some of all of your unit tests. One way to control maintenance is to centralize commonly used references into proxy objects that can hide the underlying mechanics from the test designer. This is exactly how the EasyRepro framework handles references and when extending can leverage the same approach. In this section we will cover the files used by the framework to reference DOM elements and show how to extend them to include references to our custom login page.

The App Element Reference File

The Microsoft.Dynamics365.UIAutomation.Api.UCI project uses the ElementReference.cs file as well as another file called AppElementReference.cs. The ElementReference file looks to have been brought over from the classic interface. What's unique about each is how they reference elements in the DOM which I'll cover in the next section. For now let's focus on reviewing the AppElementReference.cs file which is located in the DTO folder of the Microsoft.Dynamics365.UIAutomation.Api.UCI project.

Inside of the AppElementReference file are two objects used by EasyRepro to represent and locate elements: The AppReference and AppElements classes.

The AppReference class

The AppReference class includes sub classes that represent the objects used by EasyRepro, specifically the WebClient object, to connect to DOM elements. This allows the framework to standardize how the search for a particular container or element is performed. Centralizing the reference to the DOM elements will allow the test designer to focus on writing commands against common objects and not specifying the precise location of an element in the DOM.

The AppElement class

The AppElement class is a comma delimited key value pair consisting of the reference object property as a key and the XPath command as the value. The key represents the property name in the class object inside of AppReference while the value is the XPath location of the element in the DOM.

I highly suggest reviewing the AppElement class when extending EasyRepro as it shows recommended ways to locate and reference elements on the DOM. In the next section we will discuss the different ways you can locate elements including XPath.

Referencing elements in the Document Object Model

References to objects generally fall into four patterns:

Resolving via Control Name
Resolving via XPath
Resolving by Element ID
Resolving by CSS Class

In this article we will focus on XPath which is what is primarily used by the EasyRepro framework for the Dynamics 365 platform. However its key to understand each of the approaches for referencing as they can be used for customizations such as web resources, custom controls, etc.

Resolve with Control Name

This will search the DOM for elements with a specific name which is an attribute on the element node. This is not used by EasyRepro to my knowledge. Elements can be found by their name by using By.TagName with the FindElement method.

Resolve with Element ID

This will attempt to find an element by its unique identifier. For instance a textbox on the login form maybe identified as 'txtUserName'. Assuming this element ID is unique we could search for this particular element by an ID and return an IWebElement representation. An example from the Microsoft.Dynamics365.UIAutomation.Api.UCI project is shown below showing usage with the timeline control.

Definition:

Usage by WebClient:

Elements can be found by their ID by using By.Id with the FindElement method.

Resolve with CSS Class

This allows the ability to search by the CSS class defined on the element. Be aware that this can return multiple elements due to the nature of CSS class. There is no usage of this in EasyRepro but again this could be helpful for customizations. Elements can be found by their CSS class name by using By.Class with the FindElement method.

Resolve with XPath

XPath allows us to work quickly and efficiently to search for a known element or path within the DOM. Key functions include the contains method which allow to search node attributes or values. For instance when you review the DOM of a form you'll notice attributes such as data-id or static text housed within a span element. Coupling this attribute with the html tag can result in a surprisingly accurate way to locate an element. I suggest leveraging the current element class as well as this link from W3 Schools that goes into the schema of XPath.

Going back to an earlier section let's review how XPath along with the AppElementReference.cs file can help standardize element location.

Using XPath in the ADFS Redirect Login Method

Going back to our original example for the ADFS login method below you'll see an example of referencing the DOM elements using XPath directly with the Selenium objects driver and By.XPath. Consider the below two images:

Without a static representation of XPath:

Using static classes to represent XPath queries:

Both of these methods work and perform exactly the same. However the second method provides increased supportability if and when the login page goes through changes. For instance consider if the id of the textbox to input your username changes from txtUserName to txtLoginId. Also what if this txtUserName XPath query is propagated across hundreds or more unit tests?

Creating custom reference objects

Let's put what we have learned to use by creating a reference to our custom login page. Start by adding a class to the AppReference object and title it AdfsLogin. Inside this class declare string properties that will be used as input for your organization's login page. Typical inputs include username, password and a submit button. Here is an example:

NOTE: While this document demonstrates how to add to the AppElementReference.cs file I would suggest extending this outside of the core files as customizations will have to be merged with any updates from the framework.

Once the static class we want to use in our unit tests has been created we now need to add the XPath references to the AppElement class. Below is an image showing the key value pair discussed in the AppElement section above. The key correlates to the string value of the AdfsLogin class while the value is the XPath directive for our elements.

As shown in the image for Login_UserId we are searching the DOM for an input element with the id attribute of 'txtUserName'. XPath can be used to search for any attribute on the DOM element and can return a single value or multiple depending on what you're searching for.

Next Steps

Custom Components and Web Resources

PowerApps Control Framework controls and web resources are customizations that represent great options to learn and extend the EasyRepro framework. Try locating the container for the PCF controls and HTML web resources and work towards creating custom objects representing the elements for each as described above.

Conclusion

In this article we discussed reasons why we will may need to extend the EasyRepro framework and some techniques in doing so. We explored working with Selenium objects and creating references to help us when creating unit tests. Finally we put this in an example for working with a ADFS Redirect page on login.

Thank you again for reading! Please let me know in the comments how your journey is going!

Collection of Useful Tools for Performance Test Engineers

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:29:38 GMT

By Edwin Hernandez

We recently published an article with a collection of useful tools for QA/Test Engineers, highlighting tools that are sometimes overlooked or that you probably don't know about. If you get a chance please check that list too: Collections of Useful Tools for QA/Test Engineers.

On the same lines, this time I'm writing this article geared towards Performance Test Engineers. Whenever you work with JMeter, Visual Studio, LoadRunner or any other IDE, these tools help in the areas of scripting, debugging, performance analysis, etc.

Parts of this list has been circulating inside of the Microsoft Test Team for years. An earlier version of this list was originally put together by Geoff Gray (100-level wizard of Testing if you ask me). Taking that list as starting point, I moved things around, updated old links, replaced with newer versions were applicable and added other tools I find useful and have used throughout the years. I hope you find this useful:

TOOLS

PERFVIEW

Performance-analysis tool that helps isolate CPU- and memory-related performance issues
https://github.com/microsoft/perfview

PAL:

Performance Analysis of Logs (PAL) Tool. Reads in a performance monitor counter log and analyzes it using known thresholds.
https://github.com/clinthuffman/PAL

SYSINTERNALS

Manage, troubleshoot and diagnose your Windows systems and applications.
https://docs.microsoft.com/en-us/sysinternals/

Windows Driver Kit (WDK)

Develop, test, and deploy Windows drivers
https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk

WINDBG

Analyze crash/memory dumps.
New preview version:

https://www.microsoft.com/en-us/p/windbg-preview/9pgjgd53tn86?rtc=1&activetab=pivot:overviewtab

Psscor4 Managed-Code Debugging Extension for WINDBG

http://www.microsoft.com/en-us/download/details.aspx?id=21255

DEBUG DIAG

Debug Diagnostic Tool. Troubleshoot processes/threads, e.g. IIS.
Version 2.3:

https://www.microsoft.com/en-us/download/details.aspx?id=58210

CLR PROFILER

CLRProfiler is a free tool from Microsoft to help you diagnose memory issues with your managed app.
GitHub Repository:

https://github.com/microsoftarchive/clrprofiler/

FIDDLER

Http traffic debugger, uses a proxy to look at communication coming in and out of a host.
https://www.telerik.com/fiddler

WIRESHARK

Packet analyzer for network traffic debugging
https://www.wireshark.org/download.html

LOG PARSER 2.2

Command-line tools that provides query access to text-based data such as log files, XML files and CSV files
https://www.microsoft.com/en-us/download/details.aspx?id=24659

LINQPAD

Instantly test any LINQ,C#/F#/VB snippet or program
https://www.linqpad.net/

ROY OSHEROVE's REGEXP TOOLS

Regular Expression Syntax helper, visualizers, unit test tools, etc.
https://osherove.com/tools

SERVICE TRACE VIEWER

WCF Service Trace Viewer tool helps you correlate diagnostic traces produced by WCF listeners to locate the root cause of an error
https://docs.microsoft.com/en-us/dotnet/framework/wcf/diagnostics/tracing/using-service-trace-viewer-for-viewing-correlated-traces-and-troubleshooting?redirectedfrom=MSDN

Notepad++

Powerful text editor, features applying different encoding options, language formatting, macros, external plugins and lets you perform many string manipulation operations
https://notepad-plus-plus.org/downloads/

**There are many others text editors such as EditPad Pro, PSPad, Charny Notepad, etc. but wanted to feature at least one.

Beyond Compare

Powerful comparison tool that allows you to compare two files or folders, merge changes and visualize everything using different viewers.
https://www.scootersoftware.com/download.php

BareTail

Log File Monitoring Tool that lets you trace a log file in real time and view the tail of the file as information keeps coming in, without blocking the file for writing. Also lets you highlight filter information by color.
https://www.baremetalsoft.com/baretail/

Telerik Code Converter

Convert code from C# to VB and vice versa
https://converter.telerik.com/

WebTest Power-Editor (Serializer)

Power-Editor extension for Visual Studio WebTests (VS 2017). Allows you to modify multiple items inside a single Webtest or several WebTests and change values that are time-consuming to change one by one, such as Request properties, Transactions, etc.
https://marketplace.visualstudio.com/items?itemName=EdwinHernandezMSFT.SerializerVSIX

GUIDES

GEOFF GRAY's QUICK REFERENCE GUIDE

Visual Studio Performance Testing Quick Reference Guide (Version 3.6)
All you need to know about troubleshooting Visual Studio Web/Load Tests

https://archive.codeplex.com/?p=vsptqrg

Improving .NET Application Performance and Scalability

Best practices guide

http://www.microsoft.com/en-us/download/details.aspx?id=11711

Biztalk Server Performance Optimization Guide:

Latest available is for Biztalk Server 2013

https://docs.microsoft.com/en-us/biztalk/technical-guides/biztalk-server-2013-performance-optimization-guide

BLOGS

Some of these may be a bit dated, but the fundamentals are still very valuable:

NET Debugging for the Production Environment

http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment

Channel 9 Testing:

http://channel9.msdn.com/search?term=testing

Mark Russinovich's Blog

https://docs.microsoft.com/en-us/archive/blogs/markrussinovich/

Clint Huffman's Windows Performance Analysis Blog

https://docs.microsoft.com/en-us/archive/blogs/clinth/

"If broken it is, fix it you should" blog

https://docs.microsoft.com/en-us/archive/blogs/tess/

OTHER LISTS OF TOOLS

Scott Hanselman’s Ultimate tool list:

http://www.hanselman.com/blog/ScottHanselmans2011UltimateDeveloperAndPowerUsersToolListForWindows.aspx

SoftwareQaTest list of tools (very comprehensive):

http://www.softwareqatest.com/qatweb1.html?keepThis=true&TB_iframe=true&height=500&width=750

While this list includes mostly free tools, there are a few freeware or only trials. Also if you think I missed an important tool please let me know in the comments.

Test Automation and EasyRepro: 02 - Designing and Debugging Unit Tests

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:29:11 GMT

The following is the second on a series of articles by @Ali Youssefi that we will be cross-posting into this Test Community Blog. These articles were first published by Ali in the Dynamics community but since the topic is very related with Testing it makes sense to publish here as well.

If you didn't get a chance to catch the first one of the series, please have a look here: Test Automation and EasyRepro: 01 - Overview and Getting Started

Otherwise, please read ahead!

-----------------------------------------------------------------------------------------------------------------

Summary

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. This article will focus on designing and debugging unit tests. It will follow up on the first post's sample unit test in depth as well as provide design and troubleshooting ideas.

Getting Started

If you haven't already, you may want to clone the project that we used on the first article on this series (link above) from GitHub to DevOps then build locally and review any dependencies and/or test settings to run a simple test.

Reviewing the Open Account Sample Unit Test in depth

When working with the sample unit tests you'll soon find that while extremely helpful you'll need to modify or extend these tests to work with your customizations of the platform. The sample unit tests provide a great starting off point and can help us better understand how to work with the EasyRepro framework. Let's review the UCITestOpenActiveAccount unit test line by line.

WebClient and XrmApp objects

The WebClient is derived from the BrowserPage class and injected into the XrmApp object. The WebClient contains mainly internal methods used by the XrmApp and platform element references such as OnlineLogin, Navigation, Grid, etc as shown in the unit tests above. Typically you will not be interacting with this object until you need to extend the framework.

The XrmApp is the primary way of navigating and commanding the platform. When opened you'll see the WebClient passed into each object used in the unit tests. Review this object to better understand each of the area of the platform you can work with in your unit tests.

As you can see, each of the commands in the Open Account unit test are represented by elements related to Dynamics. These elements include commands related to the object such as working with entity forms and navigating around the platform. Each element can be explored to determine what functionality can be achieved from the framework natively.

Commands

Let's review each one of the commands in the Open Account unit test to better understand what they are doing and where we may need to modify to better meet our requirements.

Login

The Login method typically involves two paths: One using the standard Microsoft Office login and one that redirects to your organization's sign in page.

The first line shows passing in the URI of the organization, the username and password. How to set these values is covered in the first post in this series Test Automation and EasyRepro: 01 - Overview and Getting Started.

The second shows how to incorporate your organization's sign in page. Each sign in page is unique and will require understanding how to work with Selenium and the DOM of the page to input and submit these credentials. This will be covered in the Designing Tests towards Customization section.

Once the Login method has completed we will redirected to our default application in the organization. The next command details moving to another UCI application.

Open UCI App

The OpenApp method uses the UCIAppName class to navigate to a specific UCI app. The standard platform applications such as Sales Hub and Customer Service Hub, as well as others, are available in the class. I'll cover how to extend to a custom UCI app in the Designing Tests towards Customization section.

Open Sub Area

OpenSubArea introduces the first navigation into the sitemap of the Unified Interface. This method expects two parameters, the first identifying which area to open and the second which subarea to click. These string values are case sensitive.

Search

The Search method is used on the Grid object to search using the Quick Find view. Wildcard characters can be added here to simulate a user looking for values like or starting with specific characters if desired. For our unit test as long as we have the sample data in the organization we should pull up the Adventure Works (sample) record in the result set.

Open Record

The final navigation command in the unit test is to open the first record in the Grid using OpenRecord. As long as we pulled up the Adventure Works (sample) in our results this will open the first record based on the index of the rows in the view. If the index is outside of the bounds an exception will be thrown.

Think Time

Finally we have reached the end of the unit test. The final command here is a method called ThinkTime. This is used to simulate a user waiting for a period of time, in this case 3 seconds. This is a useful method to allow elements to render for use with the framework.

Designing Tests towards Customization

Adding custom UCI applications

In our sample unit test we are navigating to the Sales Hub Unified Interface but there maybe times where you will need to navigate to a specific custom application. A simple way to do so, assuming the UCI app shows in the user's drop down when logged in, is to add to the UCIAppName class.

In the example above a new string has been added to the class called 'PfeCustomApp'. The OpenApp method will search for an UCI app with called 'PFE Custom App'. This can then be used in single or many unit tests.

Searching and reviewing results in a Grid

In our sample unit test we knew what to expect due to sample data in our organization. However when implementing your own unit tests you may want perform additional tasks like counting the number of results are returned, sort the records in the view, switch the view, etc.

Above is an image from the Object Browser showcasing the native EasyRepro functionality for the Grid object. To Search use the Search method. For reviewing results look to the GetGridItems method to extract the grid items which we can iterate through.

Changing Forms and Referencing Fields

Going back to the UCITestOpenActiveAccount unit test as described above we searched for the Adventure Works (sample) record in clicked on the first record in the Quick Find view. We just touched on actions that can be performed on the view, now let's look at what can be done once we are inside the record.

When the Adventure Works (sample) record is opened we now can shift gears to working with the Entity object. The Entity object includes functionality such as but not limited to:

· Assigning the record to another user or team

· Working with field values

· Working with Sub Grids

· Switching Forms

· Switching Processes

For now let's stick with changing forms and working with field values.

Switching Forms

Start by navigating to the OpenRecord method and adding a new line. Here we will add SelectForm("<name of your form>"). This works with the form selector and allows us to change the form based on the name.

Working with Field Values

When working with most fields on the form the actions a user will perform are to clear out a value, read a value or update a value. In the below image I've added a line using the GetValue method. This returns me the current value which can be used for validation or other concerns. When referencing a field it is important to reference by its schema name and not display name. For instance on Account instead of using 'Account Name' I'm using 'name'.

To clear out a value you simply have to use ClearValue and pass in the schema name.

Finally to set a value you use the SetValue method passing in the schema name of the control (unless working with a complex type field).

Debugging Tests

When you begin to extend and debug the sample unit tests towards your customization you may run questions regarding how the test is running, who its running as, how to handle unforeseen issues, etc. For this section we will talk through some common scenarios that come up and how to address them.

My Unit Test is performing unexpected actions, what can I do?

One of the key benefits of browser automation is that we can watch the unit test in action, halt the execution, and examine the current state of the platform. One of the first steps in doing this is to run the unit test in what's called Non Headless Mode.

Running Unit Tests in Non Headless Mode

Headless Mode is what tells the unit test if we want to run the test with a GUI or not. Simply stated this will allow us to watch the browser launch and perform the actions or simply run in the background.

In the first post in this series we touched on the TestSettings object and how it tells our unit test how we want it to run. In this object is the class called BrowserOptions and within this is a property called Headless. To turn headless mode on or off, set this value to true (on) or false (off).

My Unit Test needs to run as a specific user, what can I do?

By default to run our EasyRepro unit tests (at least for the online platform) we must provide some credentials that determine who we run our tests as. This helps us identify and troubleshoot issues related to authentication and authorization. However sometimes due to your organization's setup you may have pass through authentication which will attempt to run the unit tests as the user you are logged into your machine with.

Running Unit Tests in Incognito Mode

Going back to the TestSettings object and inside of BrowserOptions we find a property called PrivateMode. By setting this property to true we can run in "Incognito" or "InPrivate" or whateevr terminology your browser of choice uses. This helps us ensure we are using the credentials we provided as well as not assuming any cached settings are applied.

My Unit Test needs to check form or grid values, what can I do?

If you have worked with the Visual Studio Unit Testing tools in the past you are probably familiar with the Assert class. This class allows a test designer to add assertions to ensure the actions performed are accurate. EasyRepro provides us the ability to check for form or grid values and elements which can then be used to with the Assert class.

Using the Assert class

Below is an example of checking if the number of results from our search is not equal to a single result. This could be helpful to determine if duplicates exist as well as other scenarios.

Reference:

microsoft.visualstudio.testtools.unittesting.assert

Next Steps

Exploring the Sample Unit Tests

At this point you should have enough experience to begin exploring the other sample unit tests available in the Microsoft.Dynamics365.UIAutomation.Sample project. These tests show how to interact with popular elements like Business Process Flows, Quick Create forms and Command Bar actions.

Each element has its own unique capabilities and should cover most of what you will need to automate testing in the platform. However you may come across a need to extend EasyRepro to account for specific use cases which we will cover in the next post in the series Extending and Working with XPath.

Conclusion

This article focuses on a specific unit tests and exploring line by line what each command does. While this covers a significant amount of interaction with the EasyRepro framework it really is only one test case. Take time to review as many of the sample unit tests as possible. Some test I find beneficial are:

· UCITestCreateCase

· SharedTestUploadTelemetry

· UCITestBusinessProcessFlowNextStage

· UCITestAccountAddNote

· UCITestRetrieveCommandBarValues

· UCITestUpdateClearFields

-----------------------------------------------------------------------------------------------------------------

We hope you guys are enjoying this series of articles on EasyRepro, please don't hesitate to leave any comments below!

How to Enable Undo & Redo in JMeter

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:28:37 GMT

By Raman Pandey

Hello everyone, this time we bring you another JMeter post for those performance testers following this blog. JMeter is a very extensible and customizable tool, however sometimes it may need a few tweaks to make it look and behave the way you want on your specific OS and setup.

For example you can configure the log, enable and configure HiDPI (High Dots per inch) scaling, change the size of fonts, icons, toolbars, among many other things. This time around though, Raman will explain how to enable and configure Undo and Redo in JMeter and why it is not enabled by default:

Undo & Redo in JMeter

Here is the solution for reverting back mistakes/changes while creating these massive JMeter scripts:

Undo and Redo are a very basic features of any editor or IDE. It is really annoying that such a basic feature is not found in JMeter. Here is the catch. JMeter has this feature but not by default and therefore we have to enable it from our end. JMeter team believes in making JMeter as light as possible and they are avoiding the extra memory consumption that would take place, therefore Undo and Redo options are kept hidden or disabled by default.

Initially, the JMeter toolbar without the Undo and Redo buttons looks like this:

Instructions:

Follow the below steps to enable these options:

Navigate to bin folder of JMeter directory
Open jmeter.properties file using Notepad or Notepad++
1. Side note here, you may prefer to make these changes in user.properties which would only apply to your user but the values on this file supersede values on the other properties file. Also if you mess up, you can always reset with the content of jmeter.properties.
Search for undo.history.size
Uncomment the line undo.history.size=0 by removing # (sharp sign)
Set a natural number as a value of undo.history.size attribute e.g. 1, 5, 10, 50 etc. (This number represents the maximum number of steps that you can undo or redo.)
Save the file and relaunch JMeter to reflect the changes.

Please note-

The number must be a natural number (1,2,3,….n) or simply say positive integer other than 0. The suggested value by JMeter is 25 which you can see in the description above this line in “jmeter.properties” file, it shows “Set it to a number > 0 (25 can be a good default)”. So, you can give 1 or 5 or 7 or 10 or 12 or 15 or 25, but not Zero. If you give Zero, there will not be any changes

Precaution:

This number represents the maximum number of steps that you can undo or redo. You can set this to a larger number e.g. 50 or 100 but the bigger it is, the more it consumes memory. When you try to undo/redo more than 5-10 steps at a time then you will notice a big slowdown in JMeter. Hence, JMeter team suggest to have a maximum value of 25 and that too for a computer with very high RAM memory.

For most of the normal users, 5 or 7 solve their purpose. Here, I have given the value as 5 which shown above.

Once you restart the JMeter you can see Undo and Redo options in the JMeter toolbar which are greyed-out; indicate no action done yet.

Now, perform some actions in JMeter so that both the buttons get colorfully visible and clickable, as illustrated below

Limitation:

Ctrl + Z or Command + Z (in Mac OS) does not work in JMeter. Only clicking the Undo/Redo button in the toolbar works.

You can also do Undo/Redo from “Edit” option of JMeter or “Context Menu” by right-clicking on any of the Element as illustrated below:

That is it. Thanks Raman for sharing!

Please leave any comments or questions below.

Collection of Useful Tools for QA/Test Engineers

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:27:55 GMT

Edited by Edwin Hernandez

Please find below a collection of publicly available tools that I find valuable for QA/Test Engineers, either for Performance Testing or UI Test automation. These tools have been passed around internally on the MSFT Test Team, some of them for years. Most of them cover a tiny niche and thus can be considered a bit buried or obscure, but if they are what you need, they can help you.

BUILD QUALITY CHECKS

(First reported by Bernhard Vogel)

Using Azure DevOps "Build Quality Checks“ tasks, you can easily control that the quality steadily goes up from build to build. There are various smart metrics to use, i.e. compare test coverage of the previous build with the current build and if it trends down > fail the build.

Policies:

The Build Quality Checks task currently supports two policies (click the link for details):

Warnings Policy - Allows you to fail builds based on the number of build warnings.
Code Coverage Policy - Allows you to fail builds based on the code coverage value of your tests.

Available here: https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks#overview

SCREEN TO GIF

(First reported by Johnathan Liu)

This tool allows you to record a selected area of your screen and save it as a Gif.
I have to share out this very cool tool called ScreenToGif that was shown to me by a couple of TAMs in the UK. It allows you to record your actions on screen and turn them into awesome animated gifs. I could see this being incredibly useful for tutorials, repros, and demos.
You can share animated gifs on Twitter/Yammer and other social networks.

Features:

Record your screen and save directly to a gif looped animation.

Pause and continue to record.
Move the window around to record what you want.
You can add Text, Subtitles, and Title Frames.
Edit the frames, add filters, revert, make yoyo style, change frame delay, add a border, add progress bars.
Export frames.
Crop and Resize.
You can work even while the program is recording.
Remove frames that you don't want.
Select a folder to save the file automatically or select one before encoding.

Add the system cursor to your recording.
Very small-sized, portable, and multi-language executable.
Start/Pause and stop your recording using your F keys.
Multi-language: Portuguese, Spanish, Romanian, Russian, Swedish, Greek, French, Simplified Chinese, Italian, Vietnamese, and Tamil.
GreenScreen unchanged pixels to save kilobytes.
You can apply actions/filters to selected frames.
Fullscreen Recording.
Snapshot Mode.
Drag and Drop to add frames in the editor.

Available here: https://www.microsoft.com/en-us/p/screentogif/9n3sqk8pds8g?activetab=pivot:overviewtab#

BOT SERVICE STRESS TOOLKIT

(First reported by Bernhard Vogel, a tool created by Daniel Amadei)

Bot Service Stress Toolkit is based on Apache JMeter (which is an open-source stress-testing product provided by the Apache Software Foundation).
Features

Leverages JMeter to make it easy and visual to perform stress tests on Bots.
Capable of not only making requests to the Bots but also receiving responses and correlating them back to the original requests, measuring the whole cycle.
Capable of asserting responses if needed, by leveraging native JMeter assertions.
Capable of loading test data from files or other data sources supported by JMeter.
Capable of displaying test results in graphs, tables, and/or saving it to CSV files by leveraging native JMeter capabilities as well.
Measures throughput and latency of your Bot Service applications taking consideration of the whole cycle from the request to when the response is received.
Able to stress test bots built-in Bot Builder SDK v3 or v4.
Able to stress test bots running locally or in Azure Bot Service.

You can find more info along with the documentation on how to install, create & run tests here: http://github.com/damadei/BotServiceStressToolkit.

WINDOWS 10 MODERN TERMINAL

(Reported by Edwin Hernandez)

New Modern Terminal was created by Microsoft as an open-source project. This Windows 10 app can have several types of shells as tabs on the same terminal window, even Unix shell.
The project is still on preview but looks great so far:

Give it a try:

https://www.microsoft.com/store/productId/9N0DX20HK701

AZURE DEVOPS DEMO GENERATOR

(Reported by Darren Rich)

Have you been looking for an example VisualStudio.com site that has:

Docker-based Java web application along with build & release definitions, to deploy the application to the Docker containers in the Azure App Service (Linux) using VSTS
SonarQube template contains code of MyShuttle, a sample application that will be provisioned and customized to analyze the code during the build and improve code quality.
Octopus template contains code for a sample PHP application which will be compiled using VSTS Build and deployed using Octopus Deploy on to an Azure App service.

Can you answer yes to any of the following questions?

I’m creating my own VSTS site to learn more about (Docker, AKS, GitHub CICD, Octopus, Selenium, Deployment Groups, WhiteSource-Bold)?
I’m using the PartsUnlimited project to do XXX?
Let me e-mail SmartPersonName and see if she can quickly create a demo VSTS site for the meeting?

If you answered yes then you should check out the Azure DevOps Demo Generator site and the corresponding docs page at https://docs.microsoft.com/en-us/vsts/demo-gen/use-vsts-demo-generator-v2?view=vsts

UNIX EPOCH CONVERTER

(Reported by Darren Rich)

It is very common to come across query string parameters that have a HUGE number assigned to them. You look around forever and do not find a solution.
One VERY COMMON source of these seemingly odd numbers is a value referred to as Unix Time. The next time you are trying to see if that number you are trying to parameterize is the Unix time you can find the current one at a very cool site.

Current Millis ‐ Milliseconds since Unix Epoch

Convert milliseconds to date - UNIX timestamp - UTC. Leap seconds: Leap seconds are one-second adjustments added to the UTC to synchronize it with solar time.

currentmillis.com

DEBUGGING TOOLS

(By Edwin Hernandez)

There are some tools used by developers and QA Engineers to do debugging of application, either on the front end or backend as well as the communications between them. These are not so obscure, these are known tools that you may already use, but just in case I wanted to include them on this list:

Wireshark. Packet analyzer for network traffic debugging

Fiddler. Http traffic debugger that uses a proxy to look at communication coming in and out of a host.

Performance Analysis of Logs (PAL) Tool. The PAL tool reads in a performance monitor counter log and analyzes it using complex, but known thresholds.

DebugDiag. Assists in troubleshooting issues such as hangs, slow performance, memory leaks or memory fragmentation, and crashes in any user-mode process.

PerfView. Performance-analysis tool that helps isolate CPU and memory-related performance issues. It is a Windows tool, but it also has some support for analyzing data collected on Linux machines. It works for a wide variety of scenarios but has several special features for investigating performance issues in code written for the .NET runtime.

And update on an old tool:

WinDbg Preview. Multipurpose debugger for Windows that can be used to debug kernel-mode and user-mode code, analyze crash dumps, and examine the CPU registers while the code executes. Microsoft just updated WinDbg to have more modern visuals, faster windows, a full-fledged scripting experience, and Time Travel Debugging. WinDbg Preview is using the same underlying engine as WinDbg, so all the commands, extensions, and workflows still work.

How to use the Windows Certificate Store in JMeter

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:27:33 GMT

By Chris Witte

The following article looks into an authentication problem that may appears when working with JMeter to create a HTTP script. JMeter is an Apache Software project mainly used for load testing (although it can also be used for other types of testing such as API tests).

Microsoft published that Visual Studio Load Tests will be deprecated. VS2019 is the last version that has this feature. They have also mentioned JMeter has one of the preferred alternatives to cover this area (more on this here). JMeter is arguably the most popular load testing tool in the market and it's a free, open source and extensible, check our previous post on this here. You should check out if you are interested in migrating or starting a load test project.

Now back to the problem:

Problem:

As you may know, authentication is almost always the most difficult part of performance scripting. The application in question here had an API gateway (Layer7) and in the process of migrating a Visual Studio Load Test to a JMeter Test Plan, we ran into issues providing a Client Certificate for authenticating with the API Gateway.

The flow is:

Client sends a request with the certificate,
A SAML token is returned,
Client sends credentials along with the SAML token, and receives a L7 Session ID.

After retrieving that L7 Session ID, the user can make requests to the API. JMeter is good at maintaining those session cookies, however providing the initial client certificate was an issue.

In JMeter you can specify a certificate store, so if you have a pk12 cert, this flow will not be an issue. In this case, the cert was in the Windows personal cert store, and the private key was marked as not exportable. Creating a Java cert store was out of the question.

So, how do we get JMeter to load and use the Windows certificate store?

It’s quite simple, however there were no resources online that I could find to show this solution. We are providing it here to hopefully save everyone else time.

Answer:

To get the windows certificate store available in JMeter launch with this command.

The important values here are keyStore = NONE and keyStoreType = Windows-MY.

jmeter -Djavax.net.ssl.keyStore=NONE -Djavax.net.ssl.keyStorePassword="" -Djavax.net.ssl.keyStoreType="Windows-MY" --username "" --password  “”

Then in JMeter you need to specify which cert to use:

Add a Keystore Configuration Element

Add a variable that will store the alias to the keystore configuration

Add the variable to the test. You can use a csv, or Redis data source. In our case we are using User Defined Variables. My cert’s alias is “chriswitte”

Now when you run your scripts it will add the client cert to your requests

I hope this does save time for someone else that may have this problem. If you have any comments or feedback feel free to leave it below. Thanks and until next time!

Test Automation and EasyRepro: 01 - Overview and Getting Started

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:26:37 GMT

The following is the first on a series of articles by @Ali Youssefi that we will be cross-posting into this Test Community Blog. These articles were first published by Ali in the Dynamics community but since the topic is very related with Testing it makes sense to publish here as well.

We will be interrelating cross-posting content with original articles and when it's a series. Stay tuned!

-----------------------------------------------------------------------------------------------------------------

Summary

EasyRepro is a framework that allow automated UI tests to be performed on a specific Dynamics 365 organization. You can use it to automate testing such as Smoke, Regression, Load, etc. The framework is built from the Open Source Selenium web drivers used by the industry across a wide range of projects and applications. The entire EasyRepro framework is Open Source and available on GitHub. The purpose of this article is to walk through the setup of the EasyRepro framework. It assumes you are familiar with concepts such as working with Unit Tests in Visual Studio, downloading NuGet packages and cloning repositories from GitHub.

Getting Started

Now that you have a basic understanding of what EasyRepro is useful for you probably would like to start working with it. Getting EasyRepro up and running is very simple as the framework is designed with flexibility and agility in mind. However, like any other utility there is some initial learning and few hurdles to get over to begin working with EasyRepro. Let's start with dependencies!

Dependencies

The first dependency involves the EasyRepro assemblies and the Selenium framework. The second involve .NET, specifically the .NET framework (.NET core can be used and is included as a feature branch!). Finally depending on how you are working with the framework you will want to include a testing framework to design, build and run your unit tests.

Choosing How to Consume the EasyRepro Framework

There are two ways of consuming the EasyRepro framework, one is using the NuGet packages directly while the other is to clone or download from the GitHub repository. The decision to use one over the other primarily depends on your need to explore or extend the framework and how you go about doing so. Working directly with the source code allows exploration into how EasyRepro interacts with Dynamics 365. However for extending the framework the approach of using the NuGet packages and building on top allows for increased flexibility.

Downloading using NuGet Package Manager

The quickest way to get started with the EasyRepro framework is to simply add a NuGet package reference to your unit test project. You can do by running this command in the NuGet Package Manager command line:

Create your unit test project and navigate to the NuGet Package Manager CLI. Use the Install-Package command to get the PowerApps.UIAutomation.Api package as show in the command below (v9.0.2 is the latest as of this writing please refer to this link for any updates:

Install-Package PowerApps.UIAutomation.Api -Version 9.0.2

This will get you the references needed to begin working with the framework immediately. Once installed you should the following packages begin to download into your unit test project:

When complete the required assemblies are available and you can begin working with the EasyRepro framework. There are some settings needed for the framework to connect to your Dynamics 365 organization which if you're new to the framework maybe unknown. If so I would suggest reviewing the next section which initiates a clone of the EasyRepro framework which happens to include a robust amount of sample unit tests that show how to interact with the framework.

Cloning from GitHub

If you're new to the framework in my opinion this is the best way to begin familiarizing yourself how it works and how to build a wide range of unit tests. This is also the way to go if you want to understand how EasyRepro is built upon the Selenium framework and how to extend.

To begin go to the official EasyRepro project located at https://github.com/Microsoft/EasyRepro. Once you're there take a moment to review the branches available. The branches are structured in a GitFlow approach so if you're wanting to work with the latest in market release of Dynamics 365 review the releases/* branches. For the latest on going development I would suggest the develop branch.

Start by cloning the project locally to review the contents and see how the interaction between the frameworks occurs.

The gif below shows cloning to Azure DevOps but cloning locally directly from GitHub is also supported.

Cloning locally from Azure DevOps

Another alternative which I highly recommend is to clone to an Azure DevOps project which can then be cloned locally. This will allow us to automate with CI/CD which we will cover in another article. If you decided to clone to Azure DevOps from GitHub the next step is to clone locally.

The gif below shows cloning locally from an Azure DevOps repository.

Reviewing the EasyRepro Source Code Projects

The EasyRepro source code includes a Visual Studio solution with three class library projects and one for sample unit tests.

The projects used by the Unified Interface are Microsoft.Dynamics365.UIAutomation.Api.UCI and Microsoft.Dynamics365.UIAutomation.Api.Browser. Most of the usage between EasyRepro and unit tests will happen with objects and commands within the Microsoft.Dynamics365.UIAutomation.Api.UCI project. This project contains objects to interact with Dynamics Unified Interface modules and forms.

The Microsoft.Dynamics365.UIAutomation.Api.Browser project is limited to interacts with the browser driver and other under the hood components.

Reviewing Sample Unit Tests

Looking into the Open Account Sample Unit Test

The unit test project Microsoft.Dynamics365.UIAutomation.Sample contains hundreds of unit tests which can serve as a great learning tool to better understand how to work with the EasyRepro framework. I highly suggest exploring these tests when you begin to utilize the framework within your test strategy. Many general and specific tasks are essentially laid out and can be transformed to your needs. Examples include opening forms (OpenRecord), navigating (OpenSubArea) and searching for records (Search), creating and updating records (Save).

For this exercise we will open up the UCITestOpenActiveAccount unit test, you can find this using Find within Visual Studio (Ctrl+F). Once found you should see something like the following:

Following the steps within the unit test you can see its designed to perform basic user actions to read an account. We start by logging into an organization (Login). Then we proceed to open the UCI application titled "Sales" (OpenApp). Once in the organization we open the Accounts sub area (OpenSubArea) and search for "Adventure" in the Quick Find View (Search). Finally we open the first record (OpenRecord(0)) in the quick find view results.

Exploring Test Settings

In the current sample Unit Test project the test settings are set in two places: the app.config file located in the root of the project and in the TestSettings.cs file, a class object used across all of the tests.

Application Configuration file

The app.config file includes string configurations that tell the tests what organization to login to, who to login as and other under the hood settings like which browser to run and how to run the tests.

Application Configuration File Settings

Property	Description
OnlineUsername	String. Used to represent the test user name.
OnlinePassword	String. Used to represent the test user password.
OnlineCrmUrl	String. Used to represent the organization (i.e. https://<your org>.crm.dynamics.com/main.aspx)
AzureKey	String. GUID representation of Azure Application Insights Instrumentation Key.
BrowserType	String. Represents enum flag for Microsoft.Dynamics365.UIAutomation.Browser.BrowserType.
RemoteBrowserType	String. Represents enum flag for Microsoft.Dynamics365.UIAutomation.Browser.BrowserType. Only used if BrowserType is Remote.
RemoteHubServer	String. Represents Selenium Server remote hub URL. Only used if BrowserType is Remote.

For this article we will focus on simply running locally with the Google Chrome browser by setting the BrowserType to "Chrome". Also inside of the app.config file are three settings we need to modify called OnlineUsername, OnlinePassword and OnlineCrmUrl. In my case I am using a trial and as you can see below I am using a "user@tenant.onmicrosoft.com" username and a "https://<orgname>.crm.dynamics.com/main.aspx" URL.

Before:

After:

Test Settings and the BrowserOptions object

Another key object is the TestSettings class and the various properties inside. This class tells the unit tests how to render the browser, where the browser driver can be located as well as other properties. The TestSettings class will need to be included in the Unit Test project and instantiate the BrowserOptions object as shown below:

In the next post we will explore how these settings can change your experience working with unit tests and what options are available.

Next Steps

Conclusion

From this article you should be able to begin using EasyRepro with your Dynamics 365 organization immediately. The next articles will go into designing and debugging unit tests, extending the EasyRepro code, implementing with Azure DevOps and other topics.

Advanced Logger for UI Test Automation using Azure DevOps

Microsoft_Testing_Team — Thu, 11 Feb 2021 19:26:09 GMT

by Edwin Hernandez

In this article, I will describe the implementation of a logger for UI Test automation and the integration with test execution from an Azure DevOps Release pipeline. This article and the demo solution are standalone articles; however, I do build on concepts we reviewed on previous articles, specifically:

You may want to read through those articles before starting with this one. Having said that, let us get started describing what the logger does.

Purpose

UI Test automation and Regression are time-consuming expensive tasks because of their initial time investment and their maintenance effort. Even if a test suite is well maintained, once Automated Regression is executed, you still need a QA Automation Engineer to analyze the results and investigate any errors.

The purpose of this logger is to provide a detailed log of the flow of the UI script, including test case steps and screenshots. The QA Engineer or a Business Analyst can use the information from the logger to follow the business logic of the UI test and more easily validate if the intended work was completed or the reason for failure if an error occurred. A descriptive log can be more useful than even an actual video recording of the test.

Logger Features

The logger creates a single HTML file that contains all messages and embedded images.
This class is meant to be used in a Page Object Model project, when the LogMessage method is called it reflects itself to get the calling method and class and it uses this information to write a message into the HTML log.
There are 4 log levels to help filtering how much information you want to log (error, warning, passed, info). This and other configurations are exposed as global variables. You can also specify the level up to which you want screenshots to be taken.
Screenshots are saved as binary strings and embed into the HTML file so that you end up with a single file. This greatly helps the integration with Azure DevOps.
This logger is adaptable since it is a standalone class. You can plug it into several types of UI Test projects. I have personally used it with WinAppDriver, CodedUI, Selenium WebDriver, and Xamarin.UITests projects with minimal changes, pretty much all that needs to be changed is the type of session object that is passed as an argument and the screenshot method.

Logger Requirements

Given the frameworks I just mentioned, you can guess that this Logger class was written in C# and is meant to be used on Visual Studio for UI Test projects running over MSTest or NUnit projects. You could adapt it to other types of frameworks, but it would require more work.
As I said this logger is meant to be used as part of a Page Object Model (POM), it is not that you couldn't use it with a straight Top to Bottom test but you would take more advantage of it in a POM project.

Other than that, the log method needs to be specifically called every time something needs to be logged. I explored the option of using the constructor or interfaces, but it would require the class to be more coupled with the test solution and I wanted it to be more portable.

Configuration Variables

The logger configuration variables should be put for example on the Class Initialize method:

A log level of 4 would log all messages, a log level of 3 would log only messages level 1 to 3. You can change the labels of the labels in the HTMLLogFile class itself.

Logging Messages from the Test Method

As I mentioned earlier, the LogMessage method needs to be called every time something is to be logged into the file. I recommend that only "Info" level messages should be logged from the Test Method, mostly to log comments and business step details that would make the HTML report easier to read, for example:

The InsertDividerForNewTest creates a header for the report, then every "Info" step is logged with or without screenshots depending on the configured level.

Logging Messages from the Page Class Methods

This may be the most interesting part, every time you use LogMessage from the Page Class Methods, the HTMLLogFile class will use reflection to get the calling method/class information and it will include this information into the HTML report, take the following for example:

By including LogMessage "Passed" at the end, the HTML log will print the following:

Thus, by having well-constructed Page Object Classes, the log would build itself with very few additional comments.

Finding the Log for Local Test Executions

To have this HTML file to be uploaded to Azure DevOps it must be part of the Test Context. This part is included in this demo in the Class Cleanup Method:

If the UI test is executed from Visual Studio (make sure you have WinAppDriver installed, please check my other post about this on the top). The Test Detail Summary should show additional output already part of the Test Context, and inside of that you can see the HTML file:

If you open that, the HTML log should have a header, then every Log Message. Please note that screenshots are expandable on hover.

Integration with Azure DevOps Release Pipeline

Now for the next part, I will only show what the result would look like. Setting up a Release Pipeline in Azure DevOps for UI Test Regression is not in the scope of this article. I intend to cover that in a future article, there are several ways to accomplish that depending on what you need to do.

For now, the screenshot below is a simple Release Pipeline that takes the Test Assemblies as an artifact, executes the Tests on a Private (Self-Hosted) Build Agent set on interactive mode.

MSTest automatically uploads to the Release Pipeline test results information (# of test passed, pass %, duration, etc.) and if you expand a specific test and navigate to the Attachments panel, the HTML Log file should be there already associated with that test. Every test will get a unique HTML file:

The benefits of running Regression from a Release Pipeline are related to traceability against a Build/Environment and even user stories and test cases. For deployment approval and for historical purposes since tests are saved into the cloud instead of local files.

Download Demo Project and HTMLLogFIle class

You can find at the bottom of this post, a file containing a Visual Studio solution that has a demo POM library project, a test project, and the HTMLLogFIle class, as well as a demo HTML report file as a reference.

If you have any comments/concerns, please reach out or comment below.