First published on MSDN on Aug 29, 2018
Authored by Edwin Hernandez
Are you a Software Test Professional? If so, what test tool do you use? As most of people when they start in QA, you may have to become proficient with only one or two test tools because you are restricted to what is being used in your company, or what was best to test a specific application or perhaps because it aligned better with your teams' ALM tool stack.
Whichever is the reason, if you are an experienced professional you may benefit from having a bigger picture of what is out there on the Enterprise-level software test tool market. If you are a QA automation newcomer the goal of this guide would be to provide guidance in finding a good tool that matches your testing needs.
I will list the most common and popular test tools, what features makes them popular and their limitations.
The list of below shows the most common UI Automation tools used for Functional Testing Automation and Regression testing (no particular order), their descriptions, features, pros and cos:
UFT/QTP - (official site here )
Quick Test Professional (QTP) now called United Functional Testing (UFT) was originally created by Mercury Interactive in the 90s and it became the de facto industry standard back then. Mercury was acquired by HP in 2006 and later HP sold its software division to Micro Focus in 2016. Therefore it's called Micro-Focus UFT now and is still one of the most used tools. However it has lost its leader position to Selenium in the last years. UFT uses VBScript as it's scripting language, it supports Web and Desktop and has protocols that ease scripting of some CRM/ERP applications like SAP and Oracle.
Selenium has been around since 2004 but I cannot find it as being a challenger to UFT until 2008-2010. It has progressively become the industry standard since then because of its flexibility to be used on several third-party IDEs and using some of the most popular modern programing languages. It supports most Web Browsers and OS but it cannot be used for Desktop testing.
Selenium WebDriver is the main component of this offering but there is also Selenium IDE which helps record and playback scripts in a declarative easier to use view, and Selenium Remote Control and Grid that allows for remote execution of multiple tests in parallel.
Since Selenium is open-source, several other projects have spun-off for purposes not originally covered by plain Selenium. Below is a list of some of the most popular, these projects often require higher skills, longer initial setup and more programming knowledge than plain Selenium, but these are the reason behind Selenium becoming so popular:
Keep in mind:
Protector is an evolving community project, most documentation out there is obsolete, so you have to figure out issues by yourself most of the time.
Appium is a Selenium based UI Automation tool for Mobile apps, either native or hybrid on iOS and Android. This is a fairly developed project that has its own IDE that includes an Emulator, UI Element Inspector, etc., you would still need to write and edit your scripts on your own IDE (VS, Eclipse, etc.). It supports Selenium Grid, so parallel test execution on remote hosts is possible.
Keep in mind:
Mobile UI automation has a more complex initial setup, you need to be able to install and configure the mobile OS SDKs, download and configure Device Emulators, have a developer account with iOS, be able to provision the code, etc. This is true for all Mobile testing not only for Appium (e.g. Xamarin UI Test have the same initial hurdles).
Winium also based on Selenium is an UI automation framework designed to test Desktop applications on Windows. Just like selenium, it is open-source, community supported (which has its own pros and cons as stated above). It supports WinForms, WPF, Universal Apps (Windows Store) and even Silverlight apps. One core limitation is that it only support working on top of Windows 10.
Keep in mind:
Winium is a recent project and it's not as mature as other Desktop UI automation offers, people complain about struggling with the API and having to resort to do a hybrid solution (e.g. mixing it with CodedUI).
There are many other projects based on Selenium that are popular, like Selendroid and Robotium.
By Microsoft, CodedUI is a powerful test framework that is superior in some aspects to other tools. For example it supports a wide range of technologies such as several IE Browsers, Desktop apps, SOAP, Windows Phone apps, etc. It also does integration with ALM very well: it integrates with VS Team Services for bug management, Agile planning, Continuous testing, etc. It supports remote execution and running multiple tests in parallel but the setup can be a bit more lengthy than for Selenium.
Even though it's easy to do get started with, It is notably difficult to maintain for larger projects, especially if using UIMaps which are auto-built UI control repositories. Also it has been reported that Microsoft won't release new versions of CodedUI (existing version will still be supported)
Tip: UIA is an accessibility framework for development purposes. It is a component of the larger CodedUI framework, but for legacy desktop applications it is sometimes easier to go directly to UIA and use the appropriate accessibility pattern to drive actions for UI controls.
TestComplete is proprietary a UI Automation tool by SmartBear, it has been around since early 2000s, it is has pricy has UFT but it became increasingly popular when compared to it because it advertises that it doesn't need programming skills (although just like UFT you do need programming skills for advance scripts) and because it was a wider range of supported programing languages and technologies.
This is a relative newcomer, it has been around since 2015. It has a more reduced range of technologies it supports, but it's free and easy to use without programming skills for simple test cases.
That is where I'm going to stop, if I describe every UI tool out there this blog post will never end. Just know that there are other tools that are very popular:
I'm going to mention one more alternative on the Mobile UI Test automation space because I believe that it has some features for that are pretty good compared with the others. I was not able to find a market research that had Xamarin as one of the offerings for UI Automation, either because Xamarin is relatively new or because it was rebranded when it was acquired by Microsoft. I really hope this changes in the future. Here are the reasons why I like Xamarin and Xamarin Tests Studio:
After a quick research, I found that most of the market share information for UI automation tools is a bit dated. And even when it is not dated, the business companies in charge of the research often mix Unit test tools and Performance Test tools with UI automation. Based on the more recent sources (from 2016-7), it seems that the following conclusions can be drawn:
When selecting a UI Automation test tool:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.