Configuration testing with VSTS

Testing a system in development on numerous machines with various combinations or configurations of hardware and software is known as configuration testing. Each supported hardware and software configuration is evaluated against the system’s or an application’s performance.

When we refer to various hardware and software configurations, it addresses various operating system versions, browsers, supporting drivers, memory capacities, hard drive configurations, CPUs, etc.

Why This Testing?

This testing aims to identify the appropriate configurations for a system or application so that they can operate without bugs, issues, or flaws in performance. Thus, with the aid of this testing, the configuration that would give the desired performance characteristics is found to be the most effective.

Verifying the system’s compatibility with other software or equipment included in the SRS (software requirement specification) is the second main objective of this testing.

Assume that the following platforms are supported by each layer:

Now, as a software tester, you must test the application using all of the different configurations of the mentioned client, server, and database platforms to assure that it functions properly in all scenarios.

We won’t limit our testing to only the software; it will also include the hardware, where we’ll need to examine all possible configurations of different hardware components. Consequently, this testing is occasionally also referred to as hardware configuration testing.

Prerequisites to execute Configuration Test

1) Preparing the Coverage Matrix

Because there are so many different hardware and software combinations that may be used, testing each one thoroughly takes a lot of effort. So, it becomes important to do the planning for the software testing effort and identify clearly which platforms will be supported.

We must create a coverage matrix that will contain all possible hardware and software configuration combinations. This coverage matrix is also sometimes referred to as BCM (Basic Configuration Matrix).

2) Configuration Prioritising

The next step is to organise the configurations after the configuration matrix has been created.

Since testing a comprehensive array of setups is not practicable, this stage is necessary. The most important configurations are therefore included and are to be extensively evaluated first, based on the client’s comments.

Once the first two phases are complete, we may test the different setups according to their priority.

Microsoft VSTS – Configuration Testing Tool

A tool called Microsoft Visual Studio Team Services (VSTS) is very helpful in testing your app in a variety of configurations according to your test plan.

To select which tests to run and on which setups, you need to establish a test plan. Make sure your environment is set up correctly for the settings you need. Once you have the combo matrix, you must put it to the test.

Steps to perform the testing:

Creating configurations and variables

Combinations of configuration variable values make up a test configuration. Examples of configuration variables include an operating system, browser, CPU type, and database. Windows 8 + 32-bit CPU or Windows 10 + 64-bit CPU are examples of configurations.

1. Firstly, Create the configuration variables and then create a configuration combining multiple variable values.
2. Open the Configurations page, click on the + icon, and click on the New configuration variable.

3. Give a name for the variable and a description. Click on Add new value, and then add a value and save it. You can add as many values as you want.

4.  Create other configuration variables by repeating the above steps.

5. Click on the + icon and choose New test configuration.

6. Give a name for the test configuration and add the variables which you have created For this configuration, select a value for each variable.

7. Now Save the new test configuration.

Assigning the configurations to the test plans/test suites

1. Select a plan to assign a configuration to a test suite, on the Test plans page. select a test suite in the Test Suites area, Choose More options to open the context menu, and click on Assign configuration.

The tests cases will get repeated with each of the configurations you have assigned, if you add multiple configurations to a test suite.

2.  Choose one or more test cases, open the context menu, and select Assign configuration. If required, override the default configuration assigned to a test case and assign the configuration as per your need.

3. Choose the configurations to assign to the test cases.

Run tests with each configuration

This phase is to set up a testing environment for a specific configuration, such as testing the app on Windows 10 with Google Chrome.

1. Click and run a test that has this configuration assigned.

A reminder of the setup shows in the status bar when the test is being executed.

Track test results for each configuration

1.  Click New, then select the new test result chart on the Charts page for your test suite.

2.  Select a chart type, such as a bar chart or pie chart, and then click Group by > Configuration and click OK.

3. Based on configurations, a chart is produced to assist you in tracking your testing. This graph may be pinned to your dashboard.

 Importance of this Testing for an Enterprise

For a company to start and maintain high-performance, scalable, and available applications, this testing is essential.

To quickly detect and address the elements that lead to inefficiencies, this testing should be carried out continuously and preferably in an automated manner (manually is highly time-consuming).

Conclusion

As important as any other sort of software testing, configuration testing identifies the optimal system configuration for the application to operate in. Additionally, it aids in locating and fixing any compatibility problems.