Qt Test Framework for Visual Studio and the Qt VS Tools

The Qt VS Test extension provides an entry point for testing toolkits (e.g. Qt Test) to run automated tests in the context of the Visual Studio IDE. The extension receives snippets of C# code, i.e. macros, from client apps, and then compiles and runs those macros. When completed (successfully or with errors), a macro can return data back to the client app, which can be compared to an expected result and yield a test status.

Contents

Pre-requisites

To use the Qt VS Test extension, the following requirements must first be observed:

  • Install Visual Studio 2019 or 2017 (any edition)
  • Install Qt VS Test extension
  • Install Qt VS Tools (if testing this extension)
  • After installation is complete, open the VS IDE, to allow extension initialization to occur.

Visual Studio auto-testing with Qt Test

Though QtVSTest is a generic tool that can be used to test any functionality within the VS IDE, and can in principle integrate with any automated testing framework, it was designed to provide QtTest applications the ability to run auto-tests in Visual Studio, specifically targeting the Qt Visual Studio Tools extension. This section shows how to setup a Qt Test application to run automated tests inside the Visual Studio IDE by connecting to the QtVSTest extension.

New Qt Test project in Qt Creator

To create a new Qt Test project in Qt Creator:

  • New Project
  • Other Project > Auto Test Project
  • Choose location
  • Details
    • Test framework: Qt Test
    • Build system: qmake
    • Choose test case name, e.g. 'foo'
    • Enable generation of initialization and cleanup code
  • Select relevant kit(s)

After the project is created:

  • Add network module to QT
  • Add $$(LOCALAPPDATA)/qtvstest to INCLUDEPATH

New Qt Test project in Visual Studio

Start Visual Studio from a development command prompt:

  • E.g., if using Visual Studio 2019, run "Developer Command Prompt for VS 2019"
  • At the command prompt, run devenv

Create a new Qt project:

  • File > New > Project
  • Qt Console Application
  • Set name and location
  • In the configurations page of the wizard, add the Network and Test modules.

After the project is created:

  • Open project properties (right-click > Properties)
  • Select "All Configurations"
  • Edit VC++ Directories > Include Directories:
    • Add $(LOCALAPPDATA)\qtvstest
  • In main.cpp: include Qt Test
    #include <QtTest>
    
  • In main.cpp: add the following class:
    class FooTest : public QObject
    {
        Q_OBJECT
    public:
        FooTest()
        {}
        ~FooTest()
        {}
    private slots:
        void initTestCase()
        {}
        void fooTestCase()
        {}
        void cleanupTestCase()
        {}
    };
    
  • In main.cpp: replace main() with:
    QTEST_APPLESS_MAIN(FooTest)
    
  • In main.cpp: add the following at the end
    #include "main.moc"
    

Connecting to Qt VS Test

  • Include MacroClient.h
    • This header contains helper functions for connecting to the Qt VS Test, sending macro code and receiving macro execution results.
  • Add a MacroClient variable to the test class
    private:
        MacroClient client;
    
  • Connect to Qt VS Test in the initTestCase() slot:
    • Calling client.connect() without a parameter will first start a new instance of the VS IDE and then connect to the Qt VS Test extension in that instance.
      1
      2
      3
      4
      void initTestCase()
      {
          QVERIFY(client.connect());
      }
      
  • Disconnect from Qt VS Test in the cleanupTestCase() slot:
    • Calling client.disconnect() with a true parameter will terminate the VS process after disconnecting.
      1
      2
      3
      4
      void cleanupTestCase()
      {
          client.disconnect(true);
      }
      

After the above steps (in either Qt Creator or Visual Studio), we should have a bare-bones Qt Test application that successfully starts Visual Studio, connects to the Qt VS Test extension, and then disconnects and closes the VS IDE.

Running macros in VS

Macros consist of C# code, together with specialized macro statements that are expanded by the Qt VS Test extension. Macro statements are lines of code starting with //# and ending at the next line break (CR+LF). These lines are interpreted by the extension and corresponding code is generated inline with the C# statements. The generated code is then compiled and the resulting function is executed. The result of running the macro function is stored in a global string variable Result and sent back to the test client when the macro execution terminates.

Simple macro

The following test case opens a message box in Visual Studio.

1
2
3
4
5
6
void fooTestCase()
{
    client.runMacro(QString()
        % "//# using System.Windows.Forms\r\n"
        % "MessageBox.Show(\"Hello from Visual Studio!!\");");
}

The //# using macro statement includes the System.Windows.Forms namespace. The macro execution and the client test app will block until the message box is closed.

Verifying macro result

The following test case opens a message box and waits 15 seconds for the user to close it. If the message box is not closed within that time the test case fails.

1
2
3
4
5
6
7
8
void fooTestCase()
{
    QCOMPARE(client.runMacro(QString()
        % "//# using System.Windows.Forms\r\n"
        % "var task = Task.Run(() => MessageBox.Show(\"Hello, close this in 15 secs!!\"));\r\n"
        % "//# wait 15000 => task.IsCompleted"),
        MACRO_OK);
}

The message box is opened on a separate thread and the //# wait macro statement is used to ensure it is closed within 15 seconds. If it is closed, the wait statement is successful and the macro returns MACRO_OK. Otherwise, a timeout exception is generated at the //# wait statement. The macro return value will be the exception error message, and the test case will fail.

It is also possible to explicitly set the return value of a macro, by setting the Result string variable.

1
2
3
QCOMPARE(client.runMacro(
    "Result = Environment.CurrentDirectory;",
    QDir::currentPath()));

Using macro files

MacroClient::runMacro() will also accept a QFile as input. The contents of the file are read and sent over to QtVSTest as the macro code to be executed.

QCOMPARE(client.runMacro(QFile(":/RebuildSolution"), MACRO_OK);

Invoking other macros

The //# call statement invokes the execution of named macros stored by QtVSTest. The MacroClient::storeMacro() function can be used to store a named macro. QtVSTest will process the macro code but, instead of executing it immediately, will store it under the provided name.

1
2
3
4
client.storeMacro("Hello", QString()
    % "//# using System.Windows.Forms\r\n"
    % "MessageBox.Show(\"Hello from Visual Studio!!\");");
QCOMPARE(client.runMacro("//# call Hello"), MACRO_OK);

Macro variables

Macros can share data by means of global variables. Macro variables are declared with the //# var statement.

1
2
3
4
5
client.runMacro("//# var InitTime => DateTime.Now");
QCOMPARE(client.runMacro(QString()
    % "//# var InitTime\r\n"
    % "MessageBox.Show(\"Test started at \" + InitTime);"),
    MACRO_OK);

Visual Studio SDK services

Macros can invoke services of the VS SDK to access Visual Studio IDE features. The //# service statement is used to obtain a reference to a VS SDK service. The following macro accesses the SVsShell service through the IVsShell interface to check if Visual Studio is running with admin rights.

1
2
3
4
5
//# using Microsoft.VisualStudio.Shell.Interop
//# service vsShell IVsShell SVsShell
bool isAdmin;
if (vsShell.IsRunningElevated(out isAdmin) != VSConstants.S_OK || !isAdmin)
    Result = MACRO_ERROR;

DTE service

Macros include in their run-time context a reference to the DTE service. This is the top-level object in the Visual Studio automation object model. The following macro uses this service to check if a solution is loaded.

if (!Dte.Solution.IsOpen)
    Result = MACRO_ERROR;

Interacting with UI elements

Macros can use UI Automation interfaces to interact with UI elements. The run-time context of a macro includes the currently selected UI element. This is an AutomationElement object in the system-wide UI automation tree (rooted on the Windows desktop). The current UI context can be accessed in macro code through the UiContext variable. At the start of a macro run, the default UI context is the root Visual Studio UI element.

NB: the tool inspect.exe, part of the Windows SDK, provides a useful graphical view of the tree of UI automation elements, and can be helpful to test the interaction with those elements.

Locating UI Automation elements

The //# ui context statement can be used to search for a descendant of the currently selected element, and set it as the new UI context. To locate a descendant element, a search path must be specified, consisting of a sequence of names of descendant nodes. The following macro selects the "Open File" button on the VS toolbar, and sets input focus to that button. The VS root UI element is assumed to be currently selected.

//# ui context => "ToolBarDockTop", "Standard", "Open File..."
UiContext.SetFocus();

By specifying VSROOT to the statement, the root UI element of Visual Studio will be used as the base for search paths. The following macro sets focus to the Close button of the Solution Explorer window, regardless of the currently selected UI context.

//# ui context VSROOT => "Solution Explorer", "Close"
UiContext.SetFocus();

It is also possible to use the Windows desktop as the base for search paths, by specifying DESKTOP in the UI context statement. The following macro opens a message box and then sets focus to the OK button.

1
2
3
4
//# using System.Windows.Forms
Task.Run(() => MessageBox.Show("Press OK to close.", "Hello"));
//# ui context DESKTOP => "Hello", "OK"
UiContext.SetFocus();

Specifying HWND in the UI context statement allows a window handle to be used to set the new UI context. The following macro waits for a process to run matching the first project in the current solution, and then sets the UI context to the main window of that process.

//# using "Process = System.Diagnostics.Process"
Func<Process> GetProjectProcess = (string projectName) =>
{
    return Process.GetProcesses()
        .Where(p => p.ProcessName == projectName)
        .First();
}

var solution = Dte.Solution as Solution2;
var project = solution.Projects.Cast<Project>().First();
//# wait Process process 15000 => GetProjectProcess(project.Name)
//# ui context HWND => process.MainWindowHandle

Using UI Automation patterns

To interact with UI elements, macros must access the control pattern interfaces for the intended functionality. This is provided by the //# ui pattern statement. The following macro code sets the text in the VS search box.

1
2
3
//# ui context VSROOT => "Search Visual Studio Text Box"
//# ui pattern Value txtSearchVS
txtSearchVS.SetValue("foo");

As in the UI context statement, a search path can be specified to locate a child UI element. However, the current UI context will not be modified by the UI pattern statement. In the following macro, both UI pattern statements are executed in the context of the VS root UI element.

1
2
3
4
5
//# ui context VSROOT
//# ui pattern SelectionItem configRelease => "Solution Configurations", "Release"
//# ui pattern SelectionItem platformX64 => "Solution Platforms", "x64"
configRelease.Select();
platformX64.Select();

Reference

Macro statements

call statement

Invoke another macro.

<code><b>//# call</b> <i><u>macro-name</u></i></code>

ref statement

Add reference to assembly.

<code><b>//# ref</b> <u><i>assembly</i></u></code><br>

using statement

Add reference to namespace.

<code><b>//# using</b> <i><u>namespace</u></i></code><br>
<code><b>//# using "</b><i><u>alias</u></i><b> = </b><u><i>namespace</i></u><b>"</b></code><br>

wait statement

Wait until expression evaluation returns non-default value. Optionally assign evaluated value to variable.

<code><b>//# wait</b> <i>[<u>timeout</u>]</i> <i>[<u>var-type</u> <u>var-name</u>]</i><b> => </b><i><u>expression</u></i></code><br>

var statement

Declare global variable

<code><b>//# var</b> <i><u>var-type</u> <u>var-name</u> [</i><b> => </b><i><u>expression</u>]</i></code><br>

service statement

Get Visual Studio SDK service.

<code><b>//# service</b> <i><u>var-name</u> <u>service-interface</u> [<u>service-type</u>]</i></code><br>

ui statement

UI automation commands.

ui context

Set UI context based on an automation element name path.

<code><b>//# ui context</b> <i>[ </i><b>VSROOT</b> | <b>DESKTOP</b><i> ] [<u>timeout</u>]</i><b> => </b><i><u>path</u></i></code><br>

ui context HWND

Set UI context based on a window handle.

<code><b>//# ui context HWND</b> <i>[<u>timeout</u>]</i><b> => </b><i><u>window-handle</u></i></code><br>

ui pattern

Access UI pattern, relative to the current UI context.

<code><b>//# ui pattern</b> <i><u>pattern-type</u> <u>var-name</u> [</i><b> => </b><i><u>path</u>]</i></code><br>

ui pattern Invoke

Activate Invoke UI pattern, relative to the current UI context.

<code><b>//# ui pattern Invoke</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>

ui pattern Toggle

Activate Toggle UI pattern, relative to the current UI context.

<code><b>//# ui pattern Toggle</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>

thread statement

Switch between UI and worker thread.

<code><b>//# thread ui</b></code><br>
<code><b>//# thread default</b></code><br>

quit statement

Exit Visual Studio.

<code><b>//# quit</b></code><br>

MacroClient class

#include <MacroClient.h>

Connect to QtVSTest

bool connect(qint64 *refPid = 0)
Parameter Description
refPid <li>refPid == 0: start VS before connection;<br><li>*refPid == 0: start VS before connection and save the new process ID in *refPid;<br><li>*refPid != 0: connect to QtVSTest in running process with ID *refPid.
Returns true if connection succeeded; false otherwise.

Disconnect from QtVSTest

void disconnect(bool close)
Parameter Description
close <li>true: disconnect from QtVSTest and close VS;<br><li>false: disconnect from QtVSTest but leave VS running.

Run macro

QString runMacro(QString macroCode)
Parameter Description
macroCode String containing C# macro code.
Returns Value of Result macro variable.

<br>

QString runMacro(QFile &macroFile)
Parameter Description
macroCode File containing C# macro code.
Returns Value of Result macro variable.

Store macro

QString storeMacro(QString macroName, QString macroCode)
Parameter Description
macroName Name under which the macro will be stored; other macros can use this name to invoke the stored macro.
macroCode String containing C# macro code to store.
Returns <li>MACRO_OK: macro stored successful and available for use by other macros; <li>otherwise, returns error description.

<br>

QString storeMacro(QString macroName, QFile &macroFile)
Parameter Description
macroName Name under which the macro will be stored; other macros can use this name to invoke the stored macro.
macroCode File containing C# macro code to store.
Returns <li>MACRO_OK: macro stored successful and available for use by other macros; <li>otherwise, returns error description.

Visual Studio SDK

Windows UI Automation

Edit Report
Pub: 29 Apr 2021 15:15 UTC
Views: 88