Qt Test Framework for Visual Studio and the Qt VS Tools
The Qt VS Test extension provides an entry point for testing toolkits (e.g. Qt Test) to run automated tests in the context of the Visual Studio IDE. The extension receives snippets of C# code, i.e. macros, from client apps, and then compiles and runs those macros. When completed (successfully or with errors), a macro can return data back to the client app, which can be compared to an expected result and yield a test status.
Contents
- Pre-requisites
- Visual Studio auto-testing with Qt Test
- Running macros in VS
- Visual Studio SDK services
- Interacting with UI elements
- Reference
Pre-requisites
To use the Qt VS Test extension, the following requirements must first be observed:
- Install Visual Studio 2019 or 2017 (any edition)
- https://docs.microsoft.com/en-us/visualstudio/install/install-visual-studio
- Minimum workloads for installation:
- Desktop development with C++
- .NET desktop development
- Visual Studio extension development
- Linux development with C++
- Install Qt VS Test extension
- Install Qt VS Tools (if testing this extension)
- After installation is complete, open the VS IDE, to allow extension initialization to occur.
Visual Studio auto-testing with Qt Test
Though QtVSTest is a generic tool that can be used to test any functionality within the VS IDE, and can in principle integrate with any automated testing framework, it was designed to provide QtTest applications the ability to run auto-tests in Visual Studio, specifically targeting the Qt Visual Studio Tools extension. This section shows how to setup a Qt Test application to run automated tests inside the Visual Studio IDE by connecting to the QtVSTest extension.
New Qt Test project in Qt Creator
To create a new Qt Test project in Qt Creator:
- New Project
- Other Project > Auto Test Project
- Choose location
- Details
- Test framework: Qt Test
- Build system: qmake
- Choose test case name, e.g. 'foo'
- Enable generation of initialization and cleanup code
- Select relevant kit(s)
After the project is created:
- Add
networkmodule toQT - Add
$$(LOCALAPPDATA)/qtvstesttoINCLUDEPATH
New Qt Test project in Visual Studio
Start Visual Studio from a development command prompt:
- E.g., if using Visual Studio 2019, run "Developer Command Prompt for VS 2019"
- At the command prompt, run
devenv
Create a new Qt project:
- File > New > Project
- Qt Console Application
- Set name and location
- In the configurations page of the wizard, add the
NetworkandTestmodules.
After the project is created:
- Open project properties (right-click > Properties)
- Select "All Configurations"
- Edit VC++ Directories > Include Directories:
- Add
$(LOCALAPPDATA)\qtvstest
- Add
- In
main.cpp: include Qt Test
- In
main.cpp: add the following class:
- In
main.cpp: replacemain()with:
- In
main.cpp: add the following at the end
Connecting to Qt VS Test
- Include
MacroClient.h- This header contains helper functions for connecting to the Qt VS Test, sending macro code and receiving macro execution results.
- Add a
MacroClientvariable to the test class
- Connect to Qt VS Test in the
initTestCase()slot:- Calling
client.connect()without a parameter will first start a new instance of the VS IDE and then connect to the Qt VS Test extension in that instance.
- Calling
- Disconnect from Qt VS Test in the
cleanupTestCase()slot:- Calling
client.disconnect()with atrueparameter will terminate the VS process after disconnecting.
- Calling
After the above steps (in either Qt Creator or Visual Studio), we should have a bare-bones Qt Test application that successfully starts Visual Studio, connects to the Qt VS Test extension, and then disconnects and closes the VS IDE.
Running macros in VS
Macros consist of C# code, together with specialized macro statements that are expanded by the Qt VS Test extension. Macro statements are lines of code starting with //# and ending at the next line break (CR+LF). These lines are interpreted by the extension and corresponding code is generated inline with the C# statements. The generated code is then compiled and the resulting function is executed. The result of running the macro function is stored in a global string variable Result and sent back to the test client when the macro execution terminates.
Simple macro
The following test case opens a message box in Visual Studio.
The //# using macro statement includes the System.Windows.Forms namespace. The macro execution and the client test app will block until the message box is closed.
Verifying macro result
The following test case opens a message box and waits 15 seconds for the user to close it. If the message box is not closed within that time the test case fails.
The message box is opened on a separate thread and the //# wait macro statement is used to ensure it is closed within 15 seconds. If it is closed, the wait statement is successful and the macro returns MACRO_OK. Otherwise, a timeout exception is generated at the //# wait statement. The macro return value will be the exception error message, and the test case will fail.
It is also possible to explicitly set the return value of a macro, by setting the Result string variable.
Using macro files
MacroClient::runMacro() will also accept a QFile as input. The contents of the file are read and sent over to QtVSTest as the macro code to be executed.
Invoking other macros
The //# call statement invokes the execution of named macros stored by QtVSTest. The MacroClient::storeMacro() function can be used to store a named macro. QtVSTest will process the macro code but, instead of executing it immediately, will store it under the provided name.
Macro variables
Macros can share data by means of global variables. Macro variables are declared with the //# var statement.
Visual Studio SDK services
Macros can invoke services of the VS SDK to access Visual Studio IDE features. The //# service statement is used to obtain a reference to a VS SDK service. The following macro accesses the SVsShell service through the IVsShell interface to check if Visual Studio is running with admin rights.
DTE service
Macros include in their run-time context a reference to the DTE service. This is the top-level object in the Visual Studio automation object model. The following macro uses this service to check if a solution is loaded.
Interacting with UI elements
Macros can use UI Automation interfaces to interact with UI elements. The run-time context of a macro includes the currently selected UI element. This is an AutomationElement object in the system-wide UI automation tree (rooted on the Windows desktop). The current UI context can be accessed in macro code through the UiContext variable. At the start of a macro run, the default UI context is the root Visual Studio UI element.
NB: the tool inspect.exe, part of the Windows SDK, provides a useful graphical view of the tree of UI automation elements, and can be helpful to test the interaction with those elements.
Locating UI Automation elements
The //# ui context statement can be used to search for a descendant of the currently selected element, and set it as the new UI context. To locate a descendant element, a search path must be specified, consisting of a sequence of names of descendant nodes. The following macro selects the "Open File" button on the VS toolbar, and sets input focus to that button. The VS root UI element is assumed to be currently selected.
By specifying VSROOT to the statement, the root UI element of Visual Studio will be used as the base for search paths. The following macro sets focus to the Close button of the Solution Explorer window, regardless of the currently selected UI context.
It is also possible to use the Windows desktop as the base for search paths, by specifying DESKTOP in the UI context statement. The following macro opens a message box and then sets focus to the OK button.
Specifying HWND in the UI context statement allows a window handle to be used to set the new UI context. The following macro waits for a process to run matching the first project in the current solution, and then sets the UI context to the main window of that process.
Using UI Automation patterns
To interact with UI elements, macros must access the control pattern interfaces for the intended functionality. This is provided by the //# ui pattern statement. The following macro code sets the text in the VS search box.
As in the UI context statement, a search path can be specified to locate a child UI element. However, the current UI context will not be modified by the UI pattern statement. In the following macro, both UI pattern statements are executed in the context of the VS root UI element.
Reference
Macro statements
call statement
Invoke another macro.
<code><b>//# call</b> <i><u>macro-name</u></i></code>
ref statement
Add reference to assembly.
<code><b>//# ref</b> <u><i>assembly</i></u></code><br>
using statement
Add reference to namespace.
<code><b>//# using</b> <i><u>namespace</u></i></code><br>
<code><b>//# using "</b><i><u>alias</u></i><b> = </b><u><i>namespace</i></u><b>"</b></code><br>
wait statement
Wait until expression evaluation returns non-default value. Optionally assign evaluated value to variable.
<code><b>//# wait</b> <i>[<u>timeout</u>]</i> <i>[<u>var-type</u> <u>var-name</u>]</i><b> => </b><i><u>expression</u></i></code><br>
var statement
Declare global variable
<code><b>//# var</b> <i><u>var-type</u> <u>var-name</u> [</i><b> => </b><i><u>expression</u>]</i></code><br>
service statement
Get Visual Studio SDK service.
<code><b>//# service</b> <i><u>var-name</u> <u>service-interface</u> [<u>service-type</u>]</i></code><br>
ui statement
UI automation commands.
ui context
Set UI context based on an automation element name path.
<code><b>//# ui context</b> <i>[ </i><b>VSROOT</b> | <b>DESKTOP</b><i> ] [<u>timeout</u>]</i><b> => </b><i><u>path</u></i></code><br>
ui context HWND
Set UI context based on a window handle.
<code><b>//# ui context HWND</b> <i>[<u>timeout</u>]</i><b> => </b><i><u>window-handle</u></i></code><br>
ui pattern
Access UI pattern, relative to the current UI context.
<code><b>//# ui pattern</b> <i><u>pattern-type</u> <u>var-name</u> [</i><b> => </b><i><u>path</u>]</i></code><br>
ui pattern Invoke
Activate Invoke UI pattern, relative to the current UI context.
<code><b>//# ui pattern Invoke</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>
ui pattern Toggle
Activate Toggle UI pattern, relative to the current UI context.
<code><b>//# ui pattern Toggle</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>
thread statement
Switch between UI and worker thread.
<code><b>//# thread ui</b></code><br>
<code><b>//# thread default</b></code><br>
quit statement
Exit Visual Studio.
<code><b>//# quit</b></code><br>
MacroClient class
Connect to QtVSTest
Parameter Description refPid<li> refPid == 0: start VS before connection;<br><li>*refPid == 0: start VS before connection and save the new process ID in*refPid;<br><li>*refPid != 0: connect to QtVSTest in running process with ID*refPid.Returns trueif connection succeeded;falseotherwise.
Disconnect from QtVSTest
Parameter Description close<li> true: disconnect from QtVSTest and close VS;<br><li>false: disconnect from QtVSTest but leave VS running.
Run macro
Parameter Description macroCodeString containing C# macro code. Returns Value of Resultmacro variable.
<br>
Parameter Description macroCodeFile containing C# macro code. Returns Value of Resultmacro variable.
Store macro
Parameter Description macroNameName under which the macro will be stored; other macros can use this name to invoke the stored macro. macroCodeString containing C# macro code to store. Returns <li> MACRO_OK: macro stored successful and available for use by other macros; <li>otherwise, returns error description.
<br>
Parameter Description macroNameName under which the macro will be stored; other macros can use this name to invoke the stored macro. macroCodeFile containing C# macro code to store. Returns <li> MACRO_OK: macro stored successful and available for use by other macros; <li>otherwise, returns error description.