Qt Test Framework for Visual Studio and the Qt VS Tools
The Qt VS Test extension provides an entry point for testing toolkits (e.g. Qt Test) to run automated tests in the context of the Visual Studio IDE. The extension receives snippets of C# code, i.e. macros, from client apps, and then compiles and runs those macros. When completed (successfully or with errors), a macro can return data back to the client app, which can be compared to an expected result and yield a test status.
Contents
- Pre-requisites
- Visual Studio auto-testing with Qt Test
- Running macros in VS
- Visual Studio SDK services
- Interacting with UI elements
- Reference
Pre-requisites
To use the Qt VS Test extension, the following requirements must first be observed:
- Install Visual Studio 2019 or 2017 (any edition)
- https://docs.microsoft.com/en-us/visualstudio/install/install-visual-studio
- Minimum workloads for installation:
- Desktop development with C++
- .NET desktop development
- Visual Studio extension development
- Linux development with C++
- Install Qt VS Test extension
- Install Qt VS Tools (if testing this extension)
- After installation is complete, open the VS IDE, to allow extension initialization to occur.
Visual Studio auto-testing with Qt Test
Though QtVSTest is a generic tool that can be used to test any functionality within the VS IDE, and can in principle integrate with any automated testing framework, it was designed to provide QtTest
applications the ability to run auto-tests in Visual Studio, specifically targeting the Qt Visual Studio Tools extension. This section shows how to setup a Qt Test application to run automated tests inside the Visual Studio IDE by connecting to the QtVSTest extension.
New Qt Test project in Qt Creator
To create a new Qt Test project in Qt Creator:
- New Project
- Other Project > Auto Test Project
- Choose location
- Details
- Test framework: Qt Test
- Build system: qmake
- Choose test case name, e.g. 'foo'
- Enable generation of initialization and cleanup code
- Select relevant kit(s)
After the project is created:
- Add
network
module toQT
- Add
$$(LOCALAPPDATA)/qtvstest
toINCLUDEPATH
New Qt Test project in Visual Studio
Start Visual Studio from a development command prompt:
- E.g., if using Visual Studio 2019, run "Developer Command Prompt for VS 2019"
- At the command prompt, run
devenv
Create a new Qt project:
- File > New > Project
- Qt Console Application
- Set name and location
- In the configurations page of the wizard, add the
Network
andTest
modules.
After the project is created:
- Open project properties (right-click > Properties)
- Select "All Configurations"
- Edit VC++ Directories > Include Directories:
- Add
$(LOCALAPPDATA)\qtvstest
- Add
- In
main.cpp
: include Qt Test
- In
main.cpp
: add the following class:
- In
main.cpp
: replacemain()
with:
- In
main.cpp
: add the following at the end
Connecting to Qt VS Test
- Include
MacroClient.h
- This header contains helper functions for connecting to the Qt VS Test, sending macro code and receiving macro execution results.
- Add a
MacroClient
variable to the test class
- Connect to Qt VS Test in the
initTestCase()
slot:- Calling
client.connect()
without a parameter will first start a new instance of the VS IDE and then connect to the Qt VS Test extension in that instance.
- Calling
- Disconnect from Qt VS Test in the
cleanupTestCase()
slot:- Calling
client.disconnect()
with atrue
parameter will terminate the VS process after disconnecting.
- Calling
After the above steps (in either Qt Creator or Visual Studio), we should have a bare-bones Qt Test application that successfully starts Visual Studio, connects to the Qt VS Test extension, and then disconnects and closes the VS IDE.
Running macros in VS
Macros consist of C# code, together with specialized macro statements that are expanded by the Qt VS Test extension. Macro statements are lines of code starting with //#
and ending at the next line break (CR+LF). These lines are interpreted by the extension and corresponding code is generated inline with the C# statements. The generated code is then compiled and the resulting function is executed. The result of running the macro function is stored in a global string variable Result
and sent back to the test client when the macro execution terminates.
Simple macro
The following test case opens a message box in Visual Studio.
The //# using
macro statement includes the System.Windows.Forms
namespace. The macro execution and the client test app will block until the message box is closed.
Verifying macro result
The following test case opens a message box and waits 15 seconds for the user to close it. If the message box is not closed within that time the test case fails.
The message box is opened on a separate thread and the //# wait
macro statement is used to ensure it is closed within 15 seconds. If it is closed, the wait statement is successful and the macro returns MACRO_OK
. Otherwise, a timeout exception is generated at the //# wait
statement. The macro return value will be the exception error message, and the test case will fail.
It is also possible to explicitly set the return value of a macro, by setting the Result
string variable.
Using macro files
MacroClient::runMacro()
will also accept a QFile
as input. The contents of the file are read and sent over to QtVSTest as the macro code to be executed.
Invoking other macros
The //# call
statement invokes the execution of named macros stored by QtVSTest. The MacroClient::storeMacro()
function can be used to store a named macro. QtVSTest will process the macro code but, instead of executing it immediately, will store it under the provided name.
Macro variables
Macros can share data by means of global variables. Macro variables are declared with the //# var
statement.
Visual Studio SDK services
Macros can invoke services of the VS SDK to access Visual Studio IDE features. The //# service
statement is used to obtain a reference to a VS SDK service. The following macro accesses the SVsShell
service through the IVsShell
interface to check if Visual Studio is running with admin rights.
DTE
service
Macros include in their run-time context a reference to the DTE
service. This is the top-level object in the Visual Studio automation object model. The following macro uses this service to check if a solution is loaded.
Interacting with UI elements
Macros can use UI Automation interfaces to interact with UI elements. The run-time context of a macro includes the currently selected UI element. This is an AutomationElement
object in the system-wide UI automation tree (rooted on the Windows desktop). The current UI context can be accessed in macro code through the UiContext
variable. At the start of a macro run, the default UI context is the root Visual Studio UI element.
NB: the tool inspect.exe
, part of the Windows SDK, provides a useful graphical view of the tree of UI automation elements, and can be helpful to test the interaction with those elements.
Locating UI Automation elements
The //# ui context
statement can be used to search for a descendant of the currently selected element, and set it as the new UI context. To locate a descendant element, a search path must be specified, consisting of a sequence of names of descendant nodes. The following macro selects the "Open File" button on the VS toolbar, and sets input focus to that button. The VS root UI element is assumed to be currently selected.
By specifying VSROOT
to the statement, the root UI element of Visual Studio will be used as the base for search paths. The following macro sets focus to the Close button of the Solution Explorer window, regardless of the currently selected UI context.
It is also possible to use the Windows desktop as the base for search paths, by specifying DESKTOP
in the UI context statement. The following macro opens a message box and then sets focus to the OK button.
Specifying HWND
in the UI context statement allows a window handle to be used to set the new UI context. The following macro waits for a process to run matching the first project in the current solution, and then sets the UI context to the main window of that process.
Using UI Automation patterns
To interact with UI elements, macros must access the control pattern interfaces for the intended functionality. This is provided by the //# ui pattern
statement. The following macro code sets the text in the VS search box.
As in the UI context statement, a search path can be specified to locate a child UI element. However, the current UI context will not be modified by the UI pattern statement. In the following macro, both UI pattern statements are executed in the context of the VS root UI element.
Reference
Macro statements
call
statement
Invoke another macro.
<code><b>//# call</b> <i><u>macro-name</u></i></code>
ref
statement
Add reference to assembly.
<code><b>//# ref</b> <u><i>assembly</i></u></code><br>
using
statement
Add reference to namespace.
<code><b>//# using</b> <i><u>namespace</u></i></code><br>
<code><b>//# using "</b><i><u>alias</u></i><b> = </b><u><i>namespace</i></u><b>"</b></code><br>
wait
statement
Wait until expression evaluation returns non-default value. Optionally assign evaluated value to variable.
<code><b>//# wait</b> <i>[<u>timeout</u>]</i> <i>[<u>var-type</u> <u>var-name</u>]</i><b> => </b><i><u>expression</u></i></code><br>
var
statement
Declare global variable
<code><b>//# var</b> <i><u>var-type</u> <u>var-name</u> [</i><b> => </b><i><u>expression</u>]</i></code><br>
service
statement
Get Visual Studio SDK service.
<code><b>//# service</b> <i><u>var-name</u> <u>service-interface</u> [<u>service-type</u>]</i></code><br>
ui
statement
UI automation commands.
ui context
Set UI context based on an automation element name path.
<code><b>//# ui context</b> <i>[ </i><b>VSROOT</b> | <b>DESKTOP</b><i> ] [<u>timeout</u>]</i><b> => </b><i><u>path</u></i></code><br>
ui context HWND
Set UI context based on a window handle.
<code><b>//# ui context HWND</b> <i>[<u>timeout</u>]</i><b> => </b><i><u>window-handle</u></i></code><br>
ui pattern
Access UI pattern, relative to the current UI context.
<code><b>//# ui pattern</b> <i><u>pattern-type</u> <u>var-name</u> [</i><b> => </b><i><u>path</u>]</i></code><br>
ui pattern Invoke
Activate Invoke
UI pattern, relative to the current UI context.
<code><b>//# ui pattern Invoke</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>
ui pattern Toggle
Activate Toggle
UI pattern, relative to the current UI context.
<code><b>//# ui pattern Toggle</b> <i>[</i><b> => </b><i><u>path</u>]</i></code><br>
thread
statement
Switch between UI and worker thread.
<code><b>//# thread ui</b></code><br>
<code><b>//# thread default</b></code><br>
quit
statement
Exit Visual Studio.
<code><b>//# quit</b></code><br>
MacroClient
class
Connect to QtVSTest
Parameter Description refPid
<li> refPid == 0
: start VS before connection;<br><li>*refPid == 0
: start VS before connection and save the new process ID in*refPid
;<br><li>*refPid != 0
: connect to QtVSTest in running process with ID*refPid
.Returns true
if connection succeeded;false
otherwise.
Disconnect from QtVSTest
Parameter Description close
<li> true
: disconnect from QtVSTest and close VS;<br><li>false
: disconnect from QtVSTest but leave VS running.
Run macro
Parameter Description macroCode
String containing C# macro code. Returns Value of Result
macro variable.
<br>
Parameter Description macroCode
File containing C# macro code. Returns Value of Result
macro variable.
Store macro
Parameter Description macroName
Name under which the macro will be stored; other macros can use this name to invoke the stored macro. macroCode
String containing C# macro code to store. Returns <li> MACRO_OK
: macro stored successful and available for use by other macros; <li>otherwise, returns error description.
<br>
Parameter Description macroName
Name under which the macro will be stored; other macros can use this name to invoke the stored macro. macroCode
File containing C# macro code to store. Returns <li> MACRO_OK
: macro stored successful and available for use by other macros; <li>otherwise, returns error description.