Qt Test Framework for Visual Studio and the Qt VS Tools
The Qt VS Test extension provides an entry point for testing toolkits (e.g. QtTest) to run automated tests in the context of the Visual Studio IDE. The extension receives snippets of C# code, i.e. macros, from client apps, and then compiles and runs those macros. When completed (successfully or with errors), a macro can return data back to the client app, which can be compared to an expected result and yield a test status.
Though it is a generic tool that can be used to test any functionality within the VS IDE, and can integrate with any testing framework, the Qt VS Test extension was developed with the specific objective of providing QtTest
applications the ability to run auto-tests on the Qt Visual Studio Tools extension.
Pre-requisites
To use the Qt VS Test extension, the following requirements must first be observed:
- Install Visual Studio 2019 or 2017 (any edition)
- https://docs.microsoft.com/en-us/visualstudio/install/install-visual-studio
- Minimum workloads for installation:
- Desktop development with C++
- .NET desktop development
- Visual Studio extension development
- Linux development with C++
- Install Qt VS Test extension
- Install Qt VS Tools (if testing this extension)
- After installation is complete, open the VS IDE at least once, to allow extension initialization to occur.
Qt Test project for Visual Studio
As mentioned above, the Qt VS Test extension is a generic tool that can be used with any test framework or project system. The following is a description of the integration of the Qt VS Test extension with QtTest application projects in Qt Creator and Visual Studio.
Create project
In Qt Creator
To create a new QtTest project in Qt Creator:
- New Project
- Other Project > Auto Test Project
- Choose location
- Details
- Test framework: Qt Test
- Build system: qmake
- Choose test case name, e.g. 'foo'
- Enable generation of initialization and cleanup code
- Select relevant kit(s)
After the project is created:
- Add
network
module toQT
- Add
$$(LOCALAPPDATA)/qtvstest
toINCLUDEPATH
In Visual Studio
Start Visual Studio from a development command prompt:
- E.g., if using Visual Studio 2019, run "Developer Command Prompt for VS 2019"
- At the command prompt, run
devenv
Create a new Qt project:
- File > New > Project
- Qt Console Application
- Set name and location
- In the configurations page of the wizard, add the
Network
andTest
modules.
After the project is created:
- Open project properties (right-click > Properties)
- Select "All Configurations"
- Edit VC++ Directories > Include Directories:
- Add
$(LOCALAPPDATA)\qtvstest
- Add
- In
main.cpp
: include Qt Test
- In
main.cpp
: add the following class:
- In
main.cpp
: replacemain()
with:
- In
main.cpp
: add the following at the end
Connecting to Qt VS Test
- Include
MacroClient.h
- This header contains helper functions for connecting to the Qt VS Test, sending macro code and receiving macro execution results.
- Add a
MacroClient
variable to the test class
- Connect to Qt VS Test in the
initTestCase()
slot:- Calling
client.connect()
without a parameter will first start a new instance of the VS IDE and then connect to the Qt VS Test extension in that instance.
- Calling
- Disconnect from Qt VS Test in the
cleanupTestCase()
slot:- Calling
client.disconnect()
with atrue
parameter will terminate the VS process after disconnecting.
- Calling
After the above steps (in either Qt Creator or Visual Studio), we should have a bare-bones Qt Test application that successfully starts Visual Studio, connects to the Qt VS Test extension, and then disconnects and closes the VS IDE.
Running macros in VS
Macros consist of C# code, together with specialized macro statements that are expanded by the Qt VS Test extension. Macro statements are lines of code starting with //#
and ending at the next line break (CR+LF). These lines are interpreted by the extension and corresponding code is generated inline with the C# statements. The generated code is then compiled and the resulting function is executed. The result of running the macro function is stored in a global string variable Result
and sent back to the test client when the macro execution terminates.
Simple macro
The following test case opens a message box in Visual Studio.
The //# using
macro statement is used to include the System.Windows.Forms
namespace. The macro execution and the client test app will block until the message box is closed. The result of the macro is not set and a default empty string is returned.
Verifying macro result
The following test case opens a message box and waits 15 seconds for the user to close it. If the message box is not closed within that time the test case fails.
The message box is opened on a separate thread and the //# wait
macro statement is used to ensure it is closed within 15 seconds. If it is closed, the wait statement is successful and the macro returns "(ok)"
. Otherwise, a timeout exception is generated at the //# wait
statement; the macro return value will be the exception error message, and the test case will fail.
Using macro files
TO_DO
Invoking other macros
TO_DO
Manipulating UI elements
TO_DO
Visual Studio SDK objects and services
TO_DO
Macro statements
using
//# using <namespace>
//# using "<alias> = <namespace>"
ref
//# ref <assembly name> [[namespace] [namespace] ...]
wait
//# wait [timeout] => <expr>
//# wait [timeout] <var_type> <var_name> => <expr>
var
//# var <type> <name> [=> <initial value>]
service
//# service <var name> <service interface> [service type]
ui
ui context VS
//# ui context [ VS [timeout] ] => <path>
ui context HWND
//# ui context HWND => <handle>
ui pattern
//# ui pattern <var_type> <var_name> [ => <path> ]
ui pattern Invoke
//# ui pattern Invoke [ => <path> ]
ui pattern Toggle
//# ui pattern Invoke [ => <path> ]
thread
//# thread <ui | default>
macro
//# macro <macro_name>
call
//# call <macro_name>
quit
//# quit
MacroClient
Class
Public functions
Related non-members
Visual Studio SDK
https://docs.microsoft.com/en-us/dotnet/api/?view=visualstudiosdk-2019
Windows UI Automation
https://docs.microsoft.com/en-us/dotnet/api/system.windows.automation?view=net-5.0
inspect.exe
https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects