Qt Test Framework for Visual Studio and the Qt VS Tools

The Qt VS Test extension provides an entry point for testing toolkits (e.g. QtTest) to run automated tests in the context of the Visual Studio IDE. The extension receives snippets of C# code, i.e. macros, from client apps, and then compiles and runs those macros. When completed (successfully or with errors), a macro can return data back to the client app, which can be compared to an expected result and yield a test status.

Though it is a generic tool that can be used to test any functionality within the VS IDE, and can integrate with any testing framework, the Qt VS Test extension was developed with the specific objective of providing QtTest applications the ability to run auto-tests on the Qt Visual Studio Tools extension.

Pre-requisites

To use the Qt VS Test extension, the following requirements must first be observed:

  • Install Visual Studio 2019 or 2017 (any edition)
  • Install Qt VS Test extension
  • Install Qt VS Tools (if testing this extension)
  • After installation is complete, open the VS IDE at least once, to allow extension initialization to occur.

Qt Test project for Visual Studio

As mentioned above, the Qt VS Test extension is a generic tool that can be used with any test framework or project system. The following is a description of the integration of the Qt VS Test extension with QtTest application projects in Qt Creator and Visual Studio.

Create project

In Qt Creator

To create a new QtTest project in Qt Creator:

  • New Project
  • Other Project > Auto Test Project
  • Choose location
  • Details
    • Test framework: Qt Test
    • Build system: qmake
    • Choose test case name, e.g. 'foo'
    • Enable generation of initialization and cleanup code
  • Select relevant kit(s)

After the project is created:

  • Add network module to QT
  • Add $$(LOCALAPPDATA)/qtvstest to INCLUDEPATH

In Visual Studio

Start Visual Studio from a development command prompt:

  • E.g., if using Visual Studio 2019, run "Developer Command Prompt for VS 2019"
  • At the command prompt, run devenv

Create a new Qt project:

  • File > New > Project
  • Qt Console Application
  • Set name and location
  • In the configurations page of the wizard, add the Network and Test modules.

After the project is created:

  • Open project properties (right-click > Properties)
  • Select "All Configurations"
  • Edit VC++ Directories > Include Directories:
    • Add $(LOCALAPPDATA)\qtvstest
  • In main.cpp: include Qt Test
    #include <QtTest>
    
  • In main.cpp: add the following class:
    class FooTest : public QObject
    {
        Q_OBJECT
    public:
        FooTest()
        {}
        ~FooTest()
        {}
    private slots:
        void initTestCase()
        {}
        void fooTestCase()
        {}
        void cleanupTestCase()
        {}
    };
    
  • In main.cpp: replace main() with:
    QTEST_APPLESS_MAIN(fooo)
    
  • In main.cpp: add the following at the end
    #include "main.moc"
    

Connecting to Qt VS Test

  • Include MacroClient.h
    • This header contains helper functions for connecting to the Qt VS Test, sending macro code and receiving macro execution results.
  • Add a MacroClient variable to the test class
    private:
        MacroClient client;
    
  • Connect to Qt VS Test in the initTestCase() slot:
    • Calling client.connect() without a parameter will first start a new instance of the VS IDE and then connect to the Qt VS Test extension in that instance.
      1
      2
      3
      4
      void initTestCase()
      {
          QVERIFY(client.connect());
      }
      
  • Disconnect from Qt VS Test in the cleanupTestCase() slot:
    • Calling client.disconnect() with a true parameter will terminate the VS process after disconnecting.
      1
      2
      3
      4
      void cleanupTestCase()
      {
          client.disconnect(true);
      }
      

After the above steps (in either Qt Creator or Visual Studio), we should have a bare-bones Qt Test application that successfully starts Visual Studio, connects to the Qt VS Test extension, and then disconnects and closes the VS IDE.

Running macros in VS

Macros consist of C# code, together with specialized macro statements that are expanded by the Qt VS Test extension. Macro statements are lines of code starting with //# and ending at the next line break (CR+LF). These lines are interpreted by the extension and corresponding code is generated inline with the C# statements. The generated code is then compiled and the resulting function is executed. The result of running the macro function is stored in a global string variable Result and sent back to the test client when the macro execution terminates.

Simple macro

The following test case opens a message box in Visual Studio.

1
2
3
4
5
6
void fooTestCase()
{
    client.runMacro(QString()
        % "//# using System.Windows.Forms\r\n"
        % "MessageBox.Show(\"Hello from Visual Studio!!\");");
}

The //# using macro statement is used to include the System.Windows.Forms namespace. The macro execution and the client test app will block until the message box is closed. The result of the macro is not set and a default empty string is returned.

Verifying macro result

The following test case opens a message box and waits 15 seconds for the user to close it. If the message box is not closed within that time the test case fails.

1
2
3
4
5
6
7
8
void fooTestCase()
{
    QCOMPARE(client.runMacro(QString()
        % "//# using System.Windows.Forms\r\n"
        % "var task = Task.Run(() => MessageBox.Show(\"Hello, close this in 15 secs!!\"));\r\n"
        % "//# wait 15000 => task.IsCompleted\r\n"
        % "Result = \"(ok)\";"), "(ok)");
}

The message box is opened on a separate thread and the //# wait macro statement is used to ensure it is closed within 15 seconds. If it is closed, the wait statement is successful and the macro returns "(ok)". Otherwise, a timeout exception is generated at the //# wait statement; the macro return value will be the exception error message, and the test case will fail.

Using macro files

TO_DO

Invoking other macros

TO_DO

Manipulating UI elements

TO_DO

Visual Studio SDK objects and services

TO_DO

Macro statements

using

//# using <namespace>
//# using "<alias> = <namespace>"

ref

//# ref <assembly name> [[namespace] [namespace] ...]

wait

//# wait [timeout] => <expr>
//# wait [timeout] <var_type> <var_name> => <expr>

var

//# var <type> <name> [=> <initial value>]

service

//# service <var name> <service interface> [service type]

ui

ui context VS

//# ui context [ VS [timeout] ] => <path>

ui context HWND

//# ui context HWND => <handle>

ui pattern

//# ui pattern <var_type> <var_name> [ => <path> ]

ui pattern Invoke

//# ui pattern Invoke [ => <path> ]

ui pattern Toggle

//# ui pattern Invoke [ => <path> ]

thread

//# thread <ui | default>

macro

//# macro <macro_name>

call

//# call <macro_name>

quit

//# quit

MacroClient Class

#include <MacroClient.h>

Public functions

MacroClient()
~MacroClient()
bool connect(qint64 *refPid = 0)
void disconnect(bool closeVs)
QString runMacro(QString macroCode)
QString runMacro(QFile &macroFile)
QString loadMacro(QFile &macroFile, QString macroName)
QString loadAndRunMacro(QFile &macroFile, QString macroHeader = QString())
#define MACRO_OK QStringLiteral("(ok)")
#define MACRO_ERROR QStringLiteral("(error)")
#define MACRO_ERROR_MSG(msg) QStringLiteral("(error)\r\n" msg)
#define MACRO_ASSERT_OK(result) QCOMPARE(result, MACRO_OK)
#define MACRO_GLOBALS(globalVars) "//# macro Globals\r\n" globalVars
#define MACRO_GLOBAL_VAR(varName, varValue) "//# var string " varName " => " varValue "\r\n"
inline bool macroResultOk(QString result) { return result == MACRO_OK; }
inline bool macroResultError(QString result) { return result.startsWith(MACRO_ERROR); }

Visual Studio SDK

https://docs.microsoft.com/en-us/dotnet/api/?view=visualstudiosdk-2019

Windows UI Automation

https://docs.microsoft.com/en-us/dotnet/api/system.windows.automation?view=net-5.0

inspect.exe

https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects

Edit Report
Pub: 20 Apr 2021 18:38 UTC
Views: 312