Introduction
Qt OPC UA is a library we have started in 2015 and maintained ever since. It is covered by multiple extensive test suites written in C++ using the Qt Test framework. The existing test suites bring up a custom built server application based on the open62541 library and then test the API of QOpcUaClient and QOpcUaNode against that server.
Conceptually, these tests are black box tests since they are written against the public headers of Qt OPC UA. The test suites have served us well over the years, but reading the test source isn’t pleasant. Often there are a lots of implementation details which make it hard to figure out the intent of the test. After our successful evaluation of Cucumber-JS for black box tests of an OPC UA server, we decided to have a look at Cucumber-CPP and what using it to build tests for the Qt OPC UA C++ API would feel like.
We provide the full source code for the case-study project on Github.
Cucumber-CPP
Cucumber-CPP is currently the only solution to run tests written in Gherkin using step definitions written in C++. Our first impression was that it is much less comfortable compared to other language specific implementations like Cucumber-JS because it doesn’t provide a way to parse the .feature files and execute the test scenarios itself. Instead, the intended way of using it is to build an executable which opens a TCP server with support for the JSON based cucumber wire protocol on localhost.
The official Ruby based Cucumber implementation with the cucumber-ruby-wire plugin must be executed manually after starting that executable. It parses the .feature files, communicates with the server using the cucumber wire protocol, requests the execution of the step definitions and gathers the results.
Test Setup
We decided to implement just a few basic scenarios to test the water and not to attempt to recreate the full test coverage of the original test suites. The examples from the repository served us as a good first reference on how to structure the code and which of the libraries built by the Cucumber-CPP project to link against.
Features And Scenarios
The main focus of the test cases was to cover the basic functionalities (read/write/browsing/monitored items/method calls) superficially and to find out if common test steps could be extracted.
We ended up with six scenarios that share at least some common steps.
Feature: Qt OPC UA Basics
Test basic functionality of QOpcUaClient and QOpcUaNode
Scenario: Write and read back (string)
Given a connected client
When I write "foo" to "ns=2;s=Demo.Static.Scalar.String"
And I read the value of "ns=2;s=Demo.Static.Scalar.String"
Then the value should be "foo"
Scenario: Write and read back (int64)
Given a connected client
When I write 42 to "ns=2;s=Demo.Static.Scalar.Int64"
And I read the value of "ns=2;s=Demo.Static.Scalar.Int64"
Then the value should be 42
Scenario: Browse children
Given a connected client
When I browse the children of "ns=3;s=TestFolder"
Then I should see that the browse result contains 86 children
And I should see that the browse result contains a child named "2:DateTimeArrayTest"
Scenario: Value monitoring
Given a connected client
When I write "foo" to "ns=2;s=Demo.Static.Scalar.String"
And I enable value monitoring for "ns=2;s=Demo.Static.Scalar.String"
And I wait 200 ms
And I write "bar" to "ns=2;s=Demo.Static.Scalar.String"
And I wait 200 ms
And I write "baz" to "ns=2;s=Demo.Static.Scalar.String"
And I wait 200 ms
Then I should have received 3 value attribute updates
And value update #0 should be "foo"
And value update #1 should be "bar"
And value update #2 should be "baz"
Scenario: Event monitoring
Given a connected client
When I enable event monitoring for "i=2253"
And I trigger an event with severity 23
And I trigger an event with severity 42
And I trigger an event with severity 5
Then I should have received an event with severity 23
And I should have received an event with severity 42
And I should have received an event with severity 5
Scenario Outline: Method calls
Given a connected client
When I call the multiply method with input arguments <InputA> and <InputB>
Then the method call should have returned Good
And output argument #0 should be <Product>
Examples:
| InputA | InputB | Product |
| 21 | 2 | 42 |
| 8 | 8 | 64 | To give an impression of the improved readability, the following snippet shows a part of the basic setup of the event monitoring unit test written in C++.
QFETCH(QOpcUaClient *, opcuaClient);
OpcuaConnector connector(opcuaClient, m_endpoint);
QScopedPointer<QOpcUaNode> serverNode(opcuaClient->node(QOpcUa::namespace0Id(QOpcUa::NodeIds::Namespace0::Server)));
QVERIFY(serverNode != nullptr);
QScopedPointer<QOpcUaNode> testFolderNode(opcuaClient->node("ns=3;s=TestFolder"));
QVERIFY(testFolderNode != nullptr);
QSignalSpy enabledSpy(serverNode.data(), &QOpcUaNode::enableMonitoringFinished);
QSignalSpy eventSpy(serverNode.data(), &QOpcUaNode::eventOccurred);
QOpcUaMonitoringParameters::EventFilter filter;
filter << QOpcUaSimpleAttributeOperand("Severity");
filter << QOpcUaSimpleAttributeOperand("Message");
QOpcUaMonitoringParameters p(0);
p.setQueueSize(10); // Without setting the queue size, we get a queue overflow event after triggering both events without waiting
p.setFilter(filter);
serverNode->enableMonitoring(QOpcUa::NodeAttribute::EventNotifier, p);
enabledSpy.wait();
QCOMPARE(enabledSpy.size(), 1);
QCOMPARE(enabledSpy.at(0).at(0).value<QOpcUa::NodeAttribute>(), QOpcUa::NodeAttribute::EventNotifier);
QCOMPARE(enabledSpy.at(0).at(1).value<QOpcUa::UaStatusCode>(), QOpcUa::UaStatusCode::Good);
QCOMPARE(serverNode->monitoringStatus(QOpcUa::NodeAttribute::EventNotifier).filter().value<QOpcUaMonitoringParameters::EventFilter>(),
filter); The CMake Project
As we are building a test setup for a Qt based library, we decided to go with the Qt Test driver that comes with Cucumber-CPP. The latest v0.7.0 tag from December 2023 explicitly invokes find_package() for the Qt 5 libraries, so we went with the current main branch, where Qt 6 support is available and can be enabled using the CUKE_ENABLE_QT_6 option in CMake.
To keep the external dependencies of the project small, we decided to add Cucumber-CPP as a git submodule and to build it with the project using add_subdirectory(). This also ensures that it is built against the same version of Qt 6 as the Qt OPC UA module under test.
The full CMakeLists.txt is available on Github.
Context Class And Step Definitions
The context class (equivalent to the World class in Cucumber-JS) and the step definitions are implemented in a single file named CucumberCppQtOpcUaTestsSteps.cpp which is built and linked against the Qt6::Test, Qt6::OpcUa and cucumber-cpp libraries. For a more extensive test setup, spreading the definitions to multiple files might be a good idea.
It is mandatory to include the QTest header before the Cucumber-CPP header to allow Cucumber-CPP to identify the test framework in use.
#include <QTest>
#include "cucumber-cpp/autodetect.hpp" Due to the asynchronous nature of the Qt OPC UA API, QSignalSpy is the test implementer’s best friend because it provides an easy way to record received signals and to wait until a signal is received or a timeout occurs, whatever happens first. While there are some step definitions where the signal carrying the result can be checked directly in the step definition, other cases like monitored items or method calls can be implemented much easier if the corresponding QSignalSpy is part of the context and available to all the following steps in a scenario.
When we attempted to run the first step, we were puzzled because the test steps did never return from the first call to QSignalSpy::wait(). It turned out that the Qt Test driver in Cucumber-CPP does not use an event loop, so we had to bring our own by instantiating a static QCoreApplication in our code. With this additional step, waiting for QSignalSpy worked as intended and the expected signals were recorded.
#include <QCoreApplication>
#include <QObject>
#include <QProcess>
#include <QSignalSpy>
#include <QtOpcUa/QOpcUaClient>
#include <QtOpcUa/QOpcUaNode>
#include <QtOpcUa/QOpcUaProvider>
class QtOpcUaTestCtx
{
public:
QtOpcUaTestCtx() { }
~QtOpcUaTestCtx() { }
QScopedPointer<QOpcUaClient> m_client;
QScopedPointer<QOpcUaNode> m_browseNode;
QList<QOpcUaReferenceDescription> m_references;
QScopedPointer<QOpcUaNode> m_monitoringNode;
QScopedPointer<QSignalSpy> m_attributeUpdatedSpy;
QScopedPointer<QSignalSpy> m_eventSpy;
QScopedPointer<QOpcUaNode> m_methodCallObjectNode;
QScopedPointer<QSignalSpy> m_methodCallSpy;
QScopedPointer<QSignalSpy> m_readSpy;
// Global state
static QProcess m_serverProcess;
static const QUrl m_serverUrl;
// A QCoreApplication is required to use QSignalSpy::wait()
static QCoreApplication m_app;
// Constants
static const QString PluginName;
static const QString TestFolderId;
static const QString MultiplyMethodId;
static const QString SeverityName;
static const QString TriggerEventId;
}; Using QString values for placeholders in the step definitions requires manually adding a stream operator for QString and std::istream:
std::istream &operator>>(std::istream &in, QString &val)
{
std::string s;
in >> s;
val = QString::fromStdString(s);
return in;
} We use the BeforeAll and AfterAll hooks provided by Cucumber-CPP to bring up the open62541 based test server before running the first scenario and to shut it down after the last scenario has finished.
// Bring up the test server and make sure it is responsive
BEFORE_ALL()
{
const auto serverPath = QString::fromUtf8(qgetenv("TEST_SERVER_PATH"));
QVERIFY(!serverPath.isEmpty());
QTcpSocket socket;
socket.connectToHost(QtOpcUaTestCtx::m_serverUrl.host(), QtOpcUaTestCtx::m_serverUrl.port());
QVERIFY2(socket.waitForConnected(1500) == false, "Server is already running");
QtOpcUaTestCtx::m_serverProcess.setProgram(serverPath);
QtOpcUaTestCtx::m_serverProcess.start();
QCOMPARE(QtOpcUaTestCtx::m_serverProcess.waitForStarted(), true);
QTest::qWait(1500);
socket.connectToHost(QtOpcUaTestCtx::m_serverUrl.host(), QtOpcUaTestCtx::m_serverUrl.port());
QVERIFY(socket.waitForConnected(1500));
}
// Tear down the test server after all scenarios are done
AFTER_ALL()
{
if (QtOpcUaTestCtx::m_serverProcess.state() == QProcess::ProcessState::Running) {
QtOpcUaTestCtx::m_serverProcess.kill();
QtOpcUaTestCtx::m_serverProcess.waitForFinished();
}
} The step definitions are implemented using macros, the string describing the step and its placeholders is passed as a regular expression. Using different regular expressions to distinguish strings or numeric values in placeholders allow to “overload” step definitions to have separate implementations for different input types to the same step (for example, see the two implementations each for “I write xxx to yyy” and “the value should be xxx”).
The cucumber::ScenarioScope class template is used to fetch the current context object to add data or to retrieve data left by previous steps.
The REGEX_PARAM macro allows typed retrieving of captured parameters from the regular expression describing the test step. The streaming operator we defined before lets us get string parameters directly as QString so we don’t have to convert them manually in our code before passing them to any Qt API.
The rest of the step definition can be written like normal Qt Test based test code and use its assertion functions like QVERIFY() and QCOMPARE().
The code snippet below only contains a few step definitions, the full source code is available on Github.
WHEN("^I write \"(.*)\" to \"(.*)\"$")
{
REGEX_PARAM(QString, value);
REGEX_PARAM(QString, nodeId);
cucumber::ScenarioScope<QtOpcUaTestCtx> ctx;
QVERIFY(ctx->m_client);
QSignalSpy writeSpy(ctx->m_client.get(), &QOpcUaClient::writeNodeAttributesFinished);
ctx->m_client->writeNodeAttributes({ { nodeId, QOpcUa::NodeAttribute::Value, value } });
QVERIFY(writeSpy.wait());
QCOMPARE(writeSpy.size(), 1);
QCOMPARE(writeSpy.at(0).at(1), QOpcUa::UaStatusCode::Good);
const auto writeResults = writeSpy.at(0).at(0).value<QList<QOpcUaWriteResult>>();
QCOMPARE(writeResults.size(), 1);
QCOMPARE(writeResults.at(0).statusCode(), QOpcUa::UaStatusCode::Good);
}
WHEN("^I write ([^\" ]*) to \"(.*)\"$")
{
REGEX_PARAM(qint64, value);
REGEX_PARAM(QString, nodeId);
cucumber::ScenarioScope<QtOpcUaTestCtx> ctx;
QVERIFY(ctx->m_client);
QSignalSpy writeSpy(ctx->m_client.get(), &QOpcUaClient::writeNodeAttributesFinished);
ctx->m_client->writeNodeAttributes({ { nodeId, QOpcUa::NodeAttribute::Value, value } });
QVERIFY(writeSpy.wait());
QCOMPARE(writeSpy.size(), 1);
QCOMPARE(writeSpy.at(0).at(1), QOpcUa::UaStatusCode::Good);
const auto writeResults = writeSpy.at(0).at(0).value<QList<QOpcUaWriteResult>>();
QCOMPARE(writeResults.size(), 1);
QCOMPARE(writeResults.at(0).statusCode(), QOpcUa::UaStatusCode::Good);
}
WHEN("^I read the value of \"(.*)\"$")
{
REGEX_PARAM(QString, nodeId);
cucumber::ScenarioScope<QtOpcUaTestCtx> ctx;
QVERIFY(ctx->m_client);
ctx->m_readSpy.reset(
new QSignalSpy(ctx->m_client.get(), &QOpcUaClient::readNodeAttributesFinished));
QVERIFY(ctx->m_client->readNodeAttributes({ nodeId }));
}
THEN("^the value should be \"(.*)\"$")
{
REGEX_PARAM(QString, value);
cucumber::ScenarioScope<QtOpcUaTestCtx> ctx;
QVERIFY(ctx->m_readSpy);
if (ctx->m_readSpy->empty())
QVERIFY(ctx->m_readSpy->wait());
QCOMPARE(ctx->m_readSpy->size(), 1);
QCOMPARE(ctx->m_readSpy->at(0).at(1), QOpcUa::UaStatusCode::Good);
const auto results = ctx->m_readSpy->at(0).at(0).value<QList<QOpcUaReadResult>>();
QCOMPARE(results.size(), 1);
QCOMPARE(results.at(0).statusCode(), QOpcUa::UaStatusCode::Good);
QCOMPARE(results.at(0).value(), value);
}
THEN("^the value should be ([^\" ]*)$")
{
REGEX_PARAM(qint64, value);
cucumber::ScenarioScope<QtOpcUaTestCtx> ctx;
QVERIFY(ctx->m_readSpy);
if (ctx->m_readSpy->empty())
QVERIFY(ctx->m_readSpy->wait());
QCOMPARE(ctx->m_readSpy->size(), 1);
QCOMPARE(ctx->m_readSpy->at(0).at(1), QOpcUa::UaStatusCode::Good);
const auto results = ctx->m_readSpy->at(0).at(0).value<QList<QOpcUaReadResult>>();
QCOMPARE(results.size(), 1);
QCOMPARE(results.at(0).statusCode(), QOpcUa::UaStatusCode::Good);
QCOMPARE(results.at(0).value(), value);
} Building And Running The Tests
Building the project requires an installation of Qt 6.x (we used Qt 6.8, other versions might require a change to the expected number of children in the “Browse children” scenario) with the matching version of the Qt OPC UA module installed and the dependencies Cucumber-CPP mentions in its README. If Qt OPC UA is built with the CMake option QT_BUILD_TESTS=ON, the open62541 based test server is available in the bin directory of the module’s installation location.
After building the C++ test project, it must be run in an environment where the TEST_SERVER_PATH environment variable contains the path to the Qt OPC UA test server. Depending on the development setup, it might be necessary to point the LD_LIBRARY_PATH environment variable to the lib directory of the Qt installation to make the necessary runtime dependencies available to the test server binary.
$ export TEST_SERVER_PATH=/path/to/open62541-testserver
$ export LD_LIBRARY_PATH=/path/to/Qt/6.x.x/gcc_64/lib # Optional, if the server requires it
$ ./CucumberCppQtOpcUaTests &
$ cucumber After cucumber has finished and disconnected, the C++ executable will exit.
If everything went well, cucumber will print out all scenarios with their steps marked in green and conclude its output with a summary:
7 scenarios (7 passed)
40 steps (40 passed)
0m4.698s To see a test fail, we modify the expected number of children in the “Browse Children” scenario and run the test setup again:
Scenario: Browse children # features/opcua.feature:16
Given a connected client # CucumberCppQtOpcUaTestsSteps.cpp:103
When I browse the children of "ns=3;s=TestFolder" # CucumberCppQtOpcUaTestsSteps.cpp:190
Then I should see that the browse result contains 86 children # CucumberCppQtOpcUaTestsSteps.cpp:208
TAP version 13
# cucumber::internal::QtTestObject
ok 1 - initTestCase()
not ok 2 - test()
---
type: QCOMPARE
message: Compared values are not the same
wanted: 87 (numExpected)
found: 86 (ctx->m_references.size())
expected: 87 (numExpected)
actual: 86 (ctx->m_references.size())
at: cucumber::internal::QtTestObject::test() (/path/to/cucumber-cpp-opcua-demo/features/step_definitions/CucumberCppQtOpcUaTestsSteps.cpp:214)
file: /path/to/cucumber-cpp-opcua-demo/features/step_definitions/CucumberCppQtOpcUaTestsSteps.cpp
line: 214
...
ok 3 - cleanupTestCase()
1..3
# tests 3
# pass 2
# fail 1
(Cucumber::Wire::Exception)
features/opcua.feature:19:in `Then I should see that the browse result contains 86 children'
And I should see that the browse result contains a child named "2:DateTimeArrayTest" # CucumberCppQtOpcUaTestsSteps.cpp:217 As we can see, Cucumber-CPP transmits a comprehensive error output to cucumber, so it is easy to find out why and where the test step failed.
Lessons Learned
When comparing our existing black box tests written using QtTest with the feature files we created for this case-study we clearly see the advantage of separating test description and test implementation. While Cucumber-CPP is not as comfortable to use as Cucumber-JS, which provides a self sufficient environment to implement and execute tests, building a test setup with it is still quite straightforward. If starting the server application and running the cucumber executable are bundled in a script/CMake or CI pipeline, this aspect will be much less relevant to the developer experience.
Our selected test scenarios achieved a good degree of reusable test step definitions. But it is improbable that this would hold up for the rest of the tests we currently have in Qt OPC UA, because there are lots of tests checking for expected values in complex structured types with nested members. These tests involve for example writing crafted values of a certain type to nodes in the server both as scalars and arrays, reading them back and then checking each member of the structured types for the expected value. Find a solution for that requires further research.
