Motivation
The steadily growing importance of OPC UA has increased the demand for server applications to integrate existing or new devices into an OPC UA based ecosystem. An OPC UA server application usually includes business logic to handle method calls or to react to external inputs. If the server implements an existing companion specification, the behavior of the server also has to conform strictly to the specification in order to ensure interoperability with applications from other manufacturers.
While a good unit test coverage of the server’s code is a helpful building block to achieve and maintain compatibility with the specification, running tests against the public interface the server exposes to its clients is equally important to ensure that modifications to the code don’t introduce unwanted behavioral changes. This article demonstrates the basics of implementing black box tests for an OPC UA server that implements the AutoId companion specification and exposes an RfidReaderDeviceType object.
We provide the full source code for the example project on Github.
What Is Black Box Testing?
Black box testing is a way of software testing where the application under test is regarded as an opaque box with interfaces to the outside world which are tested against a specification of expected behavior. This means that the tests must not require knowledge of the application’s inner workings and ideally would still pass if the entire application would be rewritten from scratch.
Depending on the architecture of the application, it might still be necessary to create some test fixture. For example, the application might depend on inputs from attached devices or external processes it communicates with. In such a case, a controllable mockup of the external system is required in order to trigger state changes or events in the application.
Cucumber and Gherkin
Cucumber is a BDD tool that describes itself as a “tool for running automated acceptance tests, written in plain language”. It uses test definitions written in the Gherkin language which consists of plain language held together by some keywords and a certain structure.
A feature is a collection of scenarios which each have a short description and consist of one or more steps starting with a keyword. Step definitions can contain placeholders to enable running the same check with different values to serve as inputs to the test setup or as expected output values.
Feature: State handling for running scans and reader online/offline status must be checked
If the reader goes offline, the server state must change to Error and starting a scan must be forbidden.
While a scan is running, no new scan can be started.
Scenario: Attempt start while scan is running
Given a connected OPC UA client
And a running scan
Then I should see that the DeviceStatus is "Scanning"
When I call the ScanStart method with DataAvailable "false"
Then I should see that the method returned "BadInvalidState"
And I should see that the DeviceStatus is "Scanning" A key idea of Cucumber is to separate the test description from the test implementation. By doing so, the tests become much more expressive as the intent of the test isn’t hidden in implementation details. The feature files can even be used to communicate/brainstorm with less technical stake holders about how certain feature should work. Another way to think about Cucumber is “executable user stories”.
How do we implement these tests? When running the tests, they are fed into a Gherkin interpreter. This interpreter will look for an implementation of each step. There are Cucumber/Gherkin implementations in around 24 languages. For our example, we chose to implement Typescript as it is a modern language with a mature Cucumber implementation as well as a good OPC UA client library.
Cucumber-JS
The official JavaScript implementation of Cucumber is available as a Node.js module on NPM and provides all necessary components to implement a test setup and to run tests written in Gherkin. It comes with built-in TypeScript declarations and transpilation support, so test setups can be written entirely in TypeScript.
The core component of a TypeScript based test setup with Cucumber-JS is the World class which must be extended by the user to hold any state that is required between the steps that make up a scenario. A new World instance is created for each scenario. For our example, the World will contain the OPC UA client instance we use to interact with the application under test.
The step definitions are added as functions outside of any class using a helper function for each keyword. Cucumber-JS passes the World class instance as the first parameter so the step definitions can access and modify its data.
Given('a connected OPC UA client', async function (this: OpcUaWorld) {
// Create a client and add it to the World class instance
});
When(
'I call the ScanStart method with DataAvailable {string}',
async function (this: OpcUaWorld, dataAvailable: string) {
// Call the method using the client of the World class instance
// Store the result of the call in the World class instance
},
);
Then(
'I should see that the method returned {string}',
async function (this: OpcUaWorld, expectedStatus: string) {
// Assert that the result stored in the World class instance matches the expected status
},
); Step definition implementations can be spread over as many source files as the developer deems useful to achieve a logical grouping of related steps.
Our Test Setup
The OPC UA Server
As our application under test, we implemented a basic OPC UA server using the NodeOPCUA module. It loads the DI and AutoId companion specifications at runtime, instantiates an object node of type RfidReaderDeviceType and writes the identification properties. A business logic implementation handles calls to the ScanStart and ScanStop methods and updates the DeviceStatus variable.
To keep the implementation simple, our server project uses a TCP interface to communicate with a simulated RFID reader (in a real world application, this would most likely be a serial port). If the simulated reader sends information about a recognized RFID tag, an OPC UA event of type RfidScanEventType is generated if the current DeviceStatus is Scanning. If the DataAvailable parameter for the ScanStart method is true, DeviceStatus changes back to Idle after the first tag has been received from the reader.
The simulated reader is implemented as a TypeScript class with a public method to inject recognized tags for test purposes.
Note that our use of NodeOPCUA was purely for convenience. Even though we are testing using Cucumber.js, the server can be written in any language as we are purely testing via its OPC UA service interface.
Features And Scenarios
Our tests are split into three features which each reside in their own file
- opcua_autoid_basics.feature contains scenarios to check if the RfidReaderDeviceType instance exists in the server’s address space and if the identification properties look as expected
- opcua_autoid_scanning.feature checks if a single shot scan and a scan with manual stop work as expected based on defined input from the simulated reader
- opcua_autoid_statehandling.feature makes sure the state handling works as expected. For example, it must not be possible to start a scan while a scan is running.
See Github for full details.
The World Class
Our OpcUaWorld class extends the Cucumber-JS World class and adds an NodeOPCUA client used to interact with the OPC UA server. Several helper functions to find the necessary nodes in the server, to call the ScanStart and ScanStop methods and to read the DeviceStatus variable are also implemented to keep the step definition implementations as compact as possible.
The necessary variables to store information gathered in When steps to have them available in the related Then steps are added as public member variables so the step definitions can access them.
export class OpcUaWorld<ParametersType = any> extends World<ParametersType> {
uaClient: OPCUAClient | null = null; // A client connected to the server
session: ClientSession | null = null; // A session on the server
readerSimulator: RfidReaderSimulator;
server: AutoIdServer;
// Other member variables required to hold state between steps
receivedEvents: Variant[][] = [];
methodCallStatus: StatusCode | null = null;
constructor(options: IWorldOptions<ParametersType>) {
super(options);
// Get bind ports from the environment
const readerPort =
parseInt(process.env.READER_SIMULATOR_PORT || '') || 5678;
const opcuaPort = parseInt(process.env.OPCUA_BIND_PORT || '') || 4840;
// Instantiate a server and a simulated reader
this.readerSimulator = new RfidReaderSimulator(readerPort);
this.server = new AutoIdServer(opcuaPort, readerPort);
}
async initialize() {
this.readerSimulator.initialize();
await this.server.initialize();
}
async shutdown() {
await this.disconnectFromServer();
await this.server.stop();
await this.readerSimulator.shutdown();
}
// Methods to create a client, to connect to the server and to create a session
// Methods to find the necessary nodes to interact with the RfidReaderDeviceType
// Methods to call ScanStart and ScanStop on the server
// Methods to read variable values
}
// Register the world constructor
setWorldConstructor(OpcUaWorld); Our World also contains the necessary member variables and methods to instantiate the OPC UA server and the simulated reader. This is a detail of our example implementation. When testing a “real” OPC UA server, it might be brought up differently.
Step Definitions
The necessary step definitions for our scenarios are spread over multiple TypeScript source files grouped by topic, for example:
- stepdefs_common.ts for basics like creating and connecting the OPA UA client, to control the simulated reader and the hooks to initialize and to shut down the World class instance
- stepdefs_server_events.ts handles the subscription and monitored item part as well as the checks performed on the received RfidScanEventType events
- stepdef_server_methods.ts implements the steps for calling the ScanStart and ScanStop methods and verifying their return status and output arguments.
See Github for full details.
Running The Tests
While Cucumber-JS can be entirely configured using command line arguments, it also supports reading a configuration file. We chose this approach to keep our test run scripts in the package.json as a simple as possible. Our file instructs Cucumber-JS to create a HTML report as well as pretty printed console output and tells it where to find the TypeScript source files containing our environment setup, World class and step definitions.
{
"default": {
"formatOptions": {
"snippetInterface": "synchronous"
},
"format": [
["html", "reports/report.html"],
"summary",
"@cucumber/pretty-formatter",
"cucumber-console-formatter"
],
"requireModule": ["ts-node/register"],
"require": ["env/*.ts", "world/*.ts", "features/step_definitions/*.ts"]
}
} The tests are added as scripts in the package.json file. The test script runs the tests while check executes a dry run without really running any test steps to check if there are any missing step definitions.
"scripts": {
"test": "npx cucumber-js",
"check": "npx cucumber-js --dry-run"
} The following images shows an excerpt of the HTML report we generated.
Conclusion
With some initial investment of time and resources in agreeing on the domain language, extracting the necessary test steps and implementing a World class with its associated step definitions, we have built a powerful test setup where the test coverage can be extended without the requirement for programming skills.
Separating the test description from the test implementation leads to more expressive tests which are easier to maintain and understand in the long run. It also enables closer collaboration with stake holders.
