Skip to content

Embedded Software Testing with Zephyr RTOS

Introduction

Testing with Zephyr RTOS

In this codelab, we will first understand the basic principles of software testing and, in particular, embedded software testing. We will then experiment with testing by running existing Zephyr RTOS tests with the provided twister test runner. Finally, we will write some simple test applications using the ztest and integrate testing into our “BikeComputer” program.

What you’ll build

  • You will run existing tests and collect test results in various formats using the twister test runner.
  • You will understand how to build your own test programs with the ztest test framework and you will write a few tests, including tests with host integration.

What you’ll learn

  • The basic principles of embedded software testing.
  • How to run tests and check test results with twister.
  • How to write your own test programs with the ztest framework.

What you’ll need

  • Zephyr Development Environment for developing and debugging your program in C++.
  • Zephyr Development Environment for building and running the tests.
  • A good understanding of the Zephyr RTOS library architecture and principles.

Embedded Software Testing

Software has become an integral part of many systems, including embedded systems. Each of us interacts with more and more software every day, and solutions are evolving faster and faster. This means incorporating new features as well as improving product stability and security.

For embedded systems, update delivery solutions are often difficult to implement and the pace at which software updates are delivered is much slower than for other systems such as computers or smartphones. However, given the nature of embedded systems, which often operate for long periods of time, software quality is critical and ways of delivering software without compromising quality need to be developed. Automated testing is an important part of solutions aimed at improving software quality. Building, testing and deploying software with many components quickly becomes impossible. In addition, manual testing involves repetitive work and costs time that is not invested in software development. Automating everything - from build to test, deployment and infrastructure - is the only way forward. This is also true for embedded software.

In this context, CI/CD, which stands for Continuous Integration/Continuous Delivery, is a methodology that helps developers deliver applications frequently by introducing automation into the development process. When working in development groups with many developers, continuous integration helps developers merge their code changes back into a shared branch on a regular basis (often daily). Once developers’ changes have been merged, they need to be validated. This is done by automatically building the application and running various levels of automated tests, typically unit and integration tests. If the automated tests find any bugs, they are reported back to the developers so that they can be fixed before a new version of the application is shipped.

When automated builds and tests are successful in CI, continuous delivery allows developers to automate the release of this validated code. From this perspective, CD can only happen in a CI/CD environment where CI is already built into the development pipeline. At the end of the CI/CD process, the application is ready to be deployed into production.

In this codelab we will demonstrate how to run and develop automated tests for your Zephyr RTOS components and application. We will show how to test individual components of a library or application. Tests that integrate multiple components will also be developed.

Motivations for Building an Automated Test Environment

Like any other product used by customers, software needs to be tested before it is delivered to users. Of course, the easiest and most obvious way to test a product is to use it for a while and make sure it behaves as expected. Since the developer knows the application and therefore knows how to quickly test the changes made to it, you might think that this is a reliable way of testing software. Of course, this is usually wrong:

  • Developers are biased towards the parts of the software they know best and the changes they have made. It is easy to forget to test some parts, or not realise the impact of a change.
  • The environment in which the software runs may be different from the environment of the machine on which it was developed and tested. Very often the environment has an impact on the way a particular piece of software can run.
  • Last but not least, testing is a boring and time-consuming task. Very often, developers will minimise the time they spend on testing. In some situations, people are hired to perform testing activities, but the problem remains.

Even in situations where developers and testers are two different groups of people, the disadvantage is that developers are no longer involved in testing and may lose sight of the big picture. On the other hand, testers have little knowledge of the changes that have been made and have to bother the developers whenever they find something they don’t understand.

Modern software development also requires the ability to change code quickly and safely. From this perspective, automated testing plays a key role, together with a clearly defined testing strategy that includes different types of tests. In general, tests range from unit tests, which focus on technical details, to acceptance tests, which show that the application meets its objectives. More details on the different types of tests are given in the next section. So tests can be different, but good tests tend to share the same characteristics:

  • A good test is deterministic: it doesn’t matter where and when it runs, it should always produce the same outputs given the same inputs. It must also be verifiable. Of course, this sometimes makes the task of writing tests difficult.
  • A good test is fully-automated: because it must be repeated endlessly, and because machines are good at repetitive tasks, tests must be performed as automated tasks by a machine.
  • A good test is responsive: it must provide fast and direct feedback. Integrating test feedback into the development process is essential and must be done quickly and efficiently.

You should also be aware that

  • Testing does not slow down development: in the long run, the time spent writing tests is an investment that allows changes to the software to be made in an efficient and robust way.
  • Testing is not just about finding bugs: finding bugs is an important purpose, but making bugs easily detectable and fixable after each and every change is even more important. This gives developers a safety net that makes it easier to make changes to the software. Without a safety net, developers would only make very conservative changes, while some less conservative changes may be needed.

The Test Pyramid

The concept of the test pyramid was introduced by Mike Cohn in his book [“Succeeding with Agile”] (https://www.mountaingoatsoftware.com/books/succeeding-with-agile-software-development-using-scrum){target="blank”}. The concept is particularly applicable to applications with a user interface, but can easily be adapted to embedded software.

In Mike Cohn’s original concept, the test pyramid consists of three layers that any test suite should consist of (image taken from Test Pyramid in Spring Boot Microservice):

  1. User Interface or System Tests
  2. Service or Integration Tests
  3. Unit Tests

Test Pyramid

Test Pyramid

When applied to embedded systems, it is not possible to adopt this concept strictly, either in the number of layers or in the naming of the layers. However, because of its simplicity, the essence of the test pyramid is a good basis on which to build an embedded software test suite. In particular, you should try to

  1. Write tests with different granularities
  2. Adapt the number of tests to their level: low-level tests require many tests while fewer tests are needed for high-level tests.

In other words, develop a test suite with lots of small and fast unit tests. You should also write a reasonable number of integration tests and very few system tests that test your application from end to end.

Test Specification in Zephyr OS

In Zephyr RTOS, tests are specified using yaml specification files named “testcase.yaml” or “sample.yaml”.

Start Testing

The first thing you need to do when setting up a testing environment and strategy for your application is to start testing the existing source components. What does that mean on the Zephyr RTOS platform?

The Zephyr RTOS platform provides several tools to assist in the development of an appropriate test suite:

  • Unit tests can be developed and built using a separate build system. Unit tests are built and run on the host/development machine and have no hardware or software dependencies.
  • ztest is the test framework used to write test programs and is provided as part of the Zephyr RTOS platform. With this framework, developers can write functional unit tests in C++, but also integration tests that can implement complex use cases that run on microcontrollers.
  • twister is the test runner provided as part of the Zephyr RTOS platform. With twister, you can build and run test programs on physical boards or emulated environments.

Discovering and Running Existing Tests

This section explains how to use twister to build and run test programs that are part of the Zephyr RTOS library, both on physical boards and in emulated environments. The use of twister is extensively documented here.

The first initial steps for building and running tests are the following:

  • You can list all tests that are available for a specific platform with
    terminal
    west twister --list-tests -T deps -p nrf5340dk/nrf5340/cpuapp
    assuming that all dependencies have been installed in the “deps” folder.
  • You can limit the tests to list or run by specifying a tag value. For example, if you want to limit the list of tests to those tagged with cpp or kernel, you can add -t cpp or -t kernel to the command. If you limit the tests to cpp, you should see that some zpp_lib tests are listed
    console
    ...
 - zpp_lib.zpp_rtos.zpp_mutex.mutex_lock_unlock
 - zpp_lib.zpp_rtos.zpp_semaphore.semaphore_release_acquire
...
  • You can then select one of the tests and compile it for the chosen platform by running, for example,
    terminal
    west twister --build-only -T deps/zpp_lib/zpp_rtos -p nrf5340dk/nrf5340/cpuapp
    Unless the selected test is not supported for the target board, the test program should compile.

When you run the above commands, you will see that all the output files generated by twister are stored in a folder called “twister-out”. The folder name is automatically renamed to “twister-out.X” (e.g. “twister-out.1”) if another folder already exists. You can choose to write the results to a specific folder by adding the -O OUTDIR option to any twister command, in which case the “OUTDIR” folder will be cleaned before the test is executed.

Running Tests on Physical Device

west twister -p nrf5340dk/nrf5340/cpuapp -T bike_computer\tests\bike_computer –device-testing –hardware-map nr5340_map.yaml

For running tests on a physical device, you may choose of the following options:

  • Run the test on a single device with the command
    terminal
    west twister -T deps/zpp_lib/zpp_rtos -p nrf5340dk/nrf5340/cpuapp --device-testing --device-serial "COM_PORT"
    where “COM_PORT” is the serial port to which your device is connected.

  • Run the test on a single or several devices by first building a device map:

    • Run the following command:

      terminal
      west twister --generate-hardware-map my_map.yaml
      This will probe the serial devices connected to your computer and generate a hardware map file such as:
      my_map.yaml
      - connected: true
  id: 0038001F3431511037393330
  platform: unknown
  product: unknown
  runner: unknown
  serial: COM5
- connected: true
  id: null
  platform: unknown
  product: unknown
  runner: unknown
  serial: COM6
- connected: true
  id: null
  platform: unknown
  product: unknown
  runner: unknown
  serial: COM7
      You need to update the content of the file with the missing information, such as:
      my_map.yaml
      - connected: true
  id: 0038001F3431511037393330
  platform: nrf5340dk/nrf5340/cpuapp
  product: J-Link
  runner: nrfutil
  serial: COM5
      If you have more than one device connected to your computer, you will need to update the file for each specific entry. Also note that you may need to update the file to match connection status and ports as you connect and reconnect devices to your computer.

    • Once the hardware map file has been updated correctly, you may run

      terminal
      west twister -p nrf5340dk/nrf5340/cpuapp -T deps\zpp_lib --device-testing --hardware-map my_map.yaml
      for executing one partical test on one or several devices connected to your computer.

Running Tests on Emulated Device

With twister and Zephyr RTOS it is possible to run test programs on emulated devices. A popular emulator family is QEMU, which allows Zephyr RTOS programs to be emulated on various machine emulators.

For building and running programs on such emulators, you first must install [QEMU]{https://www.qemu.org/}{target="blank”} by following [instructions]{https://www.qemu.org/download}{target="blank”} for your development platform. Once the required tools have been installed, you may run blinky on a chosen QEMU platform as follows:

  • Copy the “boards/qemu_x86.overlay” file to the “blinky/boards” folder.
  • Add CONFIG_GPIO_EMUL=y to the prj.conf file located in the “blinky” folder.
  • Run west build -b qemu_x86
  • Run west build -t run west build -b qemu_x86 deps\zpp_lib\zpp_rtos\tests\thread –pristine –extra-dtc-overlay D:/aes/boards/qemu_x86.overlay west twister -p qemu_x86 -T deps/zpp_lib/zpp_rtos/tests –integration

Running Tests on native Simulator

It is also possible to run tests using the native simulator on Linux platforms. The native_sim board is based on the POSIX architecture. A Zephyr OS application can be compiled on this board with the Zephyr kernel and libraries to produce a Linux executable.

To run the same tests as above using the native_sim board, you simply need to adapt the command as follows:

(.venv) D:\aes>west twister -p nrf5340dk/nrf5340/cpuapp -T bike_computer/tests/bike_computer/speedometer –device-testing –hardware-map nr5340_map.yaml

Integrating Tests in CI

Developing your Own Tests

Using twister to Build and Run Tests

(.venv) D:\aes>west twister -p nrf5340dk/nrf5340/cpuapp -T bike_computer/tests/bike_computer/speedometer –device-testing –hardware-map nr5340_map.yaml

Once these software are installed, you may start a new Mbed project and import the blinky example. You may then run the following commands in a cmd prompt or shell of your choice, at the root directory of your project: “mbed test -t GCC_ARM -m DISCO_H747I –compile-list –greentea”. This command lists all Greentea tests available for the DISCO_H747I device. You should get a long list of available tests like shown below:

> mbed test -t GCC_ARM -m DISCO_H747I --compile-list --greentea
[mbed] Working path "D:\Mbed Programs\bike-computer" (library)
[mbed] Program path "D:\Mbed Programs\bike-computer"
Test Case:
        Name: mbed-os-connectivity-mbedtls-tests-tests-mbedtls-multi
        Path: .\mbed-os\connectivity\mbedtls\tests\TESTS\mbedtls\multi
Test Case:
        Name: mbed-os-connectivity-mbedtls-tests-tests-mbedtls-sanity
        Path: .\mbed-os\connectivity\mbedtls\tests\TESTS\mbedtls\sanity
Test Case:
        Name: mbed-os-connectivity-mbedtls-tests-tests-mbedtls-selftest
        Path: .\mbed-os\connectivity\mbedtls\tests\TESTS\mbedtls\selftest
Test Case:
        Name: mbed-os-drivers-tests-tests-mbed_drivers-buffered_serial
        Path: .\mbed-os\drivers\tests\TESTS\mbed_drivers\buffered_serial
Test Case:
        Name: mbed-os-drivers-tests-tests-mbed_drivers-c_strings
        Path: .\mbed-os\drivers\tests\TESTS\mbed_drivers\c_strings
...

For each test, the test name is displayed together with its location in the Mbed OS library tree structure. If you filter the output for all tests containing “timer” in their name, you should get the following output:

> mbed test -t GCC_ARM -m DISCO_H747I --compile-list --greentea | grep timer
        Name: mbed-os-drivers-tests-tests-mbed_drivers-lp_timer
        Path: .\mbed-os\drivers\tests\TESTS\mbed_drivers\lp_timer
        Name: mbed-os-drivers-tests-tests-mbed_drivers-timer
        Path: .\mbed-os\drivers\tests\TESTS\mbed_drivers\timer
        Name: mbed-os-drivers-tests-tests-mbed_drivers-timerevent
        Path: .\mbed-os\drivers\tests\TESTS\mbed_drivers\timerevent
        Name: mbed-os-rtos-tests-tests-mbed_rtos-systimer
        Path: .\mbed-os\rtos\tests\TESTS\mbed_rtos\systimer

Building and Running a Test

For building and running tests, you may run the same “mbed test” command with other parameters. Before doing this, it is important to understand what actions are executed when building and running tests:

  1. A new folder “.\BUILD\tests\“target_name”\“toolchain_name”” is created, if necessary. This directory is used for copying all required files and for the build process. Its name is specific to the target device and to the toolchain used for testing.
  2. The non-test source code (all code not under a TESTS folder) is copied to the BUILD directory and compiled. The resulting object files are placed in the BUILD directory.
  3. The test code is copied to the BUILD directory and compiled, including the main.cpp file containing the main function to be called for executing the test. Both the test and non-test object files are linked to create the executable file used for running the test. This step is performed for each test discovered (based on the command parameters).
  4. Based on the test specification file “test_spec.json” that is also copied to the BUILD directory, the tests are run one at a time.

It is important to note the following points regarding the build process:

  • The “mbed_app.json” file defined at the root of your application is also used for building the test program. This configuration file is used for building both the non-test code and each test case.
  • The “.mbedignore” file defined at the root of your application is also used. Make sure that all required Mbed OS files are not excluded from the build, in particular all features related to tests located in the “mbed-os/features/frameworks” folder (like “mbed-os/features/frameworks/greentea-client” or “mbed-os/features/frameworks/unity”.
  • Since your application probably also contains a “main.cpp” file with a main() function, it is required that you modify your “main.cpp” application file as shown below. When building a test program, the MBED_TEST_MODE compilation symbol will be defined and this will prevent the presence of two main() symbols at link time.
main.cpp
...
#if !MBED_TEST_MODE
int main() {
    // Application code
}
#endif
...

If you read carefully the information above and made the required changes in your application, you should be ready for building and running your first test. If you enter the following command: “mbed test -t GCC_ARM -m DISCO_H747I -n mbed-os-drivers-tests-tests-mbed_drivers-timer –compile –run”, the test named “mbed-os-drivers-tests-tests-mbed_drivers-timer” defined in the “.\mbed-os\drivers\tests\TESTS\mbed_drivers\timer” folder will be built and run. The test should succeed for all cases defined in the test program and you should get test results similar to

> mbed test -t GCC_ARM -m DISCO_H747I -n mbed-os-drivers-tests-tests-mbed_drivers-timer --compile --run
...

mbedgt: test suite report:
| target              | platform_name | test suite                                     | result | elapsed_time (sec) | copy_method |
|---------------------|---------------|------------------------------------------------|--------|--------------------|-------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | OK     | 18.62              | default     |
mbedgt: test suite results: 1 OK
mbedgt: test case report:
| target              | platform_name | test suite                                     | test case                                                      | passed | failed | result | elapsed_time (sec) |
|---------------------|---------------|------------------------------------------------|----------------------------------------------------------------|--------|--------|--------|--------------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on os ticker) - measured time accumulation. | 1      | 0      | OK     | 1.14               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on os ticker) - reset.                      | 1      | 0      | OK     | 0.01               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on os ticker) - start started timer.        | 1      | 0      | OK     | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on os ticker) is stopped after creation.    | 1      | 0      | OK     | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on user ticker) - reset.                    | 1      | 0      | OK     | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on user ticker) - start started timer.      | 1      | 0      | OK     | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on user ticker) is stopped after creation.  | 1      | 0      | OK     | 0.01               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer (based on user ticker) measured time accumulation. | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - copying 5 ms.                                    | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - moving 5 ms.                                     | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - time measurement 1 ms.                           | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - time measurement 1 s.                            | 1      | 0      | OK     | 1.01               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - time measurement 10 ms.                          | 1      | 0      | OK     | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-drivers-tests-tests-mbed_drivers-timer | Test: Timer - time measurement 100 ms.                         | 1      | 0      | OK     | 0.08               |
mbedgt: test case results: 14 OK
mbedgt: completed in 19.25 sec

As you can observe, all test cases have succeeded for this particular test. You may also generate the test results in a more readable format by using the command: “mbed test -t GCC_ARM -m DISCO_H747I -n mbed-os-drivers-tests-tests-mbed_drivers-timer –run –report-html test_result.html”. If you then open the “result.html” file, your browser should display a window like the one below, in which you can open a detailed view of the test results.

Note that for exporting the test results in a specific format, you first need to compile the tests with the “–compile” option and then to run them with the “–run” option, in two separate steps. The command combining compilation and result exports does not work properly, for an undocumented reason.

HTML test report

HTML test report

Demonstrating a Test Failure Case

For illustrating a scenario where not all test cases succeed, we may choose the sys_timer test by running the “mbed test -t GCC_ARM -m DISCO_H747I -n mbed-os-rtos-tests-tests-mbed_rtos-systimer” command. If you run this test on your DISCO_H747I target device, then you should observe that one of the test cases fails:

> mbed test -t GCC_ARM -m DISCO_H747I -n mbed-os-rtos-tests-tests-mbed_rtos-systimer --compile --run
...
mbedgt: :340::FAIL: Deep sleep should be allowed
mbedgt: retry mbedhtrun 1/1
mbedgt: ['mbedhtrun', '-m', 'DISCO_H747I', '-p', 'COM17:115200', '-f', '"BUILD/tests/DISCO_H747I/GCC_ARM/mbed-os/rtos/tests/TESTS/mbed_rtos/systimer/systimer.bin"', '-e', '"mbed-os\\rtos\\tests\\TESTS\\host_tests"', '-d', 'G:', '-c', 'default', '-t', '08140221013F69703E7BF059', '-r', 'default', '-C', '4', '--sync', '5', '-P', '60'] failed after 1 count
mbedgt: checking for GCOV data...
mbedgt: test on hardware with target id: 08140221013F69703E7BF059
mbedgt: test suite 'mbed-os-rtos-tests-tests-mbed_rtos-systimer' ..................................... FAIL in 13.28 sec
        test case: 'Handler called twice' ............................................................ OK in 0.08 sec
        test case: 'Tick can be cancelled' ........................................................... OK in 0.03 sec
        test case: 'Tick count is updated correctly' ................................................. OK in 0.00 sec
        test case: 'Tick count is zero upon creation' ................................................ OK in 0.01 sec
        test case: 'Time is updated correctly' ....................................................... OK in 0.00 sec
        test case: 'Wake up from deep sleep' ......................................................... FAIL in 0.02 sec
        test case: 'Wake up from sleep' .............................................................. OK in 0.03 sec
mbedgt: test case summary: 6 passes, 1 failure
mbedgt: all tests finished!
mbedgt: shuffle seed: 0.8403045412
mbedgt: test suite report:
| target              | platform_name | test suite                                  | result | elapsed_time (sec) | copy_method |
|---------------------|---------------|---------------------------------------------|--------|--------------------|-------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | FAIL   | 13.28              | default     |
mbedgt: test suite results: 1 FAIL
mbedgt: test case report:
| target              | platform_name | test suite                                  | test case                        | passed | failed | result | elapsed_time (sec) |
|---------------------|---------------|---------------------------------------------|----------------------------------|--------|--------|--------|--------------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Handler called twice             | 1      | 0      | OK     | 0.08               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Tick can be cancelled            | 1      | 0      | OK     | 0.03               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Tick count is updated correctly  | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Tick count is zero upon creation | 1      | 0      | OK     | 0.01               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Time is updated correctly        | 1      | 0      | OK     | 0.0                |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Wake up from deep sleep          | 0      | 1      | FAIL   | 0.02               |
| DISCO_H747I-GCC_ARM | DISCO_H747I   | mbed-os-rtos-tests-tests-mbed_rtos-systimer | Wake up from sleep               | 1      | 0      | OK     | 0.03               |
mbedgt: test case results: 6 OK / 1 FAIL
mbedgt: completed in 13.94 sec
mbedgt: exited with code 1```

In this test, the “Wakeup from deep sleep” case fails. As you can read from the test result log, the error :340::FAIL: Deep sleep should be allowed is reported. By analyzing the source code of the test program at line 340, one can find the exact failure cause: the sleep manager does not allow our target device to enter deep sleep state. Understanding this error and possibly correcting the failure cause is beyond the scope of this codelab.

Writing Your Own Test Programs

The framework that we experimented in the previous section allows us to write our own test programs. Developers can write test programs by using the Greentea client Greentea client and the unity and utest frameworks, which are all located in the “mbed-os/features/frameworks” folder of the Mbed OS library.

By convention, all tests are placed under a directory named “TESTS” and under two other directories: a test group directory and a test case directory, as illustrated below

Test directory structure

Test directory structure

This convention allows the Mbed OS test tools to discover all tests available from the root directory of your application. The “TESTS” folder can be located anywhere in the tree structure of your application. For simplicity, we will create a “TESTS” folder at the root director0y of the application.

Write A Simple Test Program That Always Succeeds

Our first test program will be very simple and it will allow us to discover the most important features of the test frameworks. For developing this test program, you must first create the folder structure and create a “simple-test” test group, an “always-succeed” test case and “main.cpp” file. You may verify that your tests is correctly discovered by the the mbed tools:

> mbed test -t GCC_ARM -m DISCO_H747I --compile-list | grep always
        Name: tests-simple-test-always-succeed
        Path: .\TESTS\simple-test\always-succeed

The always-succeed test can be implemented as follows in the main.cpp file:

TESTS/simple-test/always-succeed/main.cpp
// Copyright 2022 Haute école d'ingénierie et d'architecture de Fribourg
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/****************************************************************************
 * @file main.cpp
 * @author Serge Ayer <serge.ayer@hefr.ch>
 *
 * @brief Simple example of test program that always succeeds
 *
 * @date 2022-09-01
 * @version 0.1.0
 ***************************************************************************/

#include "greentea-client/test_env.h"
#include "mbed.h"
#include "unity/unity.h"
#include "utest/utest.h"

using namespace utest::v1;

// test handler function
static control_t always_succeed(const size_t call_count) {
    // this is the always succeed test
    TEST_ASSERT_EQUAL(4, 2 * 2);

    // execute the test only once and move to the next one, without waiting
    return CaseNext;
}

static utest::v1::status_t greentea_setup(const size_t number_of_cases) {
    // Here, we specify the timeout (60s) and the host test (a built-in host test or the
    // name of our Python file)
    GREENTEA_SETUP(60, "default_auto");

    return greentea_test_setup_handler(number_of_cases);
}

// List of test cases in this file
static Case cases[] = {Case("always succeed test", always_succeed)};

static Specification specification(greentea_setup, cases);

int main() { return !Harness::run(specification); }

This test program makes use of the following libraries:

  • utest: the Harness::run() calls allows to run a series of C++ test cases given a specification object. The specification object can be defined in many different ways. In our simple example, it is made of a pointer to a function that is called for setting up the greentea test framework and of an array of Case objects. Note that the type of the template arguments are automatically deduced for the Specification constructor.
  • The array of Case objects defines all test cases that must be run, one simple case in the example above. There are again many different ways of constructing Case objects. In this example, the case is constructed with a description and with a pointer to a function that must be called at test execution time. Note that this function returns a control_t variable, which defines the behavior of the test cases execution. In this example, the test handler function returns CaseNext, which means that this test case must be executed only once and that the next test case must be executed without delay.
  • Note that all mechanisms for building tests exist in different forms and offer a high level of flexibility. For example, all handler functions are available with different possible signatures for allowing different test scenarios.

If you run this test, you should observe a successful result:

> mbed test -t GCC_ARM -m DISCO_H747I -n tests-simple-test-always-succeed --compile --run
...
mbedgt: test suite report:
| target              | platform_name | test suite                       | result | elapsed_time (sec) | copy_method |
|---------------------|---------------|----------------------------------|--------|--------------------|-------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | tests-simple-test-always-succeed | OK     | 13.42              | default     |
mbedgt: test suite results: 1 OK
mbedgt: test case report:
| target              | platform_name | test suite                       | test case           | passed | failed | result | elapsed_time (sec) |
|---------------------|---------------|----------------------------------|---------------------|--------|--------|--------|--------------------|
| DISCO_H747I-GCC_ARM | DISCO_H747I   | tests-simple-test-always-succeed | always succeed test | 1      | 0      | OK     | 0.0                |
mbedgt: test case results: 1 OK
mbedgt: completed in 14.22 sec

From the simple “always-succeed example”, and although many configuration steps are omitted for this simple example, it is important to understand how a test is specified. In general, a test specification contains:

  • a setup handler (mandatory)
  • several test cases (mandatory) and
  • a teardown handler (omitted for always-succeed).

Each test case contains:

  • a textual description,
  • a setup handler (omitted for always-succeed),
  • a teardown handler (omitted for always-succeed),
  • a failure handler (omitted for always-succeed), as well as
  • the actual test handler (mandatory).

The order of handler execution is:

  1. Test setup handler,

  2. For each test case:

    1. Test case setup handler.
    2. Test case execution handler.
    3. Test case teardown handler.

  3. Test teardown handler.

In addition to the specification above, it is worth mentioning that test cases can be run asynchronously and can be repeated several times.

Test a basic C++ component

One important feature in the C++ language is the so-called smart pointers concept. As compared to raw pointers, smart pointers are intended to help ensure that programs free memory and resources whenever they are not referenced any more. This mechanism thus prevents memory and resource leaks. It is also exception safe, meaning that the expected behavior is also guaranteed in the case of an exception.

In this section, we demonstrate how the behavior of raw and smart pointers can be tested. Raw pointers just behave as addresses to memory locations and with raw pointers, it is the responsability of the programmer to deallocate any morey space allocated with a new or a malloc. On their side, smart pointers allow to encapsulate memory allocation within classes, so that memory is released whenever the object encapsulating the memory buffer is destroyed. Smart pointers are now implemented as part of the C++ standard using the std::unique_ptr or the std::shared_ptr classes.

std::unique_ptr and std::shared_ptr both implement mechanisms for a proper deallocation of the encapsulated resource. However, they differ in the way they handle the ownership of the resource: - With std::unique_ptr only one variable can refer to the object and the allocated resource will be reclaimed when that variable is destroyed. Transfering the ownership of the resource cannot be done with the assignment operator (operator=()) and instead one must use the std::move() semantics. - With std::shared_ptr, the ownership of the resource can be shared among several variables. A reference counting mechanism is implemented and when the last reference to the resource is destroyed, then it is reclaimed. Be aware that circular references are possible with std::shared_ptr.

For testing the behavior of both raw and smart pointers, we use a simple Test structure, with the following definition:

struct Test {
    Test() {
        _instanceCount++;
        _value = 33;
    }

    ~Test() {
        _instanceCount--;
        _value = 0;
    }

    int _value;
    static uint32_t _instanceCount;
};

This structure contains a static attribute that counts the number of instances that are still alive, by incrementing/decrementing upon construction/destruction. With this mechanism, we may test the following behaviors:

  • Test that when creating a shared pointer in a given scope, it will destroyed when leaving the scope.
  • Test that multiple instances of shared pointers correctly manage the reference count and that the object is released correctly.
  • Test that when creating a raw pointer and deallocating it correctly, the destructor is called.

The tests described above can be integrated in a test program under “TESTS/simple-test/ptr-test” as shown below. If you run the test, you should observe that all test cases run successfully.

TESTS/simple-test/test-ptr/main.cpp
// Copyright 2022 Haute école d'ingénierie et d'architecture de Fribourg
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/****************************************************************************
 * @file main.cpp
 * @author Serge Ayer <serge.ayer@hefr.ch>
 *
 * @brief Simple example of test program for raw and shared pointers
 *
 * @date 2022-09-01
 * @version 0.1.0
 ***************************************************************************/

#include "greentea-client/test_env.h"
#include "mbed.h"
#include "unity/unity.h"
#include "utest/utest.h"

using namespace utest::v1;
struct Test {
    Test() {
        _instanceCount++;
        _value = kMagicNumber;
    }

    ~Test() {
        _instanceCount--;
        _value = 0;
    }

    int _value;
    static constexpr uint32_t kMagicNumber = 33;
    static uint32_t _instanceCount;
};
uint32_t Test::_instanceCount = 0;

/**
 * Test that a shared pointer correctly manages the lifetime of the underlying raw pointer
 */
void test_single_sharedptr_lifetime() {
    // Sanity-check value of counter
    TEST_ASSERT_EQUAL(0, Test::_instanceCount);

    // Create and destroy shared pointer in given scope
    {
        std::shared_ptr<Test> shared_ptr(new Test);
        TEST_ASSERT_EQUAL(1, Test::_instanceCount);
        TEST_ASSERT_EQUAL(Test::kMagicNumber, shared_ptr->_value);
    }

    // Destroy shared pointer
    TEST_ASSERT_EQUAL(0, Test::_instanceCount);
}

/**
 * Test that multiple instances of shared pointers correctly manage the reference count
 * to release the object at the correct point
 */
void test_instance_sharing() {
    std::shared_ptr<Test> shared_ptr1(nullptr);

    // Sanity-check value of counter
    TEST_ASSERT_EQUAL(0, Test::_instanceCount);

    // Create and destroy shared pointer in given scope
    {
        std::shared_ptr<Test> shared_ptr2(new Test);
        TEST_ASSERT_EQUAL(1, Test::_instanceCount);
        // share share_ptr2 with shared_ptr1
        shared_ptr1 = shared_ptr2;
        // still one instance only
        TEST_ASSERT_EQUAL(1, Test::_instanceCount);
        TEST_ASSERT_EQUAL(Test::kMagicNumber, shared_ptr1->_value);
        TEST_ASSERT(shared_ptr1.get() == shared_ptr2.get());
    }

    // shared_ptr1 still owns a raw pointer
    TEST_ASSERT_EQUAL(1, Test::_instanceCount);

    shared_ptr1 = nullptr;

    // Shared pointer has been destroyed
    TEST_ASSERT_EQUAL(0, Test::_instanceCount);
}

static utest::v1::status_t greentea_setup(const size_t number_of_cases) {
    // Here, we specify the timeout (60s) and the host test (a built-in host test or the
    // name of our Python file)
    GREENTEA_SETUP(60, "default_auto");
    return greentea_test_setup_handler(number_of_cases);
}

// List of test cases in this file
static Case cases[] = {
    Case("Test single shared pointer instance", test_single_sharedptr_lifetime),
    Case("Test instance sharing across multiple shared pointers", test_instance_sharing)};

static Specification specification(greentea_setup, cases);

int main() { return !Harness::run(specification); }

Exercice: Write a test program for unique_ptr

Exercice: Write a test program for raw pointers

Continuous Integration

Now that we have developed some automated test programs for Mbed OS, we may benefit from those test programs for automating both the build process and the test process. For doing so, we will use the facilities offered by Github and in particular GitHub Actions. Since you will be using Github for delivering your project, this makes the integration of actions even easier. We will also use a Docker image provided by Mbed OS for running builds within Github.

Automating the test process would require running the GreenTea tests against USB devices in the Docker Container. This is beyond the scope of this codelab since this would require a custom setup for running automated builds and tests. The goal is here only to demonstrate how you can automatically build your applications and how this could be extended to automate tests.

CI/CD Workflow

In the picture below (taken from GitLab CI/CD | GitLab, the typical development workflow is depicted. Our workflow will be simplified but it is useful to have an overview of the global picture:

  • Once changes have been made to a software under development, these changes can be pushed to a specific branch in a remote Gitlab repository. As we will experience later, this push triggers the CI/CD pipeline for your project.
  • The GitLab CI/CD usually runs automated scripts to build and test your application and then deploy the changes in a review application (different from the production application).
  • If all tests and deployment are successful, the code changes get reviewed and approved, a merge of the specific branch into the production branch is made and all changes are deployed to the production environment.
  • If something goes wrong, changes are rolled back or further changes are made for correcting the detected problems.

In our case, we will simplify the process and skip the branch/merge steps.

CI/CD process

CI/CD process

Integrate the Test Stage

In the previous step, we have developed a number of test programs. For automating test builds, we will now integrate those builds into the GitHub Actions. With GitHub Actions, one can automatically build, test, and deploy any application.

Using GitHub Actions is straightforward and this capability is already integrated into your Github account. For integrating your build into your repository, it is enough to create a new workflow in the Actions tab of your repository

Github actions

Github actions

Once you create the workflow, you may edit it to contain the actions described below. Note that in the notation below \{ and \} should be reproduced without the \.

.github/workflows/build-test.yml
name: Build test application
on:
  pull_request:
  push:

jobs:
  build-cli-v1:
    container:
      image: ghcr.io/armmbed/mbed-os-env:master-2022.05.21t04.23.55

    runs-on: ubuntu-20.04

    strategy:
      matrix:
        target: [K64F, DISCO_H747I]
        profile: [release, debug, develop]


    steps:
      -
        name: checkout
        uses: actions/checkout@v2

      -
        name: build-test
        run: |
          set -e
          mbed deploy
          mbed test -t GCC_ARM -m $\{\{ matrix.target \}\} --profile $\{\{ matrix.profile \}\} --compile -n tests-simple-test-always-succeed,tests-simple-test-ptr-test
          mbed compile -t GCC_ARM -m $\{\{ matrix.target \}\} --profile $\{\{ matrix.profile \}\}

A quick start to understand the basics of actions is given under Actions quickstart. For an understanding of this actions description, you need to understand the following:

  • The description is following the YAML specifications syntax. Yaml is somehow an extension of the JSON syntax for improved human readiness.
  • GitHub Actions workflow is triggered when an event occurs in the repository (specified with on). In the example above, the workflow is triggered when a push is done or when a pull request is opened.
  • Each workflow contains one or more jobs, which can run in sequential order or in parallel. Each job will run inside its own virtual machine runner or inside a container, and has one or more steps that either run a script that you define or run an action.

In the example above, we do the following

  • One job, named build-cli-v1, is run upon push or pull request.
  • The job is run inside a Docker container running the latest Mbed OS Docker image.
  • The strategy creates a build matrix for the job, that allows to define different variations to run the job. In our case, the variations are targeting different target devices and different build profiles. These variations are defined in the strategy matrix.
  • The job is made of two steps. The first step checks out your repository, so your workflow can access it. The second step builds the application.

  • The build step does the following:

  • run: | specifies a multi-line command

  • set -e instructs that the job will stop immediately if any command fails.
  • Further commands are standard git / mbed cli commands for installing library dependencies and compiling the test programs.

As you can read from the file above, you must push the test programs developed in the previous sections to your GitHub repository for a successful execution of the build-test script.

The first time the “build-test.yml” file is committed to your repository, the runner runs your jobs. The same applies for any further push to the repository. Whenever the runner runs a job, the job results are displayed in the actions tab.

Job progress

Job progress

The picture above shows the details of the Build test workflow. You can follow the workflow progress by choosing a specific run:

Job progress details

Job progress details

If you select a particular job, you can see all the details. If a job fails, you can also read the log for understanding the problem and fixing it.

Job details

Job details