Python for embedded systems testing
Python for embedded systems testing. When starting a project, there is this important question: How will we test our product’s software?
Nowadays, we have all the technology required for automated testing.
If there is an entry cost to fully automate testing of products and a maintenance cost, it enables the use of continuous integration (jenkins …) and will reap a lot of benefits: faster bug detection leading to faster bug resolution (the sooner the bugs are found during the development cycle, the least changes there is to make in order to fix it), quicker releases with more confidence in the product.
Here I present tools helping tests development. I will make the case for the Python language, then dive into some popular testing framework helping test infrastructure before talking about embedded software testing.
Are you looking for a company to assist you in the test of your embedded systems?
ELSYS has design centers in Europe (France), Eastern Europe (Serbia) and a branch in the USA (California, in the Silicon Valley).
Contents:
I am an embedded software developer with 6 years of python experience behind me, and I have been using Python for embedded software testing for 3 years.
Why Python?
Why not continue doing integration tests in C or C++ or even C# ? Before talking about tests, the Python language itself has some features that will help the developers in many ways:
Readibility and maintainability
The syntax rules of Python allow you to express concepts without writing additional code. At the same time Python, unlike other programming languages, emphasizes on code readability, and allows you to use English keywords instead of punctuations. The readable and clean code base will help you to maintain and update the tests without putting extra time and effort.
Industry proven and open source libraries
Many Industry actors support and use Python, attesting to its usefulness and robustness, find the list of company sponsoring Python here. In addition, Python has a wide community contributing to a large catalog of open source libraries, many of which are endorsed by the industry.
Multiple Programming paradigms
Like other modern programming languages, Python supports object-oriented and structured programming fully. Also, its language features support various concepts in functional and aspect-oriented programming. At the same time, Python also features a dynamic type system and automatic memory management. The programming paradigms and language features help you to use Python for easy developing of arbitrary large and complex software applications.
Multipurpose
Today python is used for graphical interfaces, web developments, AI computing, data visualization, system administration, scientific and numeric applications… Using Python for testing also enables developpers to tackle a wider range of applications with ease.
If you want more details on why use Python, click here.
Note that here I’m talking about the CPython implementation (here is a talk about various Python implementations)
Python is not almighty, it also comes with trade-offs:
- Python is slower than compiled code like C/C++. In the majority of uses covered here, that does not really matter, and it can be tweaked and optimized (performance tips)
- In a single instance of Python, threads do not work in parallel as one might think. If you want real concurrency you have to use multiprocessing (more info).
Testing frameworks
That’s it, Python has convinced you. Now you want to start developing tests. You’re maybe familiar with Google Test or Cppunit for unitary testing in C/C++ but what would you use in Python ? Here I present two frameworks with different philosophies, although I am biased toward Pytest since I’ve been using it for 3 years now.
Pytest
“The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries” (read more). Pytest’s tests are python functions using fixtures and using asserts.
Pytest is a very flexible framework allowing for a wide range of tests such as unitary, functional, integration …
Here is a test example:
It features:
- Detailed info on failing assert statements, making debugging easier.
- Auto-discovery of test modules and functions
- Modular fixtures for managing small or parametrized long-lived test resources
- Rich plugin architecture, with over 315+ external plugins and thriving community
Among the plugins:
- pytest-html generates html reports of test execution for an easy analysis of the same quality as Robot Framework.
- pytest-bdd allows the execution of tests written using gherkin, a high level acceptance testing language, similar to Robot Framework
Robot Framework
“Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA).” (read more)
Robot Framework was designed mainly for acceptance testing, where a client (even internal) “writes” tests.
A robot test is a high level formatted text using keywords. Keywords can be defined using other keywords, and at the bottom keywords are implemented in Python libraries. Here is an example:
It features :
- High-Level Architecture
- Simple Tabular Syntax
- Data-driven Test Cases
- Separate Test Data Editor
- Clear Reports
- Detailed logs
- Generic test libraries
Pytest vs Robot
They provide the same core functionalities : being able to organize and run tests and interface with a Continuous Integration tool.
One thing stands out though, Robot framework is Python with a High level keyword driven layer on top, and I find this higher level not practical. Let met explain.
From a management point of view, it seems appreciable, but how many managers read tests ? A simple description is sufficient to understand the tests enough.
From a test driven perspective, one could express a test in plain English, then have a tester develop the test using the same words. But would just reading the high level keywords validate the test? The core of the test would still be in python, and to fully understand a test, one would have to resolve all keywords to their python equivalent.
My opinion is that adding a high level layer add unneeded complexity, that might be manageable only for a small subset of tests.
This is where Pytest shines, being a bit more simple, a test is also more readable to a developer and the tools provided by Pytest make it very versatile.
To give it to Robot, it has an efficient data organization with its own data format, but Pytest with add-ons can also be tweaked to resemble Robot Framework.
Testing embedded software
First I’ll covers libraries used to interface with a product using pure python solution, then we’ll interface Python with legacy DLL or C/C+ code.
Libraries to help test embedded software
These are open source (beware of licenses though) libraries:
- A serial interface is the most simple and popular way of communicating with a product. Pyserial makes implementing your own protocol easy.
- The CAN (Controller Area Network) bus is used in automotive and in a lot of places. Python-can enables to interface with various usb-to-can devices like Ixxat’s.
- On a CAN bus you may use the CANOpen protocol, the library canopen is there to take care of it.
- The HTTP like CoAP (Constrained Application Protocol) is an IoT targeted protocol to interconnect embedded targets. One implementation I’ve used is Tanganelli/CoAPthon3.
- The Modbus protocol is still used to this day, created for “programmable logic controllers (PLCs). Modbus has become a de facto standard communication protocol and is now a commonly available means of connecting industrial electronic devices.”(wikipedia). More info here.
- If you have a product using internet protocols you might want to use pyshark which is an interface for the command line wireshark “tshark” and allows live capture from an Ethernet interface or reading from pcap files or pipe.
- Working with ZigBee or other wireless protocols in python is made possible with killerbee which provides drivers for radio components and cryptography tools, which uses Scapy to decode/encode packets.
Interfacing with existing software
Why not reuse existing software to test your embedded device ? Whether you are using dlls you no longer have source code for or C/C++ code, there is a solution for you.
Using existing DLL
- ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.
- SWIG (Simplified Wrapper and Interface Generator) is able to generate CPython (C++ code with Python API dependencies) code to wrap dlls. It uses an intermediate language to be able to generate code for custom types the dll uses. From my experience, its use is fairly advanced, but it yields good results. ctypes is simpler but more limited.
- Writing CPython code. Sometimes it could be easier just to write the wrapping code yourself if the interface is small enough or because there will be no more change to the interface.
Using existing code
I’ve used cffi to wrap a dll but not cppyy, I list it for completion.
- cffi C Foreign Function Interface for Python is able to compile C at runtime (everything in Python is runtime anyway) and provides an interface to manipulate C objects in python. It can also be used to interface with dll.
- cppyy “is an automatic, run-time, Python-C++ bindings generator, for calling C++ from Python and Python from C++.“
What’s next?
I hope you have enjoyed reading this article!
We, at ELSYS Design, specialize in embedded systems, from design to test phases, with an expertise in Python. Would you have a project in this area, don’t hesitate to contact us so that we can discuss about it in more details.