Limitations, Design, and Other Projects

Current Limitations

Note

Contributions to address these or other shortcomings are more than welcome! There are also various TODO: comments on smaller concrete issues throughout the repository – approaching any of those will benefit the library a lot!

Pull requests are the way to go. If you are unsure about anything, feel free to open an issue. See this project’s README for details on how to get started.

  • If a callback (e.g. of a subscriber in the node) raises an exception the test does not fail automatically with the exception as the reason, as that is currently not supported in ROS2. It will probably still fail because some expected message is not detected by the test. In those cases, you will have to look for messages like The following exception was never retrieved: [...] in the stderr output of pytest. It will probably be mixed in with other messages if you view it on the console.

  • Similar “silent” crashes can occur in other places, including timers and services. See tests/test_failing_nodes.py for unit tests on that behavior. A lot of them are marked with a TODO:, since it is currently not straightforward to detect such issues.

  • A failing service might deadlock a test. Consider calling services asynchronously with timeouts, e.g. with call_service() or await_future().

  • It takes some time to set up the test environment each time, particularly when using @with_launch_file. Also, some nodes or complex launch scenarios might need considerable time to process information. You may wish to append --durations=0 --durations-min=1.0 to your pytest call to show the slowest tests (more info). It might be possible to reduce the required warmup_time, since it is unclear why setting it too low breaks all message exchanges and maybe there is a solvable bug causing it.

Design Considerations

These were (mostly) the initial design considerations. Over time, this may shift.

Goals

  • Provide a method for writing node tests that can assert correct behavior via sending messages and observing message output.

  • Allow for concise tests. Little to no infrastructure code should be required for typical scenarios.

  • Automate common checks where possible (e.g. nodes should not crash).

  • Support testing single and multiple nodes in combination (i.e. support Unit tests (of single nodes), Integration tests, and System tests as per ROS2 terminology).

  • Be well-documented and easy to get started.

  • Support the most widely used platforms (that have not reached end-of-life yet). Be pragmatic.

Constraints

  • Be easy to maintain by needing only a few lines of code (below a thousand). We only use public APIs wherever possible.

  • Don’t reinvent the wheel and benefit from future improvements: Use existing functionality of Python and ROS2. This includes: unittest, pytest, and the ROS2 launch system.

  • Efficiency is not a primary concern as this mini-framework is intended to only be used for testing and not in a real robot deployment, where performance is much more of a concern.

  • Must work with colcon test (see here) and also with just pytest alone (for simpler IDE integration).

Comparison to Other Projects

Several other projects are attempting to make testing of ROS2 systems easier. However:

  • With launch_pytest, it is not easy to interact with the system, e.g. using publishers and subscribers.

  • The same holds for launch_testing.

As a side note, this functionality could not have been implemented with pytest fixtures, since we need to be able to receive different parameters for each test.