Code Coverage for ROS

This is day 3 of my 2020 National Robotics Week blog marathon!

About two years ago I created a little package called code_coverage. This package is a bit of CMake which makes it easier to run coverage testing on your ROS packages. Initially it only supported C++, but recently it has been expanded to cover Python code as well.

What is Code Coverage?
Before I get into how to use the code_coverage package, let's discuss what coverage testing is all about. We all know it is important to have tests for your code so that it does not break as you implement new features and inevitably refactor code. Coverage testing tells you what parts of your code your tests actually test. This can help you find branch paths or even entire modules of the code that are not properly tested. It can also help you know if new code is actually getting tested.

The output of a coverage test is generally some really nice webpages that show you line-by-line what code is getting executed during the test:

Using code_coverage for C++
We will start by discussing the usage of code_coverage with C++ code first, because it is actually quite a bit simpler. C++ coverage can be done almost entirely in CMake.

First, update your package.xml to have a "test_depend" on "code_coverage" package.

Next, we need to update two places in the CMakeLists.txt file. The first change should be right after you call to catkin_package. The second change is where you define your test targets. You need to define a new target, which we will typically call {package_name}_coverage_report:

That's the configuration needed. Now we can compile the code (with coverage turned on) and run the coverage report (which in turn will run the tests):

catkin_make -DENABLE_COVERAGE_TESTING=ON -DCMAKE_BUILD_TYPE=Debug PACKAGE_NAME_coverage_report

You can find these same instructions (and how to use catkin tools) in the code_coverage README.

Using code_coverage for Python
Python unit tests will automatically get coverage turned on just with the CMake configuration shown above, but Python-based rostests (those that are launched in a launch file) need some extra configuration.

First, we need to turn on coverage testing in each node using the launch-prefix. You can decide on a node-by-node basis which nodes should actually generate coverage information:

Then we turn on coverage by adding the argument in our CMakeLists.txt:

add_rostest(example_rostest.test ARGS coverage:=ENABLE_COVERAGE_TESTING)

You can find this full Python example from my co-worker Survy Vaish on GitHub.

Using codecov.io For Visualization
codecov.io is a cloud-based solution for visualizing the output of your coverage testing. It can combine all of the reports from individual packages, as well as the C++ and Python reports into some nice graphs and track results over multiple commits:

codecov.io dashboard for robot_calibration

A Full Working Example
The robot_calibration package use code_coverage, codecov.io, and Travis-CI to run code coverage testing on every pull request and commit to master branch. It uses the popular industrial-ci package as the base line and then the following changes are made:

• I set the CMAKE_ARGS in the travis.yml so that coverage is turned on, and the build type is debug.
• I created a .coverage.sh script which runs as the AFTER_SCRIPT in Industrial-CI. This script runs the coverage report target and then calls the codecov.io bash uploader.
• Since Industrial-CI runs in a docker, I introduced a .codecov.sh script which exports the required environment variables into the docker. This uses the env script from codecov.io.