Skip to main content
Version: 0.8.0

Tracing

This article has examples in the following target languages:

Tracing is a powerful tool when it comes to analysis and debugging of applications. Unfortunately, most tracing tools that are readily available are designed specifically for analyzing processes, threads and system calls. Specialized tools are required to enable analysis that is tailored to an alternative model of computation such as Reactors. The tools should be capable of understanding the fundamental concepts of the model, such as the distinction between logical and physical time, as well as structural units such as reactors and reactions. This page gives an overview of the currently supported trace mechanism, as well as an outline of alternative tools that could be useful in the future.

Tracing is different from logging. Logging produces human-readable output in textual form and incurs significant overhead. Tracing produces binary data that must be further processed to be useful and is designed to have minimal impact on the execution time of a program.

Tracing is currently supported in the C, Python, and C++ targets. The mechanism used in C and Python is different from that used in C++. Tracing in C++ requires third-party tools that may only be available in Linux. Tracing in C and Python does not require any third-party tools.

Tracing in C++​

Tracing in the C++ target of Lingua Franca is based on three third-party tools. LTTng is a Linux tool used to instrument the Lingua Franca program and to record traces in the CTF, which minimizes the overhead of instrumentation. Chrome (or Chromium) has a build in trace viewer that is used to visualize the recorded trace data in a reactor-specific way. Since the Chrome trace-viewer cannot read CTF traces directly, we use Babeltrace2 to convert the recorded CTF trace to a JSON file that the Google trace viewer can load.

Usage​

Some helper scripts that we will use below, can be found in the reactor-cpp repository.

  1. Build and install the user space tools of LTTng (lttng-ust) as described here. On Arch, there is a community package available pacman -Sy lttng-ust. On Ubuntu, you need to install lttng-tools, lttng-modules-dkms, and liblttng-ust-dev

  2. Build and install Babeltrace2 and its python bindings as described here. In most cases, the following steps should work:

    1. git clone --branch v2.0.4 git@github.com:efficios/babeltrace.git
    2. cd babeltrace
    3. ./bootstrap
    4. ./configure --prefix=/path/to/preferred/install/location --enable-python-bindings --disable-debug-info --disable-man-pages
    5. make install
  3. Make sure babeltrace is available on your path:

    1. export PYTHONPATH=${PYTHONPATH}:/path/to/preferred/install/location/lib/python3.8/site-packages
    2. export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/preferred/install/location/lib
  4. Modify the target declaration of your Lingua Franca program to enable tracing:

target Cpp { tracing: true };
  1. Build the Lingua Franca program. The current build process is not very robust and does not automatically rebuild the reactor-cpp framework with tracing support if an earlier build exist. Be sure to remove all build artifacts rm -r bin build include lib share src-gen before triggering a new build. Alternatively, if you compile with lfc, you can add -c to the command to clean before building.
  2. Start a LTTng user space session by simply running the start_tracing.sh script. This will print the directory in which the recorded traces will be placed.
  3. Run your instrumented Lingua Franca application.
  4. Stop the LTTng session using stop_tracing.sh.
  5. Convert the recorded CTF trace to a JSON file using ctf_to_json.py <lttng-session-dir>. <lttng-session-dir> is the output directory reported by start_tracing.sh. By default, this produces a file trace.json. Optionally, the default output file can be overridden using -o or --output.
  6. Open Chrome (or Chromium) and go to about://tracing. Load the previously generated JSON file to visualize it.

The Trace View​

Screenshot_20200512_165849

The trace visualization consists of two parts. The upper part (labeled Execution), shows the physical time at which reactions are executed by the scheduler or by its worker threads. The lower parts show the reactors of the program. For each reactor, all scheduled actions (red markers) and all triggered reactions (blue markers) are visualized by their logical time. All elements in the trace view can be clicked on to display more detailed information.

Supporting Tracing in Other Targets​

The same mechanism as described above can be used to trace Lingua Franca applications in other target languages. The key is to instrument the target runtime in order to produce CTF traces. LTTng comes with support for multiple languages. As a fallback solution, C can be used to define the trace points which then can be used from the target language through a foreign function interface. It should also be considered, to use an alternative library in the target language that is capable of producing CTF traces. The only requirement is that the generated CTF events have a similar structure, as it is currently used in the C++ target. See trace.hh in reactor-cpp to get an overview of the available trace points.

Trace Viewers​

This section gives a brief overview of trace viewers that could be applicable for tracing Lingua Franca applications.

Google Trace Viewer​

The Google Trace Viewer is the only viewer currently supported. Since it reads JSON files, it is easy to use and a conversion script can easily tailor the trace data such that it is correctly displayed by the viewer. Documentation of the JSON trace format can be found here. There is also a list of available color codes. The approach of using LTTng for tracing, a converter and Google Trace Viewer can also be used to sample and visualize data live. This is shown in the Scalapus project.

Trace Compass​

Trace Compass is based on Eclipse and has native support for CTF traces. Due to the Eclipse support, Trace Compass is a natural candidate for integration with the Lingua Franca IDE. However, Trace Compass is tailored for the visualization of kernel traces or running user processes. Out of the box, it cannot make sense of reactor traces. There are various ways to customize Trace Compass in order to properly display the data, but they are difficult to use and/or not well documented.

  1. Custom trace analysis and views can be defined in an XML format. This probably works well for smaller tasks, but programming a mildly complex analysis in XML without proper error checking and debugging appears ridiculous.
  2. Custom analysis and views can also be programmed in scripting languages such as Python or Java Script. However, the tools seem not to be ready for production. I was not able to get the Python support working. Using Java Script the tools worked for simple tasks, but I was not able to come to a satisfactory results due to lack of or incomplete documentation and low customizability.
  3. Writing a Trace Compass Plugin. This seems to be the most promising approach, but probably requires a considerable amount of work. Writing such a plugin and integrating it with the Lingua Franca IDE could be a nice student project though.

Vampir​

Vampir is another powerful tracing tool that is mainly developed at TU Dresden. It targets mostly HPC applications, and I am not sure if it can be adjusted to display specific information other than details of processes and threads.