Plotting logged data

Enabling logging

Most of the software components in telekyb3 are able to produce data for post-flight analysis. This logging is usually activated via a service named log (for instance, check the rotorcraft log service documentation). The logging can usually be stopped by a service called log_stop.

Log files format

Log files produced by the log services are text files that contain one record per line. Each record contains multiple data points, separated by white space and sampled simultaneously.

Records beginning with # are comments that contain generic information about the log file or the context in which it was recorded.

Records with no characters other than blanks and a newline designate discontinuities in the log; it means that at least one record could not be logged and was skipped.

The first entry of a log file that is not a comment and not a blank record is a text field, and represent the key title of the corresponding data point found in the rest of the file.

Data points are most of the time numeric. They can also be the string nan, which corresponds to an invalid value or the character -, which corresponds to a missing value (or a value not changed since the previous data point).

Most of the time, records contain a ts column which represents the sampling time of the record in seconds since the Unix epoch. This is convenient to plot synchronized data coming from different log files and components.

Plotting data with gnuplot

The log files are directly compatible with the gnuplot plotting tool. In particular, comments are properly ignored, column titles are recognized and blank records are plotted without a line joining them to indicate possible missing data. The nan values are ignored and the treatment - special value can be configured properly.

This section cannot fully describe the gnuplot tool: check the documentation for details. However, simple examples are provided to get you started.

For instance, plotting the 'imu_az' column from a log file from rotorcraft can be as simple as:

$ gnuplot
gnuplot> plot 'rotorcraft.log' using 'imu_az' with lines
Sample plot

More advanced usage include plotting multiple columns, adjusting the scale and adding some legend. Note how the columns are retrieved by name, making the plot robust to future column order changes in the log files.

gnuplot> set xlabel 'sample #'
gnuplot> set ylabel 'acceleration in m/s^2'
gnuplot> plot [][8.5:11] 'rotorcraft.log' using 'raw_az' with dots title 'raw imu a_z', 'rotorcraft.log' using 'imu_az' with lines title 'filtered imu a_z'

Or with the shorter syntax:

gnuplot> plot [][8.5:11] 'rotorcraft.log' u 'raw_az' w d t 'raw imu a_z', '' u 'imu_az' w l t 'filtered imu a_z'
Sample plot

Plotting versus time is usually more interesting than plotting versus the sample number, especially when plotting multiple log files with different sampling rates. As already mentionned, log files normally contain a ts column that represents the sample acquisition time in seconds since January 1st, 1970. As this number of seconds is difficult to interpret, it is usually better to plot against the time in seconds since the beginning of the log. This can be achieved by noting the first timestamp of a log file by visual inspection, and using a variable in gnuplot to compute the difference. This example sets the t0 variable to the first sample time and uses the column('name') expression to retrieve a column by name inside a matematical expression:

$ more rotorcraft.log
$ # note the first timestamp
$ gnuplot
gnuplot> set xlabel 'time in s'
gnuplot> set ylabel 'acceleration in m/s^2'
gnuplot> t0=1670524455.187710000
gnuplot> plot 'rotorcraft.log' u (column('ts')-t0):'imu_az' w l t 'imu a_z'
Sample plot

As a final remark, it’s worth noting that gnuplot can use a special file in your $HOME directory called .gnuplot. This file can be useful to define your most often used commands or shortcuts (check the gnuplot "Substitution and Command line macros" documentation). For instance, since it can be cumbersome to first get the origin timestamp from multiple log files and then plot data, the `$HOME/.gnuplot file can be used to define a macro doing the job for you:

# gnuplot startup file
set datafile missing '-'
set macros
set grid

t0 = `awk 2>/dev/null                                                   \
  'BEGIN { ts = systime() }                                             \
  FNR == 1 { col = 0 }                                                  \
  col && col < NF { if (ts > $col) ts = $col; nextfile }                \
  /^[^#]/ { for (i=1;i<=NF;i++) if ($i == "ts") { col=i; next } }       \
  END { print ts }' *.log || echo 0`

ts = "(column('ts')-t0)"

Line 3 defines the special string - in input data files to denote a missing data entry. Gnuplot makes a distinction between missing data and invalid data (e.g. "NaN"). For example invalid data causes a gap in a line drawn through sequential data points; missing data does not.

Lines 4-5 define common settings that are usually wanted.

Lines 7-11 define a variable t0 whose content is the result of an awk script. This script simply checks all files ending in .log in the current working directory and extracts the minimum timestamp value found in them, so that t0 eventually represents the first timestamp among all log files. This script is of course not very robust and could easily be trapped. But it can still be helpful in basic scenarios.

Finally, line 13 defines a macro that substracts t0 from the current value of the column ts, as in the previous example.

This sample .gnuplot file can be copied to your $HOME and tuned to match your needs. With it, you can simply redo the previous plot like this, without even manually checking the log file:

gnuplot> plot 'rotorcraft.log' u @ts:'imu_az' w l t 'imu a_z'