Chapter 10. Logging and analysing

10.1 Web-based tools

The web interface provides three tools for analysing CD1.1 module log files. To access them, select “CD1.1 log analyser” from the main menu. The following screen appears:

The Sender or Receiver entries will be missing if no instances have been configured. The menu will automatically extend itself if additional instances of any module are created. The three tools are considered in detail below.

10.1.1 Multiplexer log analysis tool

The multiplexor log analysis tool displays a pictorial representation of the subframes stored in the subframe database followed by a day's worth of log file entries, grouped by hour, along with the number of entries for each hour.

Missing frames are indicated with a red cross (). Frames which are present are indicated by green squares ().

The display shows the previous ten minutes of data. If you wish to examine a different time period, click the button and fill in the resulting form:

Select the desired start time (in UTC, regardless of your local time zone) and the desired display period, then click the button to generate a new display. When finished, click the button.

Below the pictorial representation is the database overview, which is grouped by hour. The number of entries should normally be constant: any discrepancy should be investigated.

Clicking any “View” link takes you to a detailed "hour view" screen for the associated hour.

The "hour view" screen displays a table, where the first column shows the time-stamp for each subframe processed and the second column gives the number of channels processed for that time-stamp.

If a gap has been detected, this is clearly indicated, in red, in the first column. Subsequent columns are labelled with CD1.1 channel names and are populated either with the word “missing”, if no subframe was received for the given channel with the given time-stamp, or with a quadrant status indicator, as shown in the following diagram. Each quadrant of the circle is labelled with a mnemonic identifier and will be either red or green, depending on the status of the associated subsystem. Green is used to indicate a satisfactory status and red indicates some cause for concern.

• The top left quadrant, S, displays the signature status: It is green if the subframe is signed and red if unsigned.

• The bottom left quadrant, G, shows the GNSS lock status: It is green if locked and red if unlocked.

• The bottom right quadrant, T, represents the timing status: It is green if the clock differential is acceptable and red if it has exceeded the configured threshold.

• The top right quadrant, M, depicts “miscellaneous” status: it is green if everything is satisfactory but turns red if any channel, security or misc. status bit is set.

The quadrant indicator also serves as a link to the subframe decoder, where the entire contents of the subframe are displayed parsed into fields, along with the interpretation of the contents. The top of such a display is shown below:

10.1.2 Receiver log analysis tool

The CD1.1 receiver log analysis tool can be used to examine the performance of the CD1.1 receiver.

To access it, select “CD1.1 log analyser” from the main menu. The following screen appears:

The Sender or Receiver entries will be hidden if no instances have been configured. The menu will automatically extend itself if additional instances of any module are created. Clicking the link for the desired receiver instance opens the receiver log analysis tool.

The receiver log analysis tool displays a pictorial representation of the subframes stored in the subframe database followed by a day's worth of log file entries, grouped by hour, along with the number of entries for each hour.

Missing frames are indicated with a red cross (). Frames which are present are indicated by green squares ().

The display shows the previous ten minutes. To examine a different time period, scroll down, click and fill in the resulting form:

Select the desired start time (in UTC, regardless of your local time zone) and the desired display period, then click the button to generate a new display. When finished, click .

Below the pictorial representation is the list of connected clients:

A link will be displayed for each connected client, along with the number of gaps currently being tracked. Clicking on a client link produces a screen like this:

There will be an entry in the table for each contiguous set of frames (i.e. there will be one more populated row than there are reported gaps). As indicated in the display, the frame identified by the “End sequence number” will not yet have been received. In the example above, frames between 194314 and 196342 (inclusive) have not yet been received.

10.1.3 Sender log analysis tool

The CD1.1 sender log analysis tool can be used to query the unacknowledged frame database, the back-fill database and the transmission log. To access it, select “CD1.1 log analyser” from the main menu. The following screen appears:

The Sender or Receiver entries will be missing if no instances have been configured. The menu will automatically extend itself if additional instances of any module are created.

Clicking the link for the desired sender instance produces a screen like this:

The first bulleted line shows the number of entries in the unacknowledged frame database and also serves as a link to a summary view. Note that, while it is not possible to edit the unacknowledged frame database via the web interface, a command line tool, cd11-framedb-tool, provides this functionality if required. See section 10.2.2 for further details.

The summary view of the unacknowledged frame database looks like this:

The second bulleted line shows the number of entries in the back-fill database, i.e. the number of frames which have been requested to be transmitted as back-fill but which not yet been sent. The number also serves as a link to a summary view. Note that, while it is not possible to edit the back-fill database via the web interface, a command line tool, cd11-backfilldb-tool, provides this functionality if required. See section 10.2.1 for further details.

The summary view of the back-fill database looks like this:

The two database summaries are followed by the transmission log search tool and a list of all transmission log files: it is on these files that the search tool operates.

Clicking the search tool link (“scan for transmission of a particular frame”) produces the following screen:

This screen allows the operator to enter a time datum with which to search the transmission log files. Note that the time datum is compared with the time-stamp on the transmitted frames, not the log-file entry time-stamps.

Entering, for example, 2017-296T15:01:00 and clicking might give the following display:

All matching log-file entries are displayed along with sequence number, authentication and channel details. Clicking on the link in the “Log entry” column displays the complete log file.

The complete log file can also be displayed by clicking on any of the links under “Available log files” on the front page of the CD1.1 sender log analysis tool.

A typical log file looks like this (the first few entries from the top, middle and end of an example log-file are shown):

10.2 Command-line tools

The following command-line tools are available for analysing and managing CD1.1 log files and database files:

• cd11-backfilldb-tool - used to summarise, dump, clear, partially clear or add entries to the back-fill database used by the CD1.1 sender, data-out-cd11.

• cd11-framedb-tool - used to summarise, dump, clear or partially clear the frame database maintained by the CD1.1 sender, data-out-cd11.

• cd11-timedb-tool - used to summarise or display headers from the frames database maintained by the CD1.1 multiplexer, data-mux-cd11.

• cd11-frame-analyser-tool - used to dump and decode a CD1.1 frame (or multiple frames), either as they are received or selected by time-stamp.

• cd11-management-tool - used to change the logging level of any of the CD1.1 modules without restarting them, and/or to change the authentication key ID.

All of these tools have detailed usage messages which are accessible by invoking the tool with, as an argument, either -h or --help.

10.2.1 cd11-backfilldb-tool

The CD1.1 sender, data-out-cd11, maintains a database of un-transmitted or unacknowledged frames, which it uses to perform back-fill. In normal operation with good network links, this database is quite likely to be empty.

The command-line program cd11-backfilldb-tool allows the listing, addition or removal of frames for which back-fill is to be performed.

The back-fill database file for the first sender instance is normally /var/lib/data-out-cd11.0/backfilldb - other instances use similar path-names with the “0” incremented for each instance. The tool should be invoked with the path to the back-fill database as the last argument.

If no other arguments are given, the tool reports the most recent frame time-stamp and the number of gaps (contiguous sequences of missing frames) in the database:

eam999 # cd11-backfilldb-tool /var/lib/data-out-cd11.0/backfilldb

Most recent timestamp : 2009-12-06T23:40:40Z

Number of missing ranges: 2

This is the same behaviour as when the -s (or --summary) argument is given.

Giving the -d (or --detail) argument displays a table showing the start and end time-stamps for each gap:

eam999 # cd11-backfilldb-tool -d /var/lib/data-out-cd11.0/backfilldb

Most recent timestamp: 2009-12-06T23:40:40Z

Gap start | Gap end

--------------------------+----------------------------

2009-12-06T21:25:20Z 2009-12-06T21:32:10Z

2009-12-06T21:42:40Z 2009-12-06T22:53:50Z

EOF

Using the -c (or --clear-before) argument with a date (in the same format as those produced on output) removes all entries for gaps before the given date. No further attempt will be made to send these frames unless they are specifically requested by the DC with an acknack packet.

Using the -C (or --clear-all) argument removes all entries from the back-fill database. This ensures that no back-fill takes place when the module is started (this does not apply to retransmission requests issued by the DC via the acknack mechanism). This could be used to prune older data, ensuring only fresh data are back-filled after a long outage.

Using the -a (or --add) argument with two dates (in the same format as those produced on output) separated by only a solidus (“/”) adds a spurious gap to the database:

eam999 # cd11-backfilldb-tool -a 2009-12-01T12:00:00Z/ 2009-12-01T12:59:50Z /var/lib/data-out-cd11.0/backfilldb

No output is produced on successful database modification but the new database entry can be seen using the -d command:

eam999 # cd11-backfilldb-tool -d /var/lib/data-out-cd11.0/backfilldb

Most recent timestamp: 2009-12-06T23:40:40Z

Gap start | Gap end

--------------------------+----------------------------

2009-12-01T12:00:00Z 2009-12-01T12:59:50Z

2009-12-06T21:25:20Z 2009-12-06T21:32:10Z

2009-12-06T21:42:40Z 2009-12-06T22:53:50Z

EOF

Full usage details for this program can be produced by passing it the -h (or --help) flag:

Usage:

cd11-backfilldb-tool [options] /path/to/backfilldb

Valid options:

-h, --help Display this screen.

-V, --version Display version number.

-s, --summary Display summary of backfilldb.

-d, --detail Dump details (entries) of

backfilldb.

-c, --clear-before <date> Clear entries before <date> from

backfilldb.

-C, --clear-all Clear all entries from

backfilldb.

-a, --add <date>/<date> Add an entry between two dates.

The default action is to display a summary of the database. This tool should only be used to modify the database when data-out-cd11 is not running. Actions are executed in reverse order to the commandline.

10.2.2 cd11-framedb-tool

The CD1.1 sender, data-out-cd11, maintains a database of transmitted frames, mapping its sequence numbers to their channels and time-stamps. Where multiple senders transmit a common subset of frames, one frame can have different sequence numbers, when seen from the points of view of the different senders. If a Data Consumer (DC) requests retransmission of a packet, it will do so by sequence number but the multiplexer stores frames by channel and time-stamp. The frame database allows the sender to receive the retransmission request from the DC by sequence number and, subsequently, to pass the request on to the multiplexer by channel and time-stamp.

The command-line program cd11-framedb-tool allows the listing or removal of frames from this database.

The frame database file for the first sender instance is normally /var/lib/data-out-cd11.0/framedb - other instances use similar path-names with the “0” incremented for each instance. The tool should be invoked with the path to the frame database as the last argument.

If no other arguments are given, the tool provides a brief summary of the contents of the database, as shown below.

eam999# cd11-framedb-tool /var/lib/data-out-cd11.0/framedb

Most recent sequence number : 38295

First unacknowledged sequence number: 38293

frame date: 2009-12-06T23:40:20Z

Last unacknowledged sequence number: 38295

frame date: 2009-12-06T23:40:40Z

eam999#

The behaviour is identical when using the -s (or --summary) argument.

A more detailed analysis of the contents of the database can be gleaned by giving the -d (or --detail) argument:

eam999 # cd11-framedb-tool -d /var/lib/data-out-cd11.0/framedb

Last sequence number transmitted: 38295

Sequence | Datestamp

---------------------+------------------------------

38293 | 2009-12-06T23:40:20Z

38294 | 2009-12-06T23:40:30Z

38295 | 2009-12-06T23:40:40Z

EOF

eam999 #

Using the -c (or --clear-before) argument with a date (in the same format as those produced on output) removes all entries before the given date. These frames will be considered to have been acknowledged by the DC.

Using the -C (or --clear-all) argument removes all entries from the frame database.

Full usage details for this program can be produced by passing it the -h (or --help) flag:

Usage:

cd11-framedb-tool [options] /path/to/framedb

Valid options:

-h, --help Display this screen.

-V, --version Display version number.

-s, --summary Display summary of framedb.

-d, --detail Dump details (entries) of

framedb.

-c, --clear-before <seq> Clear entries before <seq> from

framedb.

-C, --clear-all Clear all entries from framedb.

The default action is to display a summary of the database. This tool should only be used to modify the database when data-out-cd11 is not running. Actions are executed in reverse order to the commandline.

10.2.3 cd11-timedb-tool

The CD1.1 multiplexer, data-mux-cd11, records every frame it sends, as it sends it, in a daily database file, called the timedb. The actual database files have names like 2009-348, where 2009 is the year and 348 is the day number within that year, as specified in ISO8601.

Each instance of the multiplexer has a configurable directory in which these files are stored (see Database directory in Section 5.2.2). The default directory for the first multiplexer instance is /var/lib/data-mux-cd11.0/ where the 0 varies with instance.

The cd11-timedb-tool command-line tool allows this database to be queried. The path to a database file should be given as an argument to the tool. Used without further options, the tool produces a summary of the entries in the database file, listed hour by hour, with the number of frames stored for each hour, as below. The output is shown truncated here:

eam999# cd11-timedb-tool /var/lib/data-mux-cd11.0/2009-348

timedb version 0 file

00:00:00--00:59:59 : 360 frames

01:00:00--01:59:59 : 360 frames

04:00:00--04:59:59 : 360 frames

05:00:00--05:59:59 : 360 frames

...…

21:00:00--21:59:59 : 360 frames

22:00:00--22:59:59 : 360 frames

eam999#

Additional information can be displayed by using the -v (or --verbose) argument. The summary display, as above, is produced and then, for each time-stamp within the daily database file, the channel, subframe size and transmission status are given:

eam999# cd11-timedb-tool -v /var/lib/data-mux-cd11.0/2009-348

timedb version 0 file

00:00:00--00:59:59 : 360 frames

01:00:00--01:59:59 : 360 frames

02:00:00--02:59:59 : 360 frames

......

21:00:00--21:59:59 : 360 frames

22:00:00--22:59:59 : 360 frames

23:00:00--23:59:59 : 360 frames

-------- 00:00:00

3K55.HHZ. (600 bytes, already transmitted)

3K55.HHN. (600 bytes, already transmitted)

3K55.HHE. (600 bytes, already transmitted)

-------- 00:00:10

3K55.HHE. (600 bytes, already transmitted)

3K55.HHZ. (600 bytes, already transmitted)

3K55.HHN. (600 bytes, already transmitted)

-------- 00:00:20

3K55.HHE. (600 bytes, already transmitted)

......

-------- 23:59:40

3K55.HHZ. (600 bytes, already transmitted)

3K55.HHN. (600 bytes, already transmitted)

3K55.HHE. (600 bytes, already transmitted)

-------- 23:59:50

3K55.HHZ. (600 bytes, already transmitted)

3K55.HHN. (600 bytes, already transmitted)

3K55.HHE. (600 bytes, already transmitted)

eam999#

Full usage details for this program can be produced by passing it the -h (or --help) flag:

Usage:

cd11-timedb-tool [options] file

Options:

-h, --help Display this screen.

-V, --version Display version number.

-v, --verbose Display details of timedb.

Default if no options are specified is to display a summary of the database file. Pass --verbose to see subframe headers.

10.2.4 cd11-frame-analyser-tool

The command-line tool cd11-frame-analyser-tool can be run to receive and decode sets of subframes being transmitted from the multiplexor. The output from this program is suitable for sending to a script in order to monitor the status bits, etc.

The program requires the -s (or --socket) option to be given with, as an argument, the path to the IPC socket being used by the desired multiplexor instance. For the first instance, this path is

/var/run/data-mux-cd11.0

and subsequent instances have similar paths, with the final 0 being incremented for each instance.

Run with no further arguments, the tool displays a summary of the first frame to be assembled by the multiplexer after the command is invoked. This usually involves a delay of ten seconds or more between invocation and output.

The summary looks like this:

eam999 # cd11-frame-analyser-tool -s /var/run/data-mux-cd11.0

Frame at 2009355 15:51:00.000: 3 channels, 10000ms duration

Channel 3K55.HHZ.

Channel 3K55.HHN.

Channel 3K55.HHE.

eam999 #

More output is produced if the -v (or --verbose) option is given. The previous display is produced and this is followed by a display of the subframe flags for each channel.

eam999 # cd11-frame-analyser-tool -v -s /var/run/data-mux-cd11.0

Frame at 2009355 16:03:30.000: 3 channels, 10000ms duration

Channel 3K55.HHZ.

Channel 3K55.HHN.

Channel 3K55.HHE.

Subframe for channel 3K55.HHZ. (at 2009355 16:03:30.000, duration 10000ms)

subframe_length=600 auth_offset=596

auth_flag=0 trans_type=1 sensor_type=1 option_flag=1

cal_factor=3.000 cal_period=0.500 num_samples=800

status_format=1 status_len=32

status_data=0x00 status_security=0x00 status_misc=07 status_voltage=03

last_gps_sync=1989321 00:00:00.000 clock_differential=-2147483647us

auth_key_id=0 auth_size=0

Subframe for channel 3K55.HHN. (at 2009355 16:03:30.000, duration 10000ms)

subframe_length=600 auth_offset=596

auth_flag=0 trans_type=1 sensor_type=3 option_flag=1

cal_factor=1.000 cal_period=1.000 num_samples=800

status_format=1 status_len=32

status_data=0x00 status_security=0x00 status_misc=07 status_voltage=03

last_gps_sync=1989321 00:00:00.000 clock_differential=-2147483647us

auth_key_id=0 auth_size=0

Subframe for channel 3K55.HHE. (at 2009355 16:03:30.000, duration 10000ms)

subframe_length=600 auth_offset=596

auth_flag=0 trans_type=1 sensor_type=1 option_flag=1

cal_factor=1.000 cal_period=1.000 num_samples=800

status_format=1 status_len=32

status_data=0x00 status_security=0x00 status_misc=07 status_voltage=03

last_gps_sync=1989321 00:00:00.000 clock_differential=-2147483647us

auth_key_id=0 auth_size=0

eam999 #

If the -c (or --continuous) option is given, the tool will not exit after displaying the first frame: rather, it will continue to decode frames as they are received until interrupted by a +.

Full usage details for this program can be produced by passing it the -h (or --help) flag:

Usage:

cd11-frame-analyser-tool [options] [timestamp]

Valid options:

-h, --help Display this usage screen.

-V, --version Display version number.

-s, --socket <path> Path to multiplexor socket.

-v, --verbose Display verbose details, not just a
summary.

-c, --continuous Don't exit after first frame
received.

Default if no timestamp is specified is to dump the next real-time frame being transferred. Otherwise, a backfill request for the timestamp (which must be in ISO8601 format) is made, and the result displayed.

The filter string, if specified, is in the format command,STATION,CHANNEL,LOC where command is accept or reject. The string must be specified in quotes so that it is only a single argument.

10.2.5 cd11-management-tool

The CD1.1 management tool can be used to change the logging level and the authentication options in use by a module while it is still running. This allows these options to be changed without interrupting the subframe generation process.

The management tool is accessed from the command-line as

cd11-management-tool path-to-configuration-file(s)

As shown, the tool expects to be passed the path to one or more configuration files for running module instances.

The default configuration file locations for each module are:

If passed the -a (or --all) option, it will apply the changes to all configured instances of all CD1.1 modules. If passed the -C (or --save-config) option, it will first update the configuration file(s) with the new settings, so that if the instance is restarted the new settings will be remembered.

It will then use IPC to communicate the the new settings to the running instance(s), which take effect immediately. The configuration file must contain the path to the management socket for the instance (the default configuration tool always provides this).

Note that the management tool is not able to turn authentication on/off while the module is running. It can only change the Spyrus slot and the authentication ID field.

• To change the logging level, pass the -l (or --log-level) flag with an argument of LOG_DEBUG, LOG_INFO, LOG_NOTICE, LOG_WARNING, LOG_ERR, LOG_CRIT, LOG_ALERT or LOG_EMERG. These arguments correspond to the standard Linux syslog logging levels.

• To change the authentication key slot and/or ID, pass the -k (or --auth-key-id) flag with an argument of slot,ID

Full usage details for this program can be produced by passing it the -h (or --help) flag:

Usage:

cd11-management-tool [options] [config-files]

Valid options:

-h, --help Display this screen.

-V, --version Display version number.

-a, --all Apply changes to all instances.

-C, --save-config Save changes in config files as well
-l, --log-level <LVL> Change log level (LOG_DEBUG etc.).

-k, --auth-key-id <slot>,<id>

Change authentication key id.

You must either list the configuration files to scan/change, or pass the --all option to scan all files. Without the --save-config option, the changes will not be remembered when the daemon is restarted; with --save-config, the configuration files will be updated to cope with the new value as well.

Valid log levels are LOG_DEBUG, LOG_INFO, LOG_NOTICE, LOG_WARNING, LOG_ERR, LOG_CRIT, LOG_ALERT, LOG_EMERG. It is unsafe to switch authentication on/off, you should only change the slot in use and the authentication ID.