Guralp Systems Limited
MAN-SWA-0008 - Guralp Systems' Linux tools

PreviousNext

1. Preliminary Notes 2. Introduction 3. Installing the tools 4. Applications 5. Libraries 6. Configuration files 7. Revision history

Section Index: 4.1. data-gcf-dump 4.2. read-rawgcfdisk 4.3. gcf-to-miniseed 4.4. gcfgaps and msgaps 4.5. tcpserial 4.6. cd11-test-recv 4.7. block-device-tester 4.8. gsms-receiver

Chapter 4. Applications

4.1 data-gcf-dump

This tool is provided as part of the libdata-gcf library. Its purpose is to display a summary (or, optionally, the details) of a particular GCF file.

The input file can be any file (or device) in which GCF blocks are written sequentially at 1024 byte boundaries.

Usage:

data-gcf-dump [-d] file1.gcf [file2.gcf ...]

Without the -d flag, a single line summary is printed for each block that could be decoded from the file. With the -d flag, the block is also decoded; raw sample values are displayed in ASCII and the contents of status, unified status and strong motion blocks are printed.

4.2 read-rawgcfdisk

This tool displays the directory structure of a DFD (6TD/DM24 with IEEE1394 option) disk or a SAM disk.

Attach the disk to the system and find its device node, e.g. /dev/sdc

Note: if the disk happens to have a valid partition table, you must ensure you use the raw block device as shown and not a partition like /dev/sdc1

The disk directory may be displayed with the command:

read-rawgcfdisk /dev/sdc [/dev/sdd ...]

As shown, it is possible to display multiple directories with a single invocation.

It is also possible to use the tool on an image of a disk. To recover the directory information, only the first 9kiB (18 sectors) are required. However, this will cause the serial number, start time and end time to be omitted from the display, unless they are included in the file.

When displaying a DFD disk in which one of the transfers contains some empty blocks at the start of the lump, an exclamation mark “!” will be displayed before the start time. This is not indicative of a problem with the data.

4.3 gcf-to-miniseed

This is an offline data conversion tool which takes files of raw 1024-byte GCF blocks as input and writes out data-only SEED volumes (Mini-SEED records).

A configuration file is used to specify parameters for the conversion and, optionally, to map GCF channel names onto SEED names. The configuration file is covered in detail in section 6.1. A template configuration file may be created using the –config-template option, (which may be shortened to -T):

gcf-to-miniseed --config-template

The tool has two main modes of operation: auto-mapping mode, in which all usable data are extracted and SEED names are generated automatically, and manual mapped mode, in which the mappings must be specified in the configuration file and data from unknown channels are ignored.

To invoke the tool, use:

gcf-to-miniseed [-a] -c conf.cf input1.gcf [input2.gcf ...]

The -a or --auto-map option, if present, activates auto-mapping mode. The -c or --config option specifies the path to the configuration file. The remaining arguments are the GCF files to convert.

Once all files have been processed, a summary is printed which indicates how many blocks have been processed and a mapping of their input IDs to their output IDs. It is hoped that this will help identify if there are blocks in the data file which are not mapped in the config file.

4.3.1 The conversion process

The tool takes an arbitrary number of GCF files as inputs. The contents and order of the GCF files are irrelevant; the tool maps each file into memory, decodes all of the block headers, and then sorts the headers by channel and date. It then extracts the data using the order specified by the sorted headers, possibly interleaving reads across multiple input files.

The GCF files must consist of a set of GCF blocks residing at 1024-byte boundaries. The full specification of GCF (Güralp compressed format) can be downloaded from our website at www.guralp.com/howtos/gcf-format.shtml.

Output file naming (and the length of Mini-SEED files to output) is handled in the configuration file. It is possible to combine Mini-SEED records from different channels into the same file. Each output file may be split into time segments of between ten seconds and one day. GCF blocks which straddle time boundaries are split across two output files.

The behaviour when an invalid GCF block is encountered is specified within the configuration file. If the header is invalid, there is no way to decode the block, so it must be dropped. However, a data block may be decoded even in the presence of certain errors (either a coding error or a checksum error); the configuration file specifies what is to be done with such blocks. The options are to discard the data, use the data regardless, or use the data and set a SEED data quality header flag in the Mini-SEED record corresponding to the invalid block.

During conversion, logging output is written to STDOUT. Each log line begins with a level (INFO, WARNING or ERROR), followed by a file-name (or “global”), and then the message itself. It may be useful to capture and/or process this logging output within a wrapper script.

4.3.2 Auto-mapping mode

When the -a or --auto-map option is specified, GCF blocks with channels whose name is not mapped in the configuration file are treated specially. If the name is mapped, the mapping in the configuration file overrides the auto mapping option for this specific channel.

In auto-mapping mode, the following rules apply:

4.4 gcfgaps and msgaps

This is an offline tool for obtaining a summary of a file containing GCF data. It prints a summary of the channel names in a file with a count of how many GCF blocks per channel and whether the data are contiguous or not.

Usage:

gcfgaps list_of_filenames

where list_of_filenames is a space-separated list of files to process. A summary per file with a total and how many blocks and gaps is printed.

The gcfgaps package also provides the msgaps program. This is an offline tool for summarising the data in a Mini-SEED file.

Usage:

msgaps list_of_filenames

where list_of_filenames is a space-separated list of files to process. A summary per file and a list and count of gaps is printed.

4.5 tcpserial

A daemon which connects a TCP session to a serial port. This is intended to make serially-connected equipment at a remote site available elsewhere. The serial ports of the DCM, EAM, NAM, Affinity and *TDE ranges of equipment can be configured with one end of a tcpserial link, with the other end terminating in this daemon. The daemon can also talk to a Lantronix-style serial to Ethernet adapter (e.g. 6TD with Ethernet).

The daemon must be launched with a configuration file (see section 6.2). It does not background itself so, if launching from a shell, use an '&' character at the end of the line to background it. Alternatively, use a program like start-stop-daemon or daemonitor, both f which are available online from (www.lwithers.me.uk/usr/src/daemonitor/).

Note: the daemon exits on unrecoverable error, so running it in a looping script (or using daemonitor) is recommended.

Usage:

tcpserial -b baud -c config.cf -d /dev/ttyS0

The -b or --baud option must be specified. Standard baud rates are 1200, 4800, 9600, 19200, 38400, 57600, 115200 and 230400. If the hardware supports it, non-standard baud rates may also be used.

The -c or --config option must be specified. It gives the path to the configuration file, which specifies the operating mode of the converter (server/client) as well as the IP parameters to use.

The -d or --device option must be specified. It gives the path to the device node of the serial port to convert. The daemon must have read/write permissions on the device node.

4.5.1 Operating modes

The daemon has two major operating modes: simple client and simple server. Both of these modes just convert serial data and ignore the control lines (RTS, CTS, DSR, DTR and DCD) - hence “simple”. Other modes may be added on request.

In simple server mode, the daemon acts as a TCP server. Any incoming client connection is accepted. Only one client may be connected, so a newly-connecting client will cause any existing client to be disconnected (this is in case an old client connection is broken but the session has not timed out yet). When a client is connected, data read from the serial port are sent to the client, and data read from the client are sent to the serial port. If no client is connected, data read from the serial port are discarded.

In simple client mode, the daemon first establishes a connection to a remote server. Then data read from the server are written to the serial port and data read from the serial port are written to the server.

Simple server and simple client modes are complementary; the client is designed to connect to the server. Once this has been achieved, there is effectively a transparent pass-through between the two remote serial ports (although note that the control lines are ignored).

The operating mode, and its IP parameters, are specified in the configuration file (see section 6.2).

4.6 cd11-test-recv

This is a CD1.1 receiver which is intended for testing and analysis of a CD1.1 system or array. It does not store or decode the seismic data, but instead generates a log file suitable for analysing system operation.

The test receiver allows evaluation of the following:

It comes with several tools to analyse the log files generated by the receiver. These generate HTML reports or, using gnuplot, graphs.

4.6.1 Running the receiver

To run the receiver, a configuration file must be created. See section 6.3 for details of the configuration file.

Usage:

cd11-test-recv -c config_filename [-s]

If specified, the -s flag requests that log messages are printed to the screen (STDERR) rather than sent to syslog. It may be advantageous to run the receiver in a GNU screen session, perhaps under a tool such as daemonitor (http://www.lwithers.me.uk/usr/src/daemonitor).

If verification of frames is required, the configuration file points to a directory of certificates, CERT_DIR. Whenever a frame or subframe is received, the certificate is looked for at CERT_DIR/AUTH_KEY_ID.pem (where AUTH_KEY_ID is the base 10 integer representation of the authentication key ID), e.g. certs/1.pem or certs/57.pem. Once a certificate has been loaded, it will never be reloaded. A failure to load a certificate means the receiver will retry each time it receives another frame or subframe with a given auth key ID.

Once the receiver is running, it will generate log files (one for each incoming connection). It may be useful to periodically restart the receiver and flush the old log files as they may become unmanageably large over the course of several weeks, though a fast PC makes this less of an issue.

The log files that are generated are CSV (comma-separated value) files. Each row has the general form:

TIMESTAMP,EVENT_TYPE[,details]

The timestamp is the time of the event as recorded by the receiver. You should ensure the receiver PC's system clock is locked to NTP for valid timestamps.

An event type of BAD_FRAME or BAD_SUBFRAME indicates that a frame or subframe was received that could not be parsed. No further details will be given.

An event type of FRAME records the reception of a data frame; details will be:

SAMPLE_TIMESTAMP,DURATION_MS,AUTH_KEY_ID,SIG_OK

where SAMPLE_TIMESTAMP is the time of the first sample, DURATION_MS is the duration of the frame in milliseconds, AUTH_KEY_ID is the authentication key ID (should be 0 if authentication is off) and SIG_OK is a flag set to 1 if the frame's signature could be verified or 0 if not.

An event type of SUBFRAME records the reception of a subframe. Subframe records invariably come after frame records. The details are:

SAMPLE_TIMESTAMP,DURATION_MS,AUTH_KEY_ID,SIG_OK,CHAN_NAME,

NUM_SAMPLES,GPS_OK,TIMING_OK,MISC_OK,LAST_GPS_SYNC_TIMESTAMP,

CLOCK_DIFF

where:

4.6.2 Generating reports

The test receiver merely generates log files; there are tools to interpret the files and generate reports upon them. The first is cd11-test-logreport, which generates an XHTML report with tables summarising various aspects of the received data.

Usage:

cd11-test-logreport [-smgtTa] [-E date] [-L date] log.csv

The time range over which the report is generated may be limited by passing the -E and -L options (earliest and latest). These take ISO8601 date/times. For the end time, a loosely-specified date/time is taken to be the latest instant that could possibly apply; for example, passing -E 2010-05 and -L 2010-05 would generate a report for all data in May (2010-05-01T00:00:00Z to 2010-05-31T23:59:59.999999999Z).

Without any report flags, all reports are generated. Otherwise, reports are as follows:

4.6.3 Clock differential plots

The tool cd11-test-diffplot can be used to plot graphs of clock differential/drift over time.

Usage:

cd11-test-diffplot -o output.png -s date [-e date|-d sec]

The graph is generated by gnuplot, which must be installed on your system. The output filename is passed to the -o flag. The start of the plot must be specified to the -s flag. The end of the plot can either be specified as an absolute value to the -e flag, or as a relative value in seconds to the -d flag.

The name of a single log file is given, and the channel names to plot are then specified.

4.6.4 Latency/timeliness plots

The tool cd11-test-latencyplot can be used to plot graphs of data latency (time elapsed between first sample in frame and reception of that frame at the test receiver) over time.

Usage is identical to cd11-test-diffplot, except that in addition to specifying channel names, the keyword frame can be used to plot overall frame latency.

4.7 block-device-tester

This is a program for testing block devices. Please read the test-storage.txt file that is in the src/docs directory of the package for instructions on how to use this program.

Caution: Running this program on a device will destroy irretrievably all data on the device.

4.8 gsms-receiver

This is an example receiver for the GSMS (Guralp Seismic Monitoring System) protocol in Platinum. It is intended to be used as a starting point for a customer to take data from a gsms-out process on a Platinum unit. It receives data from a remote Platinum system and writes it to files.

Usage:

gsms-receiver -b host,service,sample-rate -W dir

gsms-receiver -t host,service,sample-rate -W dir

gsms-receiver -u host,service,sample-rate -W dir

where

The options are defined as follows:

Note: Status packets are received with the sample rate field set to zero.

PreviousNext

1. Preliminary Notes 2. Introduction 3. Installing the tools 4. Applications 5. Libraries 6. Configuration files 7. Revision history