Critical to the success of the PCP archive logging scheme is the fact that the library routines providing access to real-time feeds of performance metrics also provide access to the archive logs.

Live feeds (or real-time) sources of performance metrics and archives are literally interchangeable, with a single Performance Metrics Application Programming Interface (PMAPI) that preserves the same semantics for both styles of metric source. In this way, applications and tools developed against the PMAPI can automatically process either live or historical performance data.

The only restriction is that both live and historical data cannot be monitored simultaneously with the same invocation of a visualization tool.

One of the most important applications of archive logging services provided by PCP is in the area of retrospective analysis. In many cases, understanding today's performance problems can be assisted by side-by-side comparisons with yesterday's performance. With routine creation of performance archive logs, you can concurrently replay pictures of system performance for two or more periods in the past.

Archive logs are also an invaluable source of intelligence when trying to diagnose what went wrong, as in a performance postmortem. Because the PCP archive logs are entirely self-contained, this analysis can be performed off-site if necessary.

Each archive log contains metric values from only one host. However, many PCP tools can simultaneously visualize values from multiple archives collected from different hosts.

The archives can be replayed against the inference engine (pmie is an application that uses the PMAPI). This allows you to automate the regular, first-level analysis of system performance.

Such analysis can be performed by constructing suitable expressions to capture the essence of common resource saturation problems, then periodically creating an archive and playing it against the expressions. For example, you may wish to create a daily performance audit (run by the cron command) to detect performance regressions.

For more about pmie, see Chapter 5, “Performance Metrics Inference Engine”.

By collecting performance archives with relatively long sampling periods, or by reducing the daily archives to produce summary logs, the capacity planner can collect the base data required for forward projections, and can estimate resource demands and explore “what if” scenarios by replaying data using visualization tools and the inference engine.

Most commonly, a PCP tool would be invoked with the -a option to process an archive log some time after pmlogger had finished creating the archive. However, a tool such as oview that uses a Time Control dialog (see “Time Duration and Control” in Chapter 3) stops when the end of archive is reached, but could resume if more data is written to the PCP archive log.

---

Note: pmlogger uses buffered I/O to write the archive log so that the end of the archive may be aligned with an I/O buffer boundary, rather than with a logical archive log record. If such an archive was read by a PCP tool, it would appear truncated and might confuse the tool. These problems may be avoided by sending pmlogger a SIGUSR1 signal, or by using the flush command of pmlc to force pmlogger to flush its output buffers.

---

When a PCP archive is created by pmlogger, an archive basename must be specified and several physical files are created, as shown in Table 6-1.

The PCP archive management tools support a consistent scheme for selecting the basenames for the files in a collection of archives and for mapping these files to a suitable directory hierarchy.

Once configured, the PCP tools that manage archive logs employ a consistent scheme for selecting the basename for an archive each time pmlogger is launched, namely the current date and time in the format YYYYMMDD.HH.MM. Typically, at the end of each day, all archives for a particular host on that day would be merged to produce a single archive with a basename constructed from the date, namely YYYYMMDD. The pmlogger_daily script performs this action and a number of other routine housekeeping chores.

If you are using a deployment of PCP tools and daemons to collect metrics from a variety of hosts and storing them all at a central location, you should develop an organized strategy for storing and naming your log files.

---

Note: There are many possible configurations of pmlogger. The directory organization described in this section is recommended for any system on which pmlogger is configured for permanent execution (as opposed to short-term executions, for example, as launched from pmchart to record some performance data of current interest).

---

Typically, the IRIX filesystem structure can be used to reflect the number of hosts for which a pmlogger instance is expected to be running locally, obviating the need for lengthy and cumbersome filenames. It makes considerable sense to place all logs for a particular host in a separate directory named after that host. Because each instance of pmlogger can only log metrics fetched from a single host, this also simplifies some of the archive log management and administration tasks.

For example, consider the filesystem and naming structure shown in Figure 6-1.

Figure 6-1. Archive Log Directory Structure

The specification of where to place the archive log files for particular pmlogger instances is encoded in the configuration file /var/pcp/config/pmlogger/control, and this file should be customized on each host running an instance of pmlogger.

If many archives are being created, and the associated PCP collector systems form peer classes based upon service type (for example, Web servers, DBMS servers, NFS servers, and so on), then it may be appropriate to introduce another layer into the directory structure, or use symbolic links to group together hosts providing similar service types.

A single PCP archive may be partitioned into a number of volumes. These volumes may expedite management of the archive; however, the metadata file and at least one volume must be present before a PCP tool can process the archive.

You can control the size of an archive log volume by using the -v command line option to pmlogger. This option specifies how large a volume should become before pmlogger starts a new volume. Archive log volumes retain the same base filename as other files in the archive log, and are differentiated by a numeric suffix that is incremented with each volume change. For example, you might have a log volume sequence that looks like this:

netserver.log.0
netserver.log.1
netserver.log.2

You can also cause an existing log to be closed and a new one to be opened by sending a SIGHUP signal to pmlogger, or by using the pmlc command to change the pmlogger instructions dynamically, without interrupting pmlogger operation. Complete information on log volumes is found in the pmlogger(1) man page.

The configuration files used by pmlogger describe which metrics are to be logged. Groups of metrics may be logged at different intervals to other groups of metrics. Two states, mandatory and advisory, also apply to each group of metrics, defining whether metrics definitely should be logged or not logged, or whether a later advisory definition may change that state.

The mandatory state takes precedence if it is on or off, causing any subsequent request for a change in advisory state to have no effect. If the mandatory state is maybe, then the advisory state determines if logging is enabled or not.

The mandatory states are on, off, and maybe. The advisory states, which only affect metrics that are mandatory maybe, are on and off. Therefore, a metric that is mandatory maybe in one definition and advisory on in another definition would be logged at the advisory interval. Metrics that are not specified in the pmlogger configuration file are mandatory maybe and advisory off by default and are not logged.

A complete description of the pmlogger configuration format can be found on the pmlogger(1) man page.

Once a PCP archive log has been created, the pmdumplog utility may be used to display various information about the contents of the archive. For example, start with the following command:

pmdumplog -l /var/adm/pcplog/www.sgi.com/960731

It might produce the following output:

Log Label (Log Format Version 1)
Performance metrics from host www.sgi.com
     commencing Wed Jul 31 00:16:34.941 1996
     ending     Thu Aug  1 00:18:01.468 1996

The simplest way to discover what performance metrics are contained within an archive is to use pminfo as shown in Example 6-1:

pminfo -a /var/adm/pcplog/www.sgi.com/960731 network.mbuf
network.mbuf.alloc 
network.mbuf.typealloc
network.mbuf.clustalloc
network.mbuf.clustfree
network.mbuf.failed
network.mbuf.waited
network.mbuf.drained

A collection of one or more PCP archive logs may be combined with a control file to produce a PCP archive folio. Archive folios are created using either mkaf or the interactive record mode services of various PCP GUI monitoring tools.

You may tailor pmlogger dynamically with the pmlc command. Normally, the pmlogger configuration is read at startup. If you choose to modify the config file to change the parameters under which pmlogger operates, you must stop and restart the program for your changes to have effect. Alternatively, you may change parameters whenever required by using the pmlc interface.

To run the pmlc tool, enter:

pmlc

By default, pmlc acts on the primary instance of pmlogger on the current host. See the pmlc(1) man page for a description of command line options. When it is invoked, pmlc presents you with a prompt:

pmlc> 

You may obtain a listing of the available commands by entering a question mark (?) and pressing Enter. You see output similar to that in Example 6-2:

     show loggers [@<host>]           display <pid>s of running pmloggers
     connect _logger_id [@<host>]     connect to designated pmlogger
     status                           information about connected pmlogger
     query metric-list                show logging state of metrics
     new volume                       start a new log volume
     flush                            flush the log buffers to disk
     log { mandatory | advisory } on <interval> _metric-list
     log { mandatory | advisory } off _metric-list
     log mandatory maybe _metric-list
     timezone local|logger|'<timezone>' change reporting timezone
     help                               print this help message
     quit                               exit from pmlc
     _logger_id   is  primary | <pid> | port <n>
     _metric-list is  _metric-spec | { _metric-spec ... }
     _metric-spec is  <metric-name> | <metric-name> [ <instance> ... ]

pmlc
pmlc> show loggers @babylon
The following pmloggers are running on babylon:
       primary (1892)
pmlc> connect 1892 @babylon
pmlc> log advisory on 2 secs disk.dev.read
pmlc> query disk.dev
disk.dev.read
       adv  on  nl       5 min  [131073 or “dks0d1”]
       adv  on  nl       5 min  [131074 or “dks0d2”]
pmlc> quit

---

Note: Any changes to the set of logged metrics made via pmlc are not saved, and are lost the next time pmlogger is started with the same configuration file. Permanent changes are made by modifying the pmlogger configuration file(s).

---

Refer to the pmlc(1) and pmlogger(1) man pages for complete details.

Assume you wish to activate primary archive logging on the PCP collector host pluto. Execute all of the following tasks while logged into pluto as the superuser (root).

Assume you wish to create archive logs on the local host for performance metrics collected from the remote host bert. Execute all of the following tasks while logged into the local host as the superuser (root).

To create archive logs on the local host for performance metrics collected from multiple remote hosts, repeat the steps in Procedure 6-1 for each remote host.

Assume the local host has been set up to create archive logs of performance metrics collected from one or more hosts (which may be either the local host or a remote host).

To activate the maintenance and housekeeping scripts for a collection of archive logs, execute the following tasks while logged into the local host as the superuser (root):

The pmlogextract tool takes a number of PCP archive logs from a single host and performs the following tasks:

See the pmlogextract(1) man page for full information on this command. It replaced functionality of the pmlogmerge tool.

On each system for which PMCD is active (each PCP collector system), there is an option to have a distinguished instance of the archive logger pmlogger (the “primary” logger) launched each time PMCD is started. This may be used to ensure the creation of minimalist archive logs required for ongoing system management and capacity planning in the event of failure of a system where a remote pmlogger may be running, or because the preferred archive logger deployment is to activate pmlogger on each PCP collector system.

Run the following command as superuser on each PCP collector system where you want to activate the primary pmlogger:

chkconfig pmlogger on

The primary logger launches the next time PMCD is started. If you wish this to happen immediately, follow up with this command:

/etc/init.d/pcp start

When it is started in this fashion, the /etc/config/pmlogger.options file provides command line options for pmlogger. In the default setup, this in turn means that the initial logging state and configuration is specified in the file /var/pcp/config/pmlogger/config.default. Either one or both of these files may be modified to tailor pmlogger operation to the local requirements.