This section provides an overview of the tools required to troubleshoot CXFS:

---

Caution: Many of the commands listed are beyond the scope of this book and are provided here for quick reference only. See the other guides and man pages referenced for complete information before using these commands.

---

Gather Cluster Configuration with cxfsdump

Before reporting a problem to SGI, you should use the cxfsdump (1M) command to gather configuration information about the CXFS cluster, such as network interfaces, CXFS registry information, I/O, and cluster database contents. This will allow SGI support to solve the problem more quickly.

---

Note: The cxfsdump command requires access across IRIX and Solaris nodes in the cluster via the rcp(1M) and rsh(1M) commands. Because these commands are not provided on Windows nodes, the cxfsdump command must be run manually on each Windows node.

---

You should run cxfsdump from a CXFS administration node in the cluster:

# /usr/cluster/bin/cxfsdump

Output for IRIX and Solaris systems will be placed in a file in the directory /var/cluster/cxfsdump-data directory on the CXFS administration node on which the cxfsdump command was run. The cxfsdump command will report the name and location of the file when it is finished.

If you cluster contains Windows nodes, you must run the command manually on each Windows node.

For example:

adminnode# cxfsdump Detecting cluster configuration Executing CXFSDUMP on CLUSTER testcluster NODE o200a Gathering cluster information... Determining OS level...... Getting versions info.... Obtaining CXFS database... Checking for tie-breakers etc... Obtaining hardware inventory... Grabbing /etc/hosts..... Grabbing /etc/resolv.conf... Grabbing /ets/nsswitch.conf... Obtaining physvol information using XVM... ioctl() to xvm api node failed: Invalid argument Could not get xvm subsystem info: xvmlib_execute_ioctl: system call failed. Obtaining Volume topology information using XVM... ioctl() to xvm api node failed: Invalid argument Could not get xvm subsystem info: xvmlib_execute_ioctl: system call failed. Copying failover configuration and scsifo paths ... Gathering network information... Checking for any installed Patches.. Monitoring file system buffer cache for 3 minutes... Running Systune ... Obtaining modified system tunable parameters... Creating ICRASH CMD file... Executing ICRASH commands... Copying CXFS logs... Copying /var/cluster/ha/log/cad_log... Copying /var/cluster/ha/log/clconfd_o200a... Copying /var/cluster/ha/log/cli_o200a... Copying /var/cluster/ha/log/cmond_log... Copying /var/cluster/ha/log/crsd_o200a... Copying /var/cluster/ha/log/fs2d_log... Copying /var/cluster/ha/log/fs2d_log.old... Copying SYSLOG... Distributing /usr/cluster/bin/cxfsdump.pl to node o200c ... Distributing /usr/cluster/bin/cxfsdump.pl to node o200b ... Creating the output directory : /var/cluster/cxfsdump-data Gathering node information for the cluster testcluster ... Running RSH to node o200c... Running RSH to node o200b... Waiting for other cluster nodes to gather data... FINAL CXFSDUMP OUTPUT IN /var/cluster/cxfsdump-data/testcluster_cxfsdump20020903.tar.gz

On Windows systems, cxfsdump creates a the directory called CXFSDump in the same directory where the the passwd file is kept. The cxfsdump command will report the location where the data is stored when it is complete. For example:

FINAL CXFSDUMP output in output_filename

When you encounter a problem, identify the cluster status by answering the following questions:

To locate the problem, do the following:

Ensure that you follow the instructions in “Preliminary Cluster Configuration Steps” in Chapter 3, before configuring the cluster.

Before you start configuring another new cluster, make sure no nodes are still in a CXFS membership from a previous cluster. Enter the following:

# icrash -e 'sthread | grep cmsd'

If the output shows a cmsd kernel thread, force a CXFS shutdown by entering the following:

# /usr/cluster/bin/cmgr -p
cmgr> admin cxfs_stop

Then check for a cmsd kernel thread again:

# icrash -e 'sthread | grep cmsd'

After waiting a few moments, if the cmsd kernel thread still exists, you must reboot the machine or leave it out of the new cluster definition. It will not be able to join a new cluster in this state and it may prevent the rest of the cluster from forming a new CXFS membership.

The cluster database membership quorum must remain stable during the configuration process. If possible, use multiple windows to display the fs2d_log file for each CXFS administration node while performing configuration tasks. Enter the following:

# tail -f /var/cluster/ha/log/fs2d_log

Check the member count when it prints new quorums. Under normal circumstances, it should print a few messages when adding or deleting nodes, but it should stop within a few seconds after a new quorum is adopted.

If not enough machines respond, there will not be a quorum. In this case, the database will not be propagated.

If you detect cluster database membership quorum problems, fix them before making other changes to the database. Try restarting the cluster infrastructure daemons on the node that does not have the correct cluster database membership quorum, or on all nodes at the same time. Enter the following:

# /etc/init.d/cluster stop
# /etc/init.d/cluster start

Please provide the fs2d log files when reporting a cluster database membership quorum problem.

Be consistent in configuration files for nodes across the pool, and when configuring networks. Use the same names in the same order. See “Configure System Files” in Chapter 2.

Use the appropriate node function definition:

The GUI provides a convenient display of a cluster and its components through the view area. You should use it to see your progress and to avoid adding or removing nodes too quickly. After defining a node, you should wait for it to appear in the view area before adding another node. After defining a cluster, you should wait for it to appear before you add nodes to it. If you make changes too quickly, errors can occur.

For more information, see “Starting the GUI” in Chapter 4.

When running the GUI on IRIX, do not move to another IRIX desktop while GUI action is taking place; this can cause the GUI to crash.

You should not change the names of the log files. If you change the names of the log files, errors can occur.

Periodically, you should rotate log files to avoid filling your disk space; see “Log File Management” in Chapter 6. If you are having problems with disk space, you may want to choose a less verbose log level; see “Configure Log Groups with the GUI” in Chapter 4, or “Configure Log Groups with cmgr” in Chapter 5.

When accessing the Brocade Web Tools V2.0 through Netscape on an IRIX node, you must first enter one of the following before starting Netscape:

If this is not done, Netscape will crash with a core dump.

This section discusses performance problems with unwritten extent tracking and exclusive write tokens.

Unwritten Extent Tracking

When you define a filesystem, you can specify whether unwritten extent tracking is on (unwritten=1) or off ( unwritten=0); it is on by default.

In most cases, the use of unwritten extent tracking does not affect performance and you should use the default to provide better security.

However, unwritten extent tracking can affect performance when both of the following are true:

• A file has been preallocated

• These preallocated extents are written for the first time with records smaller than 4MB

For optimal performance with CXFS when both of these conditions are true, it may be necessary to build filesystems with unwritten=0 (off).

---

Note: There are security issues with using unwritten=0 . For more information, see the IRIX Admin: Disks and Filesystems.

---

The default root crontab file contains the following entries (line breaks inserted here for readability):

0 5 * * *  find / -local -type f '(' -name core -o -name dead.letter ')' -atime +7 
-mtime +7 -exec rm -f '{}' ';'

 0 3 * * 0 if test -x /usr/etc/fsr; then (cd /usr/tmp; /usr/etc/fsr) fi 

The first entry executes a find(1) command that looks for and removes all files with the name core or dead.letter that have not been accessed in the past seven days.

The second entry executes an fsr(1M) command that improves the organization of mounted filesystems.

The find command will be run nightly on all local filesystems. Because CXFS filesystems are considered as local on all nodes in the cluster, the nodes may generate excessive filesystem activity if they try to access the same filesystems simultaneously. Therefore, you may wish use the following sequence to disable or modify the find crontab entries on all the CXFS administration nodes except for one:

The fsr command can only be run on the metadata server, so it is not harmful to leave it in the crontab file for CXFS clients, but it will not be executed.

To avoid a loss of connectivity between the metadata server and the CXFS clients, do not oversubscribe the metadata server or the private network connecting the nodes in the cluster. Avoid unnecessary metadata traffic.

If the amount of free memory is insufficient, a node may experience delays in heartbeating and as a result will be kicked out of the CXFS membership. To observe the amount of free memory in your system, use the osview(1) tool.

See also “Out of Logical Swap Space”.

If you want redefine a node ID or the cluster ID, you must first reboot. The problem is that the kernel still has the old values, which prohibits a CXFS membership from forming. However, if you perform a reboot first, it will clear the original values and you can then redefine the node or cluster ID.

Therefore, if you use cdbreinit on a node to recreate the cluster database, you must reboot it before changing the node IDs or the cluster ID. See “Recreating the Cluster Database”.

If a node is going to be down for a while, remove it from the cluster and the pool to avoid cluster database membership and CXFS membership quorum problems. See the following sections:

If you perform a forced shutdown on a node, you must restart CXFS on that node before it can return to the cluster. If you do this while the database still shows that the node is in a cluster and is activated, the node will restart the CXFS membership daemon. Therefore, you may want to do this after resetting the database or after stopping CXFS services.

For example, enter the following on the node you wish to start:

# /usr/cluster/bin/cmgr -p
cmgr> stop cx_services on node localnode
cmgr> admin cxfs_start

See also “Forced CXFS Shutdown: Revoke Membership of Local Node” in Chapter 6.

When serial hardware reset is enabled, CXFS requires a reset successful message before it moves the metadata server. Therefore, if you have the serial hardware reset capability enabled and you must remove the reset lines for some reason, you must also disable the reset capability. See “Modify a Node Definition with the GUI” in Chapter 4, or “Modify a Node with cmgr” in Chapter 5.

---

Note: The reset capability is mandatory to ensure data integrity for clusters with only two server-capable nodes, and it is highly recommended for all server-capable nodes. Larger clusters should have an odd number of server-capable and CXFS administration nodes. See “Cluster Environment” in Chapter 1.

---

CXFS filesystems are really clustered XFS filesystems; therefore, in case of a file system corruption, you can use the xfs_check(1M) and xfs_repair(1M) commands. However, you must first ensure that you have an actual case of data corruption and retain valuable metadata information by replaying the XFS logs before running xfs_repair.

---

Caution: If you run xfs_repair without first replaying the XFS logs, you may introduce data corruption.

---

You should only run xfs_repair in case of an actual filesystem corruption; forced filesystem shutdown messages do not necessarily imply that xfs_repair should be run. Following is an example of a message that does indicate an XFS file corruption:

XFS read error in file system meta-data block 106412416

When a filesystem is forcibly shut down, the log is not empty -- it contains valuable metadata. You must replay it by mounting the filesystem. The log is only empty if the filesystem is unmounted cleanly (that is, not a forced shutdown, not a crash). You can use the following command line to see an example of the transactions captured in the log file:

# xfs_logprint -t device

If you run xfs_repair before mounting the filesystem, xfs_repair will delete all of this valuable metadata.

If you think you have a filesystem with real corruption, do this:

For more information, see the IRIX Admin: Disks and Filesystems.

If you cannot access a filesystem, check the following:

If the GUI will not run, check the following:

If the log files are consuming too much disk space, you should rotate them; see “Log File Management” in Chapter 6. You may also want to consider choosing a less-verbose log level; see the following:

If you are unable to define a node, it may be that there are hostname resolution problems. See “Hostname Resolution: /etc/sys_id, /etc/hosts, /etc/nsswitch.conf” in Chapter 2.

The following may cause the system to hang:

If a node is detected in /var/adm/SYSLOG but it never receives a Membership delivered message, it is likely that there is a network problem. See “Configure System Files” in Chapter 2.

The Membership delivered messages in the /var/adm/SYSLOG file include a list of cell IDs for nodes that are members in the new CXFS membership.

Following each cell ID is a number, the membership version , that indicates the number of times the membership has changed since the node joined the membership.

If the Membership delivered messages are appearing frequently in /var/adm/SYSLOG, this may indicate a network problem:

See “Configure System Files” in Chapter 2.

If you cannot log in to a CXFS administration node, you can use one of the following commands, assuming the node you are on is listed in the other nodes' .rhosts files:

# rsh hostname ksh -i
# rsh hostname csh -i

The following message indicates a problem (output lines wrapped here for readability):

ALERT: I/O error in filesystem ("/mnt") meta-data dev 0xbd block 0x41df03 ("xlog_iodone")
ALERT:     b_error 0 b_bcount 32768 b_resid 0
NOTICE: xfs_force_shutdown(/mnt,0x2) called from line 966 of file ../fs/xfs/xfs_log.c. 
  Return address = 0xc0000000008626e8
ALERT: I/O Error Detected.  Shutting down filesystem: /mnt
ALERT: Please umount the filesystem, and rectify the problem(s)

You can fix this problem using xfs_repair only if there is no metadata in the XFS log. See “Appropriate Use of xfs_repair”, for the appropriate procedure.

I/O errors can also appear if the node is unable to access the storage. This can happen for several reasons:

If you have defined filesystems and then rename your cluster (by deleting the old cluster and defining a new cluster), CXFS will not be able to mount the existing filesystems. This happens because the clustered XVM volume on which your CXFS filesystem resides is not accessible to the new cluster, and the volumes are therefore considered as foreign.

In order to mount the filesystem on the new cluster, you must use the XVM steal command to bring the clustered XVM volume into the domain of the new cluster. For more information, see the XVM Volume Manager Administrator's Guide .

If you create new slices on a previously sliced disk that have the same starting blocks as slices already existing on the disk, and if the old slices had filesystems, then the GUI will display those old filesystems even though they may not be valid.

A client_timeout value is set by the clconfd and cxfs_client daemons. The value depends on the order in which filesystems are mounted on the various nodes. The value adapts to help ensure that all filesystems get mounted in a timely manner. The value has no effect on the filesystem operation after it is mounted.

The value for client_timeout may differ among nodes, and therefore having multiple values is not really a problem.

The retry value is forced to be 0 and you cannot change it.

---

Caution: You should not attempt to change the client_timeout value. Improperly setting the values for client_timeout and retry could cause the mount command to keep waiting for a server and could delay the availability of the CXFS filesystems.

---

You can expect to see the following messages. They are normal and do not indicate a problem.

If the clconfd daemon exits immediately after it starts up, it means that the CXFS license has not been properly installed. For information about the associated error message, see “License Error”.

You must install the license on each node before you can use CXFS. If you increase the number of CPUs in your system, you may need a new license. See Chapter 2, “IRIX Systems: Installation of CXFS Software and System Preparation”.

The following example /var/adm/SYSLOG message indicates an oversubscribed system:

ALERT: inetd [164] - out of logical swap space during fork while
allocating uarea - see swap(1M)
Availsmem 8207 availrmem 427 rlx freemem 10, real freemem 9

See “Use System Capacity Wisely”.

The cluster daemons could also be leaking memory in this case. You may need to restart them:

# /etc/init.d/cluster restart

For example:

Mar  1 15:06:18 5A:nt-test-07 unix: NOTICE: Physvol (name cip4) has no 
CLUSTER name id: set to ""

This message means the following:

The following message in /var/adm/SYSLOG indicates a kernel-triggered revocation of CXFS membership:

Membership lost - withdrawing from cluster

You must actively allow CXFS membership for the local node in this situation. See “Allow Membership of the Local Node with the GUI” in Chapter 4, or “Allow Membership of the Local Node with cmgr” in Chapter 5.

If you see the following message in the /var/cluster/ha/log/clconf_hostname logfile, it means that the CXFS license was not properly installed:

CXFS not properly licensed for this host.  Run
               '/usr/cluster/bin/cxfslicense -d'
       for detailed failure information. 

If you do not have the CXFS license properly installed, you will see the following error on the console when trying to run CXFS:

Cluster services:CXFS not properly licensed for this host.  Run
        '/usr/cluster/bin/cxfslicense -d'
for detailed failure information.  After fixing the
license, please run '/etc/init.d/cluster restart'.

An error such as the following example will appear in the SYSLOG file:

Mar  4 12:58:05 6X:typhoon-q32 crsd[533]: <<CI> N crs 0> Crsd restarted.
Mar  4 12:58:05 6X:typhoon-q32 clconfd[537]: <<CI> N clconf 0> 
Mar  4 12:58:05 5B:typhoon-q32 CLCONFD failed the CXFS license check.Use the 
Mar  4 12:58:05 5B:typhoon-q32    '/usr/cluster/bin/cxfslicense -d'
Mar  4 12:58:05 5B:typhoon-q32 command to diagnose the license problem.

If the clconfd daemon dies right after it starts up, this error is present.

You must install the license on each node before you can use CXFS. See Chapter 2, “IRIX Systems: Installation of CXFS Software and System Preparation”.

If you have conflicting cluster ID numbers at your site, you will see errors such as the following:

WARNING: mtcp ignoring  alive message from 1 with wrong ip addr 128.162.89.34
WARNING: mtcp ignoring  alive message from 0 with wrong ip addr 128.162.89.33

A cluster ID number must be unique. To solve this problem, make the cluster ID numbers unique.

This error can occur if you redefine the cluster configuration and start CXFS services while some nodes have stale information from a previous configuration.

To solve the problem, first try the steps in “Eliminate a Residual Cluster”. If that does not work, reboot the nodes that have stale information. You can determine which nodes have stale information as follows: stale nodes will complain about all of the nodes, but the up-to-date nodes will complain only about the stale nodes. The /var/cluster/ha/log/clconfd_ log file on the stale nodes will also show error messages about SGI_CMS_CONFIG_ID failures.

If there are too many error messages to recognize the stale nodes, reboot every node.

CXFS logs both normal operations and critical errors to /var/adm/SYSLOG, as well as to individual log files for each log group.

In general, errors in the /var/adm/SYSLOG file take the following form:

timestamp priority_&_facility : hostname process[ID]: <internal_info> CODE message_text 

For example:

Sep  7 11:12:59 6X:cxfs0 cli[5830]: < E clconf 0> CI_IPCERR_NOSERVER, clconf
ipc: ipcclnt_connect() failed, file /var/cluster/ha/comm/clconfd-ipc_cxfs0

Table 10-1 shows the parts of the preceding SYSLOG message.

The following sections present only the message identifiers and text.

CXFS maintains logs for each of the CXFS daemons. For information about customizing these logs, see “Set Log Configuration with the GUI” in Chapter 4.

Log file messages take the following form:

daemon_log timestamp internal_process: message_text

For example:

cad_log:Thu Sep  2 17:25:06.092  cclconf_poll_clconfd: clconf_poll failed with error CI_IPCERR_NOPULSE

Table 10-2, shows the parts in the preceding message.

If CXFS services to do not restart after a reboot, it may be that the start flag was turned off by using the stop function of the GUI or the cmgr(1M) command. In this case, issuing a /etc/init.d/cluster start will not restart the services. You must start CXFS services. If you use the GUI or cmgr to restart the services, the configuration will be set so that future reboots will also restart CXFS services.

For information, see “Start CXFS Services with the GUI” in Chapter 4, or “Start CXFS Services with cmgr” in Chapter 5.

To clear the cluster database on all the nodes of the cluster, do the following, completing each step on each node before moving to the next step:

---

Caution: This procedure deletes all configuration information.

---

See “Eliminate a Residual Cluster”, to get rid of possible stale cluster configuration in the kernel. If needed, reboot the nodes.

Enter the following individually on every node to reboot the cluster:

# reboot

For information about nodes running operating systems other than IRIX, see the CXFS MultiOS for CXFS Client-Only Nodes: Installation and Configuration Guide.

If you want CXFS services to restart whenever the node is rebooted, use the GUI or cmgr(1M) to start CXFS services. For information, see “Start CXFS Services with the GUI” in Chapter 4, and “Start CXFS Services with cmgr” in Chapter 5.

The following are situations that may require a rebooting:

Suppose the following:

To allow node2 to form a cluster by itself, you must do the following:

The cluster will attempt to communicate with the node1 because it is still configured in the cluster, even though it is down. Therefore, it may take some time for the CXFS kernel membership to form and for filesystems to mount.

The cluster flag to chkconfig(1M) controls the other cluster administration daemons and the replicated cluster database. If it is turned off, the database daemons will not be started at the next reboot and the local copy of the database will not be updated if you make changes to the cluster configuration on the other nodes. This could cause problems later, especially if a majority of nodes are not running the database daemons.

If the cluster daemons are causing serious trouble and prevent the machine from booting, you can recover the node by booting in single-user mode, turning off the cluster flag, and booting in multiuser mode:

# init 1
# /etc/chkconfig cluster off
# init 2

To stop and restart cluster infrastructure daemons, enter the following:

# /etc/init.d/cluster stop 
# /etc/init.d/cluster start

These commands affect the cluster infrastructure daemons only.

---

Caution: When the cluster infrastructure daemons are stopped, the node will not receive database updates and will not update the kernel configuration. This can have very unpleasant side effects. Under most circumstances, the infrastructure daemons should remain running at all times. Use these commands only as directed.

---

See also “Restarting CXFS Services”. For general information about the daemons, see “Daemons” in Appendix A.

To recreate the initial cluster database, do the following:

---

Note: See also “Reboot Before Changing Node ID or Cluster ID”.

---