The ACT Kernel Dump Analysis Tool

Introduction

ACT is a tool developed by engineers at Sun over the course of several years to aid in the process of analysing kernel dumps. The ACT tool analyzes a system kernel dump and generates a human-readable text summary. Frequently, this text summary can be sent to Sun (provided you have a support contract), rather than uploading a potentially huge core file. It was not unusual to ftp 1GB+ core files to Sun on a regular basis for analysis. This saves a lot of time and bandwidth.

ACT prints detailed and accurate information about:

• Where the kernel panicked
• A complete list of threads on the system
• The contents of the /etc/system file which was read when the failed system booted
• A list of kernel modules that were loaded at the time of the panic
• The output of the kernel message buffer
• Full deadlock detection relating to threads blocked on mutexes or readers/writer locks
• Threads blocked in either getblk() or biowait()

ACT is delivered in a standard Sun package format. Simply unzip and untar the package and install it as any other package using pkgadd. The ACT package is installed in the directory /opt/CTEact. The actual executable can be found in /opt/CTEact/bin/act.

When possible, ACT should always be run from the server that produced the core to be analysed. ACT can be run on a server other than the one which produced the core, however, care should be taken to insure that both boxes are running the same version of Solaris at the same kernel patch revision. Results may not be accurate when running ACT on a server other than the one which produced the core file.

Instructions

ACT takes the following options which may appear in any order:

-d corefile

ACT assumes that the file corefile contains the kernel core image. This file could be /dev/mem if you want ACT to analyze the running system.

-l

Should be used when running act on a live system.

-n namelist

ACT assumes that the file namelist contains a valid kernel namelist. This file could be /dev/ksyms if you want ACT to analyze the running system.

-s directory

Tells ACT to split its output into several files writing the data into the directory specified to aid readability. The files created are:


     biowait     getblk     modules   msgbuf    mutex  
     rwlock      threads    system    summary   sunsolve 

-u

Displays stack information in an alternate form.

-z

This informs ACT to display timezone information in localtime rather than GMT.

Usage Examples

# ./act -l (output on live server to screen)

# ./act -l -s /tmp/dir/ (output from live server to seperate files)

# ./act -d /var/crash/hostname/vmcore.0 -s /tmp/dir/ (output core file to seperate files in /tmp/dir)

# ./act -d /var/crash/hostname/vmcore.0 > /tmp/act_out (output core file to file /tmp/act_out)

Downloading ACT

ACT, also known by its package name of CTEact, is not a very easy program to find. Searches in the public and private areas of SunSolve do not show any relevant links. The author originally obtained ACT through a Sun Support Engineer. A recent look at SunSolve in the "Diagnostic Tools" section of the site shows a script called "ISCDA", or Initial System Crash Dump Analysis. It appears that the ISCDA may be intended as a replacement for ACT, even though it does not perform all of the existing functions of ACT. Please contact your Sun Support Representative to see if you can still obtain a copy of ACT, or try contacting Sun Technical Support. Often the engineers there will email a copy of the package when they need specific information relating to a service call.