Disk Error - HP Openview !

From: Ayaz Anjum (ayaz@mdsuae.ae)
Date: Mon Mar 07 2005 - 01:41:34 EST


HI Guys !

On one of our servers that is connected to 3960 SAN, i am getting the
following messages everyday in the morning at 8:00 AM

******************************************************
Mar 2 08:00:36 bill01 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/ssd@g60020f200000c18f40b3942c0002eaa5 (ssd79):
Mar 2 08:00:36 bill01 Error for Command: read defect data Error
Level: Informational
Mar 2 08:00:36 bill01 scsi: [ID 107833 kern.notice] Requested Block: 0
Error Block: 0
Mar 2 08:00:36 bill01 scsi: [ID 107833 kern.notice] Vendor: SUN
Serial Number: 0019114110
Mar 2 08:00:36 bill01 scsi: [ID 107833 kern.notice] Sense Key: Illegal
Request
Mar 2 08:00:36 bill01 scsi: [ID 107833 kern.notice] ASC: 0x20 (invalid
command operation code), ASCQ: 0x0, FRU: 0x0
Mar 2 08:00:37 bill01 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/ssd@g60020f200000c18f40b394d6000c9934 (ssd78):
Mar 2 08:00:37 bill01 Error for Command: read defect data Error
Level: Informational
Mar 2 08:00:37 bill01 scsi: [ID 107833 kern.notice] Requested Block: 0
Error Block: 0
Mar 2 08:00:37 bill01 scsi: [ID 107833 kern.notice] Vendor: SUN
Serial Number: 0082565210
Mar 2 08:00:37 bill01 scsi: [ID 107833 kern.notice] Sense Key: Illegal
Request
Mar 2 08:00:37 bill01 scsi: [ID 107833 kern.notice] ASC: 0x20 (invalid
command operation code), ASCQ: 0x0, FRU: 0x0
Mar 2 08:00:40 bill01 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/ssd@g60020f200000c18f40b395b0000be19d (ssd77):
Mar 2 08:00:40 bill01 Error for Command: read defect data Error
Level: Informational
Mar 2 08:00:40 bill01 scsi: [ID 107833 kern.notice] Requested Block: 0
Error Block: 0
Mar 2 08:00:40 bill01 scsi: [ID 107833 kern.notice] Vendor: SUN
Serial Number: 0077865310
Mar 2 08:00:40 bill01 scsi: [ID 107833 kern.notice] Sense Key: Illegal
Request
Mar 2 08:00:40 bill01 scsi: [ID 107833 kern.notice] ASC: 0x20 (invalid
command operation code), ASCQ: 0x0, FRU: 0x0
******************************************************

There is one error once for each volume from SAN and not from internal disk.
So if i have 20 LUNs from SAN i get 20 of these messages once for each
volume/LUN in morning at 8:00 AM

Looking at ps output i found a script runs from HP-Open View whcih scans the
disks. I am appending the script at end of my mail for reference. Could not
figure out from script which options of format are causing the error output
? Searching on web i found some similar errors popping up with disks that
have old firmware and using format options that are not supported. s

Can someone look into the script to see which format options are being used.
Any suggestions ?

thanks

Ayaz

********************HP OV
Script*********************************************
#!/usr/bin/ksh
# OPC_WHAT_STRING="@(#)HP OpenView Smart Plug-In for Unix OS for Sun Solaris
B.02.53 (01/30/04)"
############################################################################
####
#
# File: osspi_discmod.phydisk.sh.sol
# Description: Node Solaris: module that discovers any disks and partitions
# on a managed node writes hardware pathes into seperate file
# for further processing
# Language: Korn shell
# Package: N/A
#
# (c) Copyright 2004 Hewlett-Packard Development Company, L.P.
#
############################################################################
####

if [[ "x${OSSPI_TRACE}" != "x" ]]; then
  set -xv
fi

. osspi_platdef
. osspi_xmllib

#---------------------------------------------------------------------------

---
# Function:    check_file_location
# Description: checks the file and directory for existance, if necessary
#              new directories will be created
# Return code: 0 - OK
# Parms:       a file
#---------------------------------------------------------------------------
---
check_file_location()
{
    # the file exists, you can exit immediately
    [[ -f $1 ]] && return 0
    # check for existance of parent directory structure, eventually create
    typeset dir=`dirname $1`
    [[ ! -d $dir ]] && mkdir -m 755 -p $dir
    # create the file
    touch $1
    chmod 644 $1
    return
}
#---------------------------------------------------------------------------
---
# Function:    get_trace_level
# Description: Retrieves the trace level from the SPI config file. Prints
level
#              to stdout.
#              The surrounding code implements some caching for one run of
the
#              calling script.
# Return code: 0 - always
# Parms:       none
#---------------------------------------------------------------------------
---
unset OSSPI_TRC_LEVEL
get_trace_level()
{
    [[ ! -z "${OSSPI_TRC_LEVEL}" ]] && echo ${OSSPI_TRC_LEVEL} && return 0
    if [[ -r ${OSSPI_CFG} ]]; then
	OSSPI_TRC_LEVEL=`${AWK} '/^TRACE_LEVEL/{ print $2; exit 0 }' ${OSSPI_CFG}`
    fi
    [[ -z "${OSSPI_TRC_LEVEL}" ]] && OSSPI_TRC_LEVEL=0
    echo ${OSSPI_TRC_LEVEL}
    return 0
}
OSSPI_TRC_LEVEL=`get_trace_level`
#---------------------------------------------------------------------------
---
# Function:    get_trace_obj
# Description: Retrieves the trace object from the SPI config file. Prints
#              to stdout.
#              The surrounding code implements some caching for one run of
the
#              calling script.
# Return code: 0 - always
# Parms:       none
#---------------------------------------------------------------------------
---
unset OSSPI_TRC_OBJ
get_trace_obj()
{
    [[ ! -z "${OSSPI_TRC_OBJ}" ]] && echo ${OSSPI_TRC_OBJ} && return 0
    if [[ -r ${OSSPI_CFG} ]]; then
	OSSPI_TRC_OBJ=`${AWK} '/^TRACE_OBJ/{ print $2; exit 0 }' ${OSSPI_CFG}`
    fi
    echo "${OSSPI_TRC_OBJ:-${PGM_NAME}}"
    return 0
}
OSSPI_TRC_OBJ=`get_trace_obj`
#---------------------------------------------------------------------------
---
# Function:    log_trace
# Description: Prints a trace statement into the SPI trace file. Format is:
#               <date> OSSPI(<process>): <text>
#              where only the <text> has to be passed as parameter.
# Return code: 0 - always
# Parms:       any. First parameter interpreted as trace level if between
1 - 9
#---------------------------------------------------------------------------
---
log_trace()
{
    typeset lv=0
    case $1 in
	[0-9])
	    lv=$1
	    shift
    esac
    check_file_location ${OSSPI_TRC_FILE}
    if [[ `get_trace_level` -gt ${lv} && "`get_trace_obj`" =
"${PGM_NAME}" ]]; then
	echo `date +"${DATE_FMT}"` OSSPI\(${PGM_NAME}-$$\): $* >> ${OSSPI_TRC_FILE}
    fi
    return 0
}
#---------------------------------------------------------------------------
---
# Function:    log_error
# Description: Prints a error/warn statement into the SPI error log file.
Format
#              is:  <date> [<severity>] OSSPI(<process>): <text>
#              where only the <text> has to be passed as parameter.
#              If environment variable OSSPI_LOG_ERROR_TARGET is set to
stderr,
#              error text is additionally printed to stderr.
#              If environment variable OSSPI_LOG_ERROR_TARGET is set to
other
#              than none or stderr, a message with error text is generated
via
#              opcmsg.
# Return code: 0 - always
# Environment: OSSPI_LOG_ERROR_TARGET - specifies, if error statement is
#                                       printed to logfile only, or as well
to
#                                       stderr, or as well via opcmsg
#              OSSPI_OPCMSG_EXTRA     - specifies extra arguments for the
#                                       opcmsg call
# Parms:       any
#              if first param is a number between 0 and 4 it is mapped:
#                  0: Info statement for log file
#                >=2: Error statement for log file
#                0-4: for opcmsg call mapped to VPO severities (0=normal)
#---------------------------------------------------------------------------
---
set -A VPO_SEV Normal Warning Minor Major Critical
set -A LOG_SEV INFO WARNING ERROR ERROR ERROR
log_error()
{
    typeset -i sev=1
    if [[ "$1" = [0-4] ]]; then
	sev=$1
	shift
    fi
    # add prefix with date, label, product name, program name, and process
ID
    typeset d=`date +"${DATE_FMT}"`
    typeset msg_prefix="${d} [${LOG_SEV[$sev]}] OSSPI(${PGM_NAME}-$$):"
    typeset msg="$msg_prefix $*"
    check_file_location ${OSSPI_ERR_FILE}
    # error statement is always printed to logfile
    echo $msg >> ${OSSPI_ERR_FILE}
    # if set by environment variable, error statement is also printed to
stderr
    # (might be expanded later on for other ERROR_TARGETS).
    case "$OSSPI_LOG_ERROR_TARGET" in
	none)
	    ;;
	stderr)
	    echo "${LOG_SEV[$sev]}: $*" >&2
	    ;;
	*)
	    ${OPC_BIN_DIR}/opcmsg sev=${VPO_SEV[$sev]} "app=HP OSSPI" \
		    "msg_g=OSSPI-Internal" obj="${PGM_NAME}" \
                    "msg_t=$*" ${OSSPI_OPCMSG_EXTRA} \
		    2> /dev/null
	    if [[ $? -ne 0 && -w ${OSSPI_ERR_FILE} ]]; then
		echo "$msg_prefix Problem with generating VPO message: $*" \
		    >> ${OSSPI_ERR_FILE}
	    fi
	    ;;
    esac
    return 0
}
OSSPI_DEFECT_CFG=${OSSPI_VAR_CFG_DIR}/osspi_defect.cfg
#---------------------------------------------------------------------------
--
# Function:	Discover_PerfTool
# Description:	Returns the performance tool type of node [MW|CD|GP|NP]
#---------------------------------------------------------------------------
--
Discover_PerfTool()
{
        typeset -i instances=0
        PerfType="NP" # set performance agent to default state -
no-performance tool active on node
	log_trace "Assuming by default no peformance tool present on node -
PerfType=${PerfType}"
        Id=$(${OPCDCODE} ${OV_VAR_DIR}/conf/OpC/agtreg | ${AWK} '/^AGENT_ID/
{ if ( $2 == "12" ) { printf ("%s\n", $2); } }')
        if [[ ${Id} = 12 ]]; then
                instances=$(${AWK} 'BEGIN{inst=0}
                        {if ($2 == "12" && $3 ~ /.+coda/) { inst++; }}
                        END{print inst;}' ${OPC_VAR_DIR}/tmp/OpC/pids 2>
/dev/null)
			log_trace "Coda running ${instances} instances on this node"
                if [[ ${instances} -ge 1 ]]; then
                        PerfType="CD"
			log_trace "Coda installed and active on node - PerfType=${PerfType}"
                fi
        fi
        pabinary=""
        # check if either OVPA or MWA is installed on node
        if [[ -x ${OVPA} ]]; then
                pabinary=${OVPA}
        elif [[ -x ${MWA} ]]; then
                pabinary=${MWA}
        fi
        # Requires MWA/OVPA to have ITO Messages on
        if [[ -n ${pabinary} ]]; then
                ${pabinary} status all > /dev/null 2>&1
                res=$?
		log_trace "OVPA is installed on node - 'ovpa status all' exited with
${res}"
                if [[ ${res} = 0 ]]; then
                        PerfType="MW"
			log_trace "OVPA installed and active on node - PerfType=${PerfType}"
                fi
        elif [[ -x ${GP} ]]; then # check if Glance present on the node
                PerfType="GP"
		log_trace "Glance is present on node - PerfType=${PerfType}"
        fi
        ${ECHO} ${PerfType}
        return 0
}
if [[ -f ${OSSPI_CFG} ]]; then
	PerfType=$(${AWK} '/AUTODEPLOY_PERFTYPE/ {print $2;}' ${OSSPI_CFG})
	log_trace "Using performance product deployment key set in ${OSSPI_CFG}
file..."
fi
if [[ -z ${PerfType} ]]; then
	log_trace "Variable AUTODEPLOY_PERFTYPE variable not set in config file
${OSSPI_CFG}"
	log_trace "Detecting performance product installed on node..."
	PerfType=$(Discover_PerfTool)
	res=$?
	log_trace "Discover_PerfTool returned performance product type as
${PerfType:-[null]}"
	if [[ ${res} -ne 0 || -z ${PerfType} ]]; then
		PerfType="NP"
		log_trace "Performance product type is set to ${PerfType}"
	fi
fi
perfstr="$(xmltag_attribute perfTool ${PerfType})"
############################################################################
####
# MAIN Function
############################################################################
####
# this is a wrapper for the original format tool
# it wrapps the interactiv properties of format
_format() {
  rm -f ${OSSPI_TMP_DIR}/format_cmd.tmp 2> /dev/null
  D=$1
  shift
  for CMD in $@; do
    echo ${CMD} >> ${OSSPI_TMP_DIR}/format_cmd.tmp
  done
  echo "quit" >> ${OSSPI_TMP_DIR}/format_cmd.tmp
  format -d ${D} < ${OSSPI_TMP_DIR}/format_cmd.tmp 2> /dev/null
  rm -f ${OSSPI_TMP_DIR}/format_cmd.tmp 2> /dev/null
}
RETVAL=1
# create needed OSSPI directories
[[ -d ${OSSPI_TMP_DIR} ]] || mkdir -m 0750 -p ${OSSPI_TMP_DIR}
[[ -d ${OSSPI_VAR_CFG_DIR} ]] || mkdir -m 0750 -p ${OSSPI_VAR_CFG_DIR}
# make sure temporary files are empty
rm -f ${OSSPI_DISK_CFG}.tmp >/dev/null 2>/dev/null
rm -f ${OSSPI_DEFECT_CFG}.short.tmp >/dev/null 2>/dev/null
rm -f ${OSSPI_DEFECT_CFG}.short.tmp >/dev/null 2>/dev/null
touch ${OSSPI_DEFECT_CFG}.long
touch ${OSSPI_DEFECT_CFG}.short
####################################################################
#
# Main
#
xmltag_start_module
PDD_BF="`osspi_ovcatgets -c osspi_scripts -m 87 -t \"Physical Disk
Drives\"`"
DISK_BF="`osspi_ovcatgets -c osspi_scripts -m 116 -t \"Disk\"`"
PART_BF="`osspi_ovcatgets -c osspi_scripts -m 117 -t \"Part\"`"
xmltag_service "${OSSPI_SVC_PHYDISK}" "${OSSPI_CLS_PHYDISK}" "${perfstr}"
xmltag_composition "${OSSPI_SVC_PHYDISK}" "${OSSPI_SVC_NODE}" ""
"${OSSPI_CLS_NODE}"
echo
RETVAL=1
trap "rm -f ${OSSPI_TMP_DIR}/def.tmp" EXIT INT QUIT
typeset -i diskser=0
# scan raw devices in the dev directory for candidates
ls /dev/rdsk/*0 | while read DEVPATH; do
  `prtvtoc ${DEVPATH} 2>/dev/null >/dev/null`
  if [[ $? != 0 ]]; then  # EMC Disk or CDROM ?
     log_trace 2 "Module phydisk: cannot inspect device ${DEVPATH}
skipping."
     continue
  fi
  TMP=${DEVPATH#/dev/rdsk/}
  DEV=${TMP%s0}
  ATTRIB=""
  hwpath=""
  RETVAL=0
  # check if device is a disk and than
  # scan for inquery to find SCSI devices
  # be aware that format is an interactive tool
  BUFFER=`_format ${DEV}`
  [[ -z `echo ${BUFFER} | ${GREP} "selecting"` ]] && continue;
  if [[ ! -z `echo ${BUFFER} | ${GREP} inquiry` ]]; then
    CTRLER="scsi"
    # write defects configuration for defect monitoring
    osspi_defect ${DEVPATH} 2> /dev/null > ${OSSPI_TMP_DIR}/def.tmp
    if [[ $? -eq 0 ]]; then
      read T1 NSECTS T2 NTRACKS < ${OSSPI_TMP_DIR}/def.tmp
      ENTRY=`${GREP} "^${DEV} " ${OSSPI_DEFECT_CFG}.short 2>/dev/null`
      if [[ $? -eq 0 ]]; then
	echo "${ENTRY}" >> ${OSSPI_DEFECT_CFG}.short.tmp
	ENTRY=`${GREP} "^${DEV} " ${OSSPI_DEFECT_CFG}.long 2>/dev/null`
	echo "${ENTRY}" >> ${OSSPI_DEFECT_CFG}.long.tmp
      else
	echo "${DEVPATH} ${NSECTS} ${NTRACKS}" >> ${OSSPI_DEFECT_CFG}.short.tmp
	echo "${DEVPATH} ${NSECTS} ${NTRACKS}" >> ${OSSPI_DEFECT_CFG}.long.tmp
      fi
    fi
    echo "${DEV}s* ${DEV}s* ${DEVPATH}" >> ${OSSPI_DISK_CFG}.tmp
  else
    CTRLER="ide"
    hwpath=`_format ${DEV} current | ${GREP} 'ide@'`
    echo "dad${diskser} ${DEV}s* ${DEVPATH}" >> ${OSSPI_DISK_CFG}.tmp
    diskser=`expr ${diskser} + 1`
  fi
  DISKTMP="${DISK_BF} ${DEVPATH}"
  TMP=`xmltag_attribute subtag ${DEVPATH} label "${DISKTMP}" controller
"${CTRLER}" ${hwpath:+hwpath ${hwpath}}`
  xmltag_service "${OSSPI_SVC_PHYDISK}:${DEVPATH}"
"${OSSPI_CLS_PHYDISK}_sub" \
		 "${TMP}"
  xmltag_composition "${OSSPI_SVC_PHYDISK}:${DEVPATH}"
"${OSSPI_SVC_PHYDISK}"
  echo
  # we have found a disk now
  # search partitions in this disk
  prtvtoc -h ${DEVPATH} | while read ID TAG FLAGS F C L MP; do
   ATTRIB=""
   SIZE=`echo ${C} 512 \* p quit | dc`
   RO=false
   [[ ${FLAGS%?} == "1" ]] && RO=true
   MOUNTABLE=false
   [[ ${FLAGS#?} == "0" ]] && MOUNTABLE=true
# _format ${DEV} partition print quit | ${GREP} -E
"^[[:space:]]+[[:digit:]]+[[:space:]]+(stand|var|home|usr|swap|root|backup)"
| while read ID TAG FLAG CYL SIZE; do
    PARTPATH="/dev/rdsk/${DEV}s${ID}"
    TMP=`xmltag_attribute size ${SIZE}`
    ATTRIB="${ATTRIB}\n${TMP}"
    TMP=`xmltag_attribute ro  ${RO}`
    ATTRIB="${ATTRIB}\n${TMP}"
    TMP=`xmltag_attribute mountable  ${MOUNTABLE}`
    ATTRIB="${ATTRIB}\n${TMP}"
    PARTTMP="${PART_BF} ${DEV}s${ID}"
    TMP=`xmltag_attribute subtag ${PARTPATH} label "${PARTTMP}" size ${SIZE}
ro ${RO} mountable ${MOUNTABLE}`
    xmltag_service "${OSSPI_SVC_NODE}:lsu:${PARTPATH}"
"${OSSPI_CLS_PHYDISK}_sub2" \
		   "${TMP}"
    xmltag_composition "${OSSPI_SVC_NODE}:lsu:${PARTPATH}"
"${OSSPI_SVC_PHYDISK}:${DEVPATH}"
  done
done
# move temporary files in place
mv -f ${OSSPI_DEFECT_CFG}.short.tmp ${OSSPI_DEFECT_CFG}.short 2>/dev/null
mv -f ${OSSPI_DEFECT_CFG}.long.tmp ${OSSPI_DEFECT_CFG}.long 2>/dev/null
mv -f ${OSSPI_DISK_CFG}.tmp ${OSSPI_DISK_CFG} 2>/dev/null
xmltag_end_module
[[ ${RETVAL} -eq 1 ]] && log_trace 4 "Module phydisk: Nothing to discover."
exit ${RETVAL}
# End
****************************************************************************
**
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers


This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:30:18 EDT