Help to debug a problem

From: Vaddadi, Sridhar (svaddadi@road-inc.com)
Date: Wed Jan 22 2003 - 22:07:56 EST


Hi Managers,

        I am seeing a process in "top" (PID 18193) , which seems to be
oracle process but not found in v$session & v$process tables in Oracle.
Truss/lsof will not show any details (completely blank). If we kill this
process, we find few more within few hours. Can someone please check the
enclosed details of the process and see what it could be up to or some clues
on how to debug this problem.

Thanks in advance for the help.

Will summarize.

Sridhar

        
#top

load averages: 1.45, 1.08, 0.64
02:30:43
294 processes: 292 sleeping, 2 on cpu
CPU states: 16.3% idle, 81.8% user, 1.9% kernel, 0.0% iowait, 0.0% swap
Memory: 2048M real, 318M free, 2523M swap in use, 3156M swap free

  PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
18193 oracle 1 10 0 30M 12M cpu1 8:43 48.63% oracle

# pcred 18193
18193: e/r/suid=1001 e/r/sgid=101

# pfiles 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:32,120 ino:97439 uid:0 gid:0 rdev:24,1
      O_RDWR|O_LARGEFILE
   1: S_IFCHR mode:0666 dev:32,120 ino:97439 uid:0 gid:0 rdev:24,1
      O_RDWR|O_LARGEFILE
   2: S_IFCHR mode:0666 dev:32,120 ino:97439 uid:0 gid:0 rdev:24,1
      O_RDWR|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:32,120 ino:97644 uid:0 gid:3 rdev:13,2
      O_RDONLY
   4: S_IFSOCK mode:0666 dev:259,0 ino:9650 uid:0 gid:0 size:0
      O_RDWR
        sockname: AF_INET 0.0.0.0 port: 0
   5: S_IFCHR mode:0666 dev:32,120 ino:97644 uid:0 gid:3 rdev:13,2
      O_RDONLY
   6: S_IFCHR mode:0666 dev:32,120 ino:97644 uid:0 gid:3 rdev:13,2
      O_RDONLY
   7: S_IFCHR mode:0666 dev:32,120 ino:97644 uid:0 gid:3 rdev:13,2
      O_RDONLY
   8: S_IFDOOR mode:0444 dev:264,0 ino:57516 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[234]

# pflags 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
        data model = _ILP32 flags = PR_ORPHAN
  /1: flags = PR_PCINVAL

#
# pldd 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
/oracle/app/product/8.1.6/lib/libskgxp8.so
/oracle/app/product/8.1.6/lib/libwtc8.so
/oracle/app/product/8.1.6/lib/libjox8.so
/oracle/app/product/8.1.6/lib/libdsbtsh8.so
/usr/lib/libnsl.so.1
/usr/lib/libsocket.so.1
/usr/lib/libgen.so.1
/usr/lib/libdl.so.1
/usr/lib/libsched.so.1
/usr/lib/libc.so.1
/usr/lib/libaio.so.1
/usr/lib/librt.so.1
/usr/lib/libkstat.so.1
/usr/lib/libm.so.1
/usr/lib/libmp.so.2
/usr/platform/sun4u/lib/libc_psr.so.1
#
#
# pmap 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
00010000 23320K read/exec /oracle/app/product/8.1.6/bin/oracle
016E4000 216K read/write/exec /oracle/app/product/8.1.6/bin/oracle
0171A000 280K read/write/exec [ heap ]
FEB00000 680K read/exec /usr/lib/libc.so.1
FEBBA000 32K read/write/exec /usr/lib/libc.so.1
FEBD0000 8K read/write/exec [ anon ]
FEBE0000 16K read/exec /usr/lib/libmp.so.2
FEBF4000 8K read/write/exec /usr/lib/libmp.so.2
FEC00000 5224K read/exec /oracle/app/product/8.1.6/lib/libjox8.so
FF128000 184K read/write/exec /oracle/app/product/8.1.6/lib/libjox8.so
FF156000 8K read/write/exec /oracle/app/product/8.1.6/lib/libjox8.so
FF160000 16K read/exec /usr/platform/sun4u/lib/libc_psr.so.1
FF170000 88K read/exec /usr/lib/libm.so.1
FF194000 8K read/write/exec /usr/lib/libm.so.1
FF1A0000 8K read/exec /usr/lib/libkstat.so.1
FF1B2000 8K read/write/exec /usr/lib/libkstat.so.1
FF1C0000 24K read/exec /usr/lib/librt.so.1
FF1D6000 8K read/write/exec /usr/lib/librt.so.1
FF1E0000 32K read/exec /usr/lib/libaio.so.1
FF1F8000 8K read/write/exec /usr/lib/libaio.so.1
FF200000 8K read/exec /usr/lib/libsched.so.1
FF212000 8K read/write/exec /usr/lib/libsched.so.1
FF220000 8K read/write/exec/shared [ anon ]
FF230000 8K read/write/exec [ anon ]
FF240000 24K read/exec /usr/lib/libgen.so.1
FF256000 8K read/write/exec /usr/lib/libgen.so.1
FF260000 40K read/exec /usr/lib/libsocket.so.1
FF27A000 8K read/write/exec /usr/lib/libsocket.so.1
FF280000 560K read/exec /usr/lib/libnsl.so.1
FF31C000 32K read/write/exec /usr/lib/libnsl.so.1
FF324000 32K read/write/exec /usr/lib/libnsl.so.1
FF330000 32K read/exec
/oracle/app/product/8.1.6/lib/libdsbtsh8.so
FF346000 8K read/write/exec
/oracle/app/product/8.1.6/lib/libdsbtsh8.so
FF348000 8K read/write/exec
/oracle/app/product/8.1.6/lib/libdsbtsh8.so
FF350000 8K read/exec /oracle/app/product/8.1.6/lib/libwtc8.so
FF360000 8K read/write/exec /oracle/app/product/8.1.6/lib/libwtc8.so
FF370000 8K read/exec
/oracle/app/product/8.1.6/lib/libskgxp8.so
FF380000 8K read/write/exec
/oracle/app/product/8.1.6/lib/libskgxp8.so
FF390000 8K read/exec /usr/lib/libdl.so.1
FF3A0000 8K read/write/exec [ anon ]
FF3B0000 152K read/exec /usr/lib/ld.so.1
FF3E6000 8K read/write/exec /usr/lib/ld.so.1
FFBE4000 48K read/write/exec [ stack ]
 total 31216K
#
#
# psig 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
HUP ignored
INT ignored
QUIT caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
ILL caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
TRAP caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
ABRT caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
EMT default
FPE caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
KILL default
BUS caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
SEGV caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
SYS caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
PIPE ignored
ALRM caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
TERM ignored
USR1 ignored
USR2 caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
CLD default NOCLDSTOP
PWR default
WINCH caught SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
URG default
POLL default
STOP default
TSTP default
CONT default
TTIN default
TTOU default
VTALRM ignored
PROF ignored
XCPU caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
XFSZ caught RESTART,SIGINFO
HUP,INT,QUIT,TRAP,EMT,PIPE,ALRM,TERM,USR1,USR2,CLD,PWR,WINCH,URG,POLL,TSTP,C
ONT,TTIN,TTOU,VTALRM,PROF,WAITING,LWP,FREEZE,THAW,CANCEL,LOST,RTMIN,RTMIN+1,
RTMIN+2,RTMIN+3,RTMAX-3,RTMAX-2,RTMAX-1,RTMAX
WAITING default
LWP default
FREEZE default
THAW default
CANCEL default
LOST default
RTMIN ignored
RTMIN+1 ignored
RTMIN+2 ignored
RTMIN+3 ignored
RTMAX-3 ignored
RTMAX-2 ignored
RTMAX-1 ignored
RTMAX ignored
#
#
# pstack 18193
18193: oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
 00ce2ee4 kghrcdepth (1, 3fffffc, ce1c30, 0, 10000000, 1746cac) + 58
 00ce2fdc kghrcappl (16f1d94, 16f1c68, ce1c30, 0, 173e714, 10000000) + 3c
 001409e0 ksmdpg (16f1c64, 0, 16f3800, 16f1c00, 16f1c68, 16f3000) + 90
 000ea46c opidcl (145f400, 16f1c00, 16f3400, ffbef21c, 16f1e08, 0) + 280
 000ea110 opidrv (16f4954, ffbef2e0, 16f1e08, 2f6c6f67, 0, 0) + 624
 000e9048 sou2o (ffbef5c0, 3c, 4, ffbef5b0, c3a0, c000) + 14
 000e4b8c main (0, ffbef76c, 17137f0, 16f1800, 0, 0) + a0
 000e4ad4 _start (0, 0, 0, 0, 0, 0) + dc
#
# ptime 18193
ptime: exec failed

real 0.003
user 0.000
sys 0.001
#
#
# ptree 18193
18193 oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
#
#
# pwdx 18193
18193: /oracle/app/product/8.1.6/dbs
#
#
# ptree -a 18193
1 /etc/init -
  18193 oracleatroad (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
#
#
#

Sridhar Vaddadi
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:25:39 EDT