From: Martin.Rodger@intier.com
Date: Wed Jan 02 2008 - 07:09:12 EST
Hi there,
Would appreciate some help if possible.
Had a system panic at 16:06 on 24/12. System rebooted and has been fine
since.
To my knowledge system would have been idle all day.
Results of dbx:
# dbx -k vmunix.1 vmzcore.1
dbx version 5.1
Type 'help' for help.
stopped at [boot:2683 ,0xfffffc000068edfc] Source not available
warning: Files compiled -g3: parameter values probably wrong
(dbx)
(dbx) p panicstr
0xfffffc00008dd130 = "simple_lock: time limit exceeded"
(dbx) t
> 0 boot(reason = (unallocated - symbol optimized away), howto =
(unallocated - symbol optimized away)) ["../../../../src/kernel/ar
ch/alpha/machdep.c":2683, 0xfffffc000068edfc]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1401, 0xfffffc00002b0490]
2 simple_lock_fault(slp = (unallocated - symbol optimized away), state =
(unallocated - symbol optimized away), caller = (unalloc
ated - symbol optimized away), arg = (unallocated - symbol optimized away),
fmt = (unallocated - symbol optimized away), error = (un
allocated - symbol optimized away))
["../../../../src/kernel/kern/lock.c":2841, 0xfffffc00002d7174]
3 simple_lock_time_violation(slp = (unallocated - symbol optimized
away), state = (unallocated - symbol optimized away), caller =
(unallocated - symbol optimized away))
["../../../../src/kernel/kern/lock.c":2974, 0xfffffc00002d7588]
4 thread_list_wakeup(whead = (unallocated - symbol optimized away), s =
(unallocated - symbol optimized away), result = (unalloca
ted - symbol optimized away))
["../../../../src/kernel/kern/sched_prim.c":5756, 0xfffffc00002eead4]
5 event_post(0x1, 0x0, 0xfffffc01af90c800, 0xfffffc01af90c800,
0xfffffc00002b1b80) ["../../../../src/kernel/kern/event.c":171, 0x
fffffc00002c89d0]
6 select_wakeup(0xfffffc000ef2c5c0, 0xfffffc000ef2c540, 0x0, 0x0,
0xfffffc01d9ff1980) ["../../../../src/kernel/bsd/subr_select.c"
:410, 0xfffffc00002b1b80]
7 sowakeup(0xfffffc01d9ff1980, 0x0, 0xfffffc000030a9c4,
0xfffffc0096891180, 0xfffffc012efb7e00) ["../../../../src/kernel/bsd/uipc
_socket2.c":1513, 0xfffffc000025b058]
8 uipc_usrreq(0x0, 0xfffffe0638ce77e0, 0x0, 0x7f10, 0x0)
["../../../../src/kernel/bsd/uipc_usrreq.c":617, 0xfffffc000030a9c0]
More (n if no)?
9 sosend(0x1, 0xfffffc0096891280, 0x9, 0x0, 0xfffffc01f849f100)
["../../../../src/kernel/bsd/uipc_socket.c":3121, 0xfffffc0000257
c80]
10 rwuio(0xfffffe0638ce0000, 0xfffffc016715fc00, 0xfffffc01ffede180,
0xfffffe0638ce78f0, 0x1) ["../../../../src/kernel/bsd/sys_gen
eric.c":2292, 0xfffffc00002c47f4]
11 _F64_writev(0xfffffc0000276cc7, 0xfffffc0000000001, 0x0, 0x0,
0xfffffc0000675ee0) ["../../../../src/kernel/bsd/sys_generic.c":1
978, 0xfffffc00002c40cc]
12 syscall(0x2, 0x14004bfb8, 0xd, 0x120022100, 0x0)
["../../../../src/kernel/arch/alpha/syscall_trap.c":725,
0xfffffc0000672940]
13 _Xsyscall(0x8, 0x3ff801374f8, 0x14000e7b0, 0x11, 0x14003ec20)
["../../../../src/kernel/arch/alpha/locore.s":1814, 0xfffffc00006
7611c]
(dbx)
Its an ES40 running Tru64 Unix 5.1a Patch kit 6.
Will monitor for re-occurrence assumed a "one off" incident as present.
Any ideas on root cause?
Regards
Martin Rodger
IT Manager
Intier Automotive, Interiors Ltd. (Redditch Divisions)
Tel: 01527 548523, Fax: 01527 548650, Mob: 07740 442990
Crash Data Log
#
# Crash Data Collection (Version 1.4)
#
_crash_data_collection_time: Mon Dec 24 16:07:17 GMT 2007
_current_directory: /
_crash_kernel: /var/adm/crash/vmunix.1
_crash_core: /var/adm/crash/vmzcore.1
_crash_arch: alpha
_crash_os: Compaq Tru64 UNIX
_host_version: V5.1A (Rev. 1885)
Compaq Tru64 UNIX V5.1A (Rev. 1885); Tue Mar 15 12:26:09 GMT 2005
_crash_version: V5.1A (Rev. 1885)
Compaq Tru64 UNIX V5.1A (Rev. 1885); Tue Mar 15 12:26:09 GMT 2005
warning: Files compiled -g3: parameter values probably wrong
_crashtime: struct {
tv_sec = 1198512224
tv_usec = 943148
}
_boottime: struct {
tv_sec = 1198368390
tv_usec = 822665
}
_config: struct {
sysname = "OSF1"
nodename = "rimmer"
release = "V5.1"
version = "1885"
machine = "alpha"
}
_cpu: 57
_system_string: 0xffffffffff92b030 = "AlphaServer ES40"
_ncpus: 4
_avail_cpus: 4
_partial_dump: 1
_physmem(MBytes): 8191
_panic_string: 0xfffffc00008dd130 = "simple_lock: time limit exceeded"
_paniccpu: 0
_panic_thread: 0xfffffc016715fc00
_preserved_message_buffer_begin:
struct {
hdr = struct {
msg_magic = 0x880524
msg_bufx = 0xd20
msg_bufr = 0xbbb
msg_size = 0x7ffe0
}
msg_bufc = "Memory trolling not supported, cpu Major id 8, Minor id 7
Alpha boot: available memory from 0x9b78000 to 0x1fffe4000
Compaq Tru64 UNIX V5.1A (Rev. 1885); Tue Mar 15 12:26:09 GMT 2005
physical memory = 8192.00 megabytes.
available memory = 8031.75 megabytes.
using 10468 buffers containing 81.78 megabytes of memory
Master cpu at slot 0
Starting secondary cpu 1
Starting secondary cpu 2
Starting secondary cpu 3
Firmware revision: 6.3-2
PALcode: UNIX version 1.90-4
AlphaServer ES40
pci1 (primary bus:1) at nexus
ee0 at pci1 slot 2
ee0: COMPAQ Intel 82559 (10/100 Mbps) Ethernet Interface
ee0: Driver Rev = V1.0.23, Chip Rev = 8, hardware address:
00-50-8B-AE-60-DE
isp0 at pci1 slot 3
isp0: QLOGIC ISP1040B/V2
isp0: Firmware revision 5.57 (loaded by console)
isp0: Fast RAM timing enabled.
ee0: Autonegotiated, 100 Mbps full duplex
scsi0 at isp0 slot 0 rad 0
emx0 at pci1 slot 5
KGPSA-CA : Driver Rev 2.11 : F/W Rev 3.03X2(1.11) : wwn 1000-0000-c925-7509
emx0: Using console topology setting of : Fabric
scsi1 at emx0 slot 0 rad 0
pci0 (primary bus:0) at nexus
comet0: Card type 'Elsa GLoria' with 8MB framebuffer memory.
comet0 at pci0 slot 1
isp1 at pci0 slot 2
isp1: QLOGIC ISP1040B/V2
isp1: Firmware revision 5.57 (loaded by console)
isp1: Fast RAM timing enabled.
scsi6 at isp1 slot 0 rad 0
ee1 at pci0 slot 3
ee1: COMPAQ Intel 82559 (10/100 Mbps) Ethernet Interface
ee1: Driver Rev = V1.0.23, Chip Rev = 8, hardware address:
00-50-8B-AE-62-F2
emx2 at pci0 slot 4
KGPSA-CA : Driver Rev 2.11 : F/W Rev 3.81A1(2.01A0) : wwn
1000-0000-c928-1190
emx2: Using console topology setting of : Fabric
scsi3 at emx2 slot 0 rad 0
isa0 at pci0
gpc0 at isa0
gpc1 not probed
ace0 at isa0
ace1 at isa0
lp0 at isa0
fdi0 at isa0
fd0 at fdi0 unit 0
ee1: Autonegotiated, 100 Mbps full duplex
ata0 at pci0 slot 15
ata0: ACER M1543C
scsi4 at ata0 slot 0 rad 0
scsi5 at ata0 slot 1 rad 0
usb0 at pci0 slot 19
Created FRU table binary error log packet
kernel console: ace0
dli: configured
NetRAIN configured.
ATM Subsystem configured with 4 restart threads
ATMUNI: configured
ATMSIG: 3.x (module=uni3x) configured
ILMI: 3.x (module=ilmi) configured
ATM IP: configured
ATM LANE: configured.
ATM IFMP: configured
vm_swap_init: swap is set to eager allocation mode
Environmental Monitoring Subsystem Configured.
SuperLAT. Copyright 1994 Meridian Technology Corp. All rights reserved.
<4>lsm:volio: Illegal vminor encountered<4>lsm:volio: Illegal vminor
encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio: Illegal
vminor encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio:
Illegal vminor encountered<4>lsm:volio: Illegal vminor
encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio: Illegal
vminor encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio:
Illegal vminor encountered<4>lsm:volio: Illegal vminor
encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio: Illegal
vminor encountered<4>lsm:volio: Illegal vminor encountered<4>lsm:volio:
Illegal vminor encountered
simple_lock: time limit exceeded
pc of caller: 0xfffffc00002eead4
lock address: 0xfffffc01af90c700
lock info addr: 0xfffffc0000d36400
lock class name: thread.lock
current lock state: 0xd400023e002c8a61
(cpu=?,pc=0xfffffc00002c8a60,busy)
panic (cpu 0): simple_lock: time limit exceeded
syncing disks... 9 9"
}
_preserved_message_buffer_end:
_kernel_process_status_begin:
PID COMM
00000 kernel idle
00001 init
00016 kloadsrv
00018 hotswapd
00107 esmd
00118 update
00234 evmd
00268 evmlogger
00269 evmchmgr
00416 niffd
00454 syslogd
00458 binlogd
00545 portmap
00548 nfsiod
00730 xntpd
00743 sendmail
00767 snmpd
00779 svrSystem_mib
00784 svrMgt_mib
00785 os_mibs
00786 pmgrd
00788 cpq_mibs
00823 cpqthresh_mib
00828 steamd
00842 insightd
00929 envmond
00932 rcmgr
00942 config_hmmod
00952 advfsd
00955 sysman_hmmod
00965 inetd
00966 inetd
00977 cron
01013 lpd
01037 smbd
01039 nmbd
01083 smbd
01117 smbd
01167 smbd
01523 iigcn
01609 iimerge
02022 iimerge
02038 iimerge
02138 iigcc
02224 iigcc
02308 ladebug
02309 ladebug
02310 ladebug
02339 dtlogin
02390 java
02419 Xdec
02421 dtlogin
02422 smsd
02423 getty
02438 dtgreet
02521 smbd
462976 sleep
463922 misclog_get
463936 evmsort
463953 evmshow
463954 evmmail.cron
463962 sh
463964 evmget_srv
463965 evmmail.cron
463973 evmget
463975 sh
464030 grep
464046 evmshow
464049 merge_template
464061 misclog_get
464092 cat
464097 text2evm
464105 awk
464334 sleep
464591 sleep
464767 sleep
464804 sleep
464813 sleep
464820 smbd
464847 sleep
23450 monitor_db_event
23627 sap.listen
23683 spabautopack.lis
24929 sh
24931 landrover.listen
26624 sh
26625 monitor_db_event
27942 sh
27944 edi.listen
28367 sh
28370 edi.listen
28639 sap.listen
358653 alerter
358659 sh
256380 collect
_kernel_process_status_end:
_current_pid: 234
_current_tid: 0xfffffc016715fc00
_proc_thread_list_begin:
thread 0xfffffc016715fc00 stopped at [boot:2683 ,0xfffffc000068edfc]
Source not available
_proc_thread_list_end:
_dump_begin:
> 0 boot(reason = (unallocated - symbol optimized away), howto =
(unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/machdep.c":2683, 0xfffffc000068edfc]
mp = (unallocated - symbol optimized away)
nmp = 0x1
sp = (unallocated - symbol optimized away)
vp = (unallocated - symbol optimized away)
error = (unallocated - symbol optimized away)
mycpu = 0
rs = 0
thread = (nil)
rpb_ptr = (nil)
rpb_cpu = (nil)
item_list = struct {
function = 0
out_flags = 2919009
in_flags = 3556770366
rtn_status = 18446739675673324920
next_function = 0xfffffc00008dc248
input_data = 18446739681687501824
output_data = 0
}
platform_reset = 0xfffffc000087b950
print_halting_after_panic = 0
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1401, 0xfffffc00002b0490]
i = 0
mycpu = 0
saved_spl = 7
pcpu = 0x3e9
sync_mode = 8894800
2 simple_lock_fault(slp = (unallocated - symbol optimized away), state =
(unallocated - symbol optimized away), caller = (unallocated - symbol
optimized away), arg = (unallocated - symbol optimized away), fmt =
(unallocated - symbol optimized away), error = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/lock.c":2841,
0xfffffc00002d7174]
saved_rpcc = 758010550173857852
saved_lbolt = 147303199
saved_spl = 4
i = (unallocated - symbol optimized away)
lip = (nil)
fmtbuf = "cpu=?,pc=0xfffffc00002c8a60,busy"
3 simple_lock_time_violation(slp = (unallocated - symbol optimized
away), state = (unallocated - symbol optimized away), caller = (unallocated
- symbol optimized away)) ["../../../../src/kernel/kern/lock.c":2974,
0xfffffc00002d7588]
4 thread_list_wakeup(whead = (unallocated - symbol optimized away), s =
(unallocated - symbol optimized away), result = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/sched_prim.c":5756,
0xfffffc00002eead4]
state = 6
need_swapin_wakeup = 0
thread = 0xfffffc01af90c700
qe = 0xfffffe0638ce75d0
cur_thread_wakeup = 0
cur_thread = 0xfffffc016715fc00
5 event_post(0x1, 0x0, 0xfffffc01af90c800, 0xfffffc01af90c800,
0xfffffc00002b1b80) ["../../../../src/kernel/kern/event.c":171,
0xfffffc00002c89d0]
6 select_wakeup(0xfffffc000ef2c5c0, 0xfffffc000ef2c540, 0x0, 0x0,
0xfffffc01d9ff1980) ["../../../../src/kernel/bsd/subr_select.c":410,
0xfffffc00002b1b80]
7 sowakeup(0xfffffc01d9ff1980, 0x0, 0xfffffc000030a9c4,
0xfffffc0096891180, 0xfffffc012efb7e00)
["../../../../src/kernel/bsd/uipc_socket2.c":1513, 0xfffffc000025b058]
8 uipc_usrreq(0x0, 0xfffffe0638ce77e0, 0x0, 0x7f10, 0x0)
["../../../../src/kernel/bsd/uipc_usrreq.c":617, 0xfffffc000030a9c0]
9 sosend(0x1, 0xfffffc0096891280, 0x9, 0x0, 0xfffffc01f849f100)
["../../../../src/kernel/bsd/uipc_socket.c":3121, 0xfffffc0000257c80]
10 rwuio(0xfffffe0638ce0000, 0xfffffc016715fc00, 0xfffffc01ffede180,
0xfffffe0638ce78f0, 0x1) ["../../../../src/kernel/bsd/sys_generic.c":2292,
0xfffffc00002c47f4]
11 _F64_writev(0xfffffc0000276cc7, 0xfffffc0000000001, 0x0, 0x0,
0xfffffc0000675ee0) ["../../../../src/kernel/bsd/sys_generic.c":1978,
0xfffffc00002c40cc]
12 syscall(0x2, 0x14004bfb8, 0xd, 0x120022100, 0x0)
["../../../../src/kernel/arch/alpha/syscall_trap.c":725,
0xfffffc0000672940]
13 _Xsyscall(0x8, 0x3ff801374f8, 0x14000e7b0, 0x11, 0x14003ec20)
["../../../../src/kernel/arch/alpha/locore.s":1814, 0xfffffc000067611c]
_dump_end:
_kernel_thread_list_begin:
thread 0xfffffc01ffe0aa80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0a700 stopped at [boot:2930 ,0xfffffc000068f3e0]
Source not available
thread 0xfffffc01ffe0a380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0b500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0b180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0ae00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0bc00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe0b880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe20000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe20a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe20700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe20380 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
thread 0xfffffc01ffe21500 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
thread 0xfffffc01ffe21180 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
thread 0xfffffc01ffe20e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe21c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe21880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9152000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9152a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9152700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9152380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9153500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9153180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9152e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9153c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f9153880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe96000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe96a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe96700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe96380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01ffe96e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f870e700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f870e380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f870f500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f870f180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f870f880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8538000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8538a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8539500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8539180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8538e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8430000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8430a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8431500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8431180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8430e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8459c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8459880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8498000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8498a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8498700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8498380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8499500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8499180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8498e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8499c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8499880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a4000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a4a80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a4700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a4380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a5500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a5180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a4e00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a5c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84a5880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84baa80 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84ba700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84ba380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84bb500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84bb180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84bae00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84bbc00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84bb880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84ca000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84ca700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84ca380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84cb500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84cb180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84cae00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f84cbc00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7dbe000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7dbe700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7dbe380 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7dbf500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7e89880 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7e99180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc012dcba700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc016715e000 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc016715f180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7e99c00 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f7e99500 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8459180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc01f8538700 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
thread 0xfffffc00b3bf9180 stopped at [thread_block:3230
,0xfffffc00002eab10] Source not available
_kernel_thread_list_end:
_savedefp: (nil)
_kernel_memory_fault_data_begin:
struct {
fault_va = 0x0
fault_pc = 0x0
fault_ra = 0x0
fault_sp = 0x0
access = 0x0
status = 0x0
cpunum = 0x0
count = 0x0
pcb = (nil)
thread = (nil)
task = (nil)
proc = (nil)
}
_kernel_memory_fault_data_end:
_uptime: 39.95 hours
warning: Files compiled -g3: parameter values probably wrong
paniccpu: 0x0
machine_slot[paniccpu]: struct {
is_cpu = 0x1
cpu_type = 0xf
cpu_subtype = 0x22
running = 0x1
cpu_ticks = {
[0] 0x0
[1] 0x0
[2] 0x0
[3] 0x0
[4] 0x0
}
clock_freq = 0x400
error_restart = 0x0
cpu_panicstr = 0xfffffc00008dd130 = "simple_lock: time limit exceeded"
cpu_panic_thread = 0xfffffc016715fc00
}
tset machine_slot[paniccpu].cpu_panic_thread:
Begin Trace for machine_slot[paniccpu].cpu_panic_thread:
> 0 boot(reason = (unallocated - symbol optimized away), howto =
(unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/machdep.c":2683, 0xfffffc000068edfc]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1401, 0xfffffc00002b0490]
2 simple_lock_fault(slp = (unallocated - symbol optimized away), state =
(unallocated - symbol optimized away), caller = (unallocated - symbol
optimized away), arg = (unallocated - symbol optimized away), fmt =
(unallocated - symbol optimized away), error = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/lock.c":2841,
0xfffffc00002d7174]
3 simple_lock_time_violation(slp = (unallocated - symbol optimized
away), state = (unallocated - symbol optimized away), caller = (unallocated
- symbol optimized away)) ["../../../../src/kernel/kern/lock.c":2974,
0xfffffc00002d7588]
4 thread_list_wakeup(whead = (unallocated - symbol optimized away), s =
(unallocated - symbol optimized away), result = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/sched_prim.c":5756,
0xfffffc00002eead4]
5 event_post(0x1, 0x0, 0xfffffc01af90c800, 0xfffffc01af90c800,
0xfffffc00002b1b80) ["../../../../src/kernel/kern/event.c":171,
0xfffffc00002c89d0]
6 select_wakeup(0xfffffc000ef2c5c0, 0xfffffc000ef2c540, 0x0, 0x0,
0xfffffc01d9ff1980) ["../../../../src/kernel/bsd/subr_select.c":410,
0xfffffc00002b1b80]
7 sowakeup(0xfffffc01d9ff1980, 0x0, 0xfffffc000030a9c4,
0xfffffc0096891180, 0xfffffc012efb7e00)
["../../../../src/kernel/bsd/uipc_socket2.c":1513, 0xfffffc000025b058]
8 uipc_usrreq(0x0, 0xfffffe0638ce77e0, 0x0, 0x7f10, 0x0)
["../../../../src/kernel/bsd/uipc_usrreq.c":617, 0xfffffc000030a9c0]
9 sosend(0x1, 0xfffffc0096891280, 0x9, 0x0, 0xfffffc01f849f100)
["../../../../src/kernel/bsd/uipc_socket.c":3121, 0xfffffc0000257c80]
10 rwuio(0xfffffe0638ce0000, 0xfffffc016715fc00, 0xfffffc01ffede180,
0xfffffe0638ce78f0, 0x1) ["../../../../src/kernel/bsd/sys_generic.c":2292,
0xfffffc00002c47f4]
11 _F64_writev(0xfffffc0000276cc7, 0xfffffc0000000001, 0x0, 0x0,
0xfffffc0000675ee0) ["../../../../src/kernel/bsd/sys_generic.c":1978,
0xfffffc00002c40cc]
12 syscall(0x2, 0x14004bfb8, 0xd, 0x120022100, 0x0)
["../../../../src/kernel/arch/alpha/syscall_trap.c":725,
0xfffffc0000672940]
13 _Xsyscall(0x8, 0x3ff801374f8, 0x14000e7b0, 0x11, 0x14003ec20)
["../../../../src/kernel/arch/alpha/locore.s":1814, 0xfffffc000067611c]
End Trace for machine_slot[paniccpu].cpu_panic_thread:
warning: Files compiled -g3: parameter values probably wrong
_stack_trace[0]_begin:
> 0 boot(reason = (unallocated - symbol optimized away), howto =
(unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/machdep.c":2683, 0xfffffc000068edfc]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1401, 0xfffffc00002b0490]
2 simple_lock_fault(slp = (unallocated - symbol optimized away), state =
(unallocated - symbol optimized away), caller = (unallocated - symbol
optimized away), arg = (unallocated - symbol optimized away), fmt =
(unallocated - symbol optimized away), error = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/lock.c":2841,
0xfffffc00002d7174]
3 simple_lock_time_violation(slp = (unallocated - symbol optimized
away), state = (unallocated - symbol optimized away), caller = (unallocated
- symbol optimized away)) ["../../../../src/kernel/kern/lock.c":2974,
0xfffffc00002d7588]
4 thread_list_wakeup(whead = (unallocated - symbol optimized away), s =
(unallocated - symbol optimized away), result = (unallocated - symbol
optimized away)) ["../../../../src/kernel/kern/sched_prim.c":5756,
0xfffffc00002eead4]
5 event_post(0x1, 0x0, 0xfffffc01af90c800, 0xfffffc01af90c800,
0xfffffc00002b1b80) ["../../../../src/kernel/kern/event.c":171,
0xfffffc00002c89d0]
6 select_wakeup(0xfffffc000ef2c5c0, 0xfffffc000ef2c540, 0x0, 0x0,
0xfffffc01d9ff1980) ["../../../../src/kernel/bsd/subr_select.c":410,
0xfffffc00002b1b80]
7 sowakeup(0xfffffc01d9ff1980, 0x0, 0xfffffc000030a9c4,
0xfffffc0096891180, 0xfffffc012efb7e00)
["../../../../src/kernel/bsd/uipc_socket2.c":1513, 0xfffffc000025b058]
8 uipc_usrreq(0x0, 0xfffffe0638ce77e0, 0x0, 0x7f10, 0x0)
["../../../../src/kernel/bsd/uipc_usrreq.c":617, 0xfffffc000030a9c0]
9 sosend(0x1, 0xfffffc0096891280, 0x9, 0x0, 0xfffffc01f849f100)
["../../../../src/kernel/bsd/uipc_socket.c":3121, 0xfffffc0000257c80]
10 rwuio(0xfffffe0638ce0000, 0xfffffc016715fc00, 0xfffffc01ffede180,
0xfffffe0638ce78f0, 0x1) ["../../../../src/kernel/bsd/sys_generic.c":2292,
0xfffffc00002c47f4]
11 _F64_writev(0xfffffc0000276cc7, 0xfffffc0000000001, 0x0, 0x0,
0xfffffc0000675ee0) ["../../../../src/kernel/bsd/sys_generic.c":1978,
0xfffffc00002c40cc]
12 syscall(0x2, 0x14004bfb8, 0xd, 0x120022100, 0x0)
["../../../../src/kernel/arch/alpha/syscall_trap.c":725,
0xfffffc0000672940]
13 _Xsyscall(0x8, 0x3ff801374f8, 0x14000e7b0, 0x11, 0x14003ec20)
["../../../../src/kernel/arch/alpha/locore.s":1814, 0xfffffc000067611c]
_stack_trace[0]_end:
warning: Files compiled -g3: parameter values probably wrong
thread 0xfffffc01ffe20380 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
_stack_trace[1]_begin:
> 0 stop_secondary_cpu(do_lwc = (unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/cpu.c":1336, 0xfffffc000068884c]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1367, 0xfffffc00002b0414]
2 xcpu_panic_ipir() ["../../../../src/kernel/bsd/subr_prf.c":1596,
0xfffffc00002b09ec]
3 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":1570,
0xfffffc0000688da0]
4 _XentInt(0x0, 0xfffffc00002ecc90, 0xfffffc00008ba0d0,
0xfffffc0009b7c000, 0xfffffc0000b90000)
["../../../../src/kernel/arch/alpha/locore.s":1611, 0xfffffc0000675f3c]
5 idle_thread() ["../../../../src/kernel/kern/sched_prim.c":4530,
0xfffffc00002ecc90]
_stack_trace[1]_end:
warning: Files compiled -g3: parameter values probably wrong
thread 0xfffffc01ffe21500 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
_stack_trace[2]_begin:
> 0 stop_secondary_cpu(do_lwc = (unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/cpu.c":1336, 0xfffffc000068884c]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1367, 0xfffffc00002b0414]
2 xcpu_panic_ipir() ["../../../../src/kernel/bsd/subr_prf.c":1596,
0xfffffc00002b09ec]
3 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":1570,
0xfffffc0000688da0]
4 _XentInt(0x0, 0xfffffc00002ecc18, 0xfffffc00008ba0d0,
0xfffffc0001de4040, 0xfffffc0000b90000)
["../../../../src/kernel/arch/alpha/locore.s":1611, 0xfffffc0000675f3c]
5 idle_thread() ["../../../../src/kernel/kern/sched_prim.c":4480,
0xfffffc00002ecc14]
_stack_trace[2]_end:
warning: Files compiled -g3: parameter values probably wrong
thread 0xfffffc01ffe21180 stopped at [stop_secondary_cpu:1336
+0x4,0xfffffc000068884c] Source not available
_stack_trace[3]_begin:
> 0 stop_secondary_cpu(do_lwc = (unallocated - symbol optimized away))
["../../../../src/kernel/arch/alpha/cpu.c":1336, 0xfffffc000068884c]
1 panic(s = (unallocated - symbol optimized away))
["../../../../src/kernel/bsd/subr_prf.c":1367, 0xfffffc00002b0414]
2 xcpu_panic_ipir() ["../../../../src/kernel/bsd/subr_prf.c":1596,
0xfffffc00002b09ec]
3 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":1570,
0xfffffc0000688da0]
4 _XentInt(0x0, 0xfffffc00002ecc10, 0xfffffc00008ba0d0,
0xfffffc0001de4040, 0xfffffc0000b90000)
["../../../../src/kernel/arch/alpha/locore.s":1611, 0xfffffc0000675f3c]
5 idle_thread() ["../../../../src/kernel/kern/sched_prim.c":4479,
0xfffffc00002ecc10]
_stack_trace[3]_end:
warning: Files compiled -g3: parameter values probably wrong
_kdbx_sum_start:
Hostname : rimmer
cpu: AlphaServer ES40 avail: 4
Boot-time: Sun Dec 23 00:06:30 2007
Time: Mon Dec 24 16:03:44 2007
Kernel : OSF1 release V5.1 version 1885 (alpha)
_kdbx_sum_end:
_kdbx_swap_start:
Swap device name Size In Use Free
-------------------------------- ---------- ---------- ----------
/dev/disk/dsk3b 17211864k 8k 17211856k
2151483p 1p 2151482p
-------------------------------- ---------- ---------- ----------
Total swap partitions: 1 17211864k 8k 17211856k
2151483p 1p 2151482p
_kdbx_swap_end:
_kdbx_proc_start:
Addr PID PPID PGRP UID NICE SIGCATCH P_SIG Event
Flags
=========== ===== ===== ===== ===== ==== ======== ======== ===========
============
k0xffe06240 0 0 0 0 0 00000000 00000000 NULL in
sys
k0xffe06cc0 1 0 1 0 0 307a62ff 00000000 NULL in
contign pagv
k0xf85b8cc0 16 1 16 0 0 00000001 00000000 NULL in
pagv
k0xf87accc0 18 1 18 0 0 00001ef8 00000000 NULL in
pagv
k0x0e06acc0 107 1 107 0 0 00003001 00000000 NULL in
pagv
k0x42540240 118 1 118 0 0 00002000 00000000 NULL in
pagv
k0x66f3a240 234 1 234 0 0 fffffffffffaf047 00000000
NULL in pagv
k0x963ca240 268 234 234 0 0 00083001 00000000 NULL in
pagv
k0x963cb740 269 234 234 0 0 00003001 00000000 NULL in
pagv
k0x963cacc0 416 1 416 0 0 00003000 00000000 NULL in
pagv
k0xacb42cc0 454 1 454 0 0 70087001 00000000 NULL in
pagv
k0x2c412cc0 458 1 458 0 0 20007001 00000000 NULL in
pagv
k0x96170cc0 545 1 545 0 0 00080628 00000000 NULL in
pagv
k0xad3e4cc0 548 1 548 0 0 00000000 00000000 NULL in
pagv
k0xacb42240 730 1 730 0 -88 60086207 00000000 NULL in
pagv
k0xad3e4240 743 1 743 0 0 20086ef9 00000000 NULL in
pagv
k0xad3e5740 767 1 767 0 0 20004003 00000000 NULL in
pagv
k0x2ccc0240 779 1 779 0 0 00004002 00000000 NULL in
pagv
k0xb6b0d740 784 1 784 0 0 00004002 00000000 NULL in
pagv
k0x2ccc0cc0 785 1 785 0 0 00004002 00000000 NULL in
pagv
k0xf7e72cc0 786 1 786 0 -1 00004002 00000000 NULL in
pagv
k0x2c412240 788 1 788 0 0 00002002 00000000 NULL in
pagv
k0xb3e68cc0 823 1 823 0 0 00004002 00000000 NULL in
pagv
k0xff6e9740 828 1 399 0 0 00001eff 00000000 NULL in
pagv ctty
k0xb3e68240 842 1 841 0 0 03806efa 00000000 NULL in
pagv ctty
k0x968ff740 929 1 929 0 0 00004003 00000000 NULL in
pagv
k0x0e06b740 932 929 929 0 0 00000000 00000000 NULL in
exit pagv
k0xaed2f740 942 1 942 0 0 00000ef8 00000000 NULL in
pagv
k0x66f3acc0 952 1 399 0 -1 60027eff 00000000 NULL in
pagv ctty
k0xb376e240 955 1 955 0 0 00000ef8 00000000 NULL in
pagv
k0x2c413740 965 1 965 0 0 40084005 00000000 NULL in
pagv
k0xac641740 966 965 965 0 0 60083001 00000000 NULL in
pagv
k0xb6b0ccc0 977 1 977 0 0 00006000 00000000 NULL in
pagv
k0x68372240 1013 1 1013 0 0 00085007 00000000 NULL in
pagv
k0xb3e69740 1037 1 1037 0 0 20084601 00000000 NULL in
pagv
k0xb376ecc0 1039 1 1039 0 0 20004601 00000000 NULL in
pagv
k0x12c10cc0 1083 1037 1037 0 0 20084601 00000000 NULL in
pagv
k0x2ccc1740 1117 1037 1037 0 0 20084601 00000000 NULL in
pagv
k0x68372cc0 1167 1037 1037 0 0 20084601 00000000 NULL in
pagv
k0x3369a240 1523 1 1522 101 0 00000ef8 00000000 NULL in
pagv ctty
k0x12c10240 1609 1 1607 101 0 77886efd 00000000 NULL in
pagv ctty
k0x2f854240 2022 1 2021 101 0 37886efd 00000000 NULL in
pagv ctty
k0xff6e8cc0 2038 1 2036 101 0 77886efd 00000000 NULL in
pagv ctty
k0x96170240 2138 1 2133 101 0 00000ef8 00000000 NULL in
pagv ctty
k0xff6e8240 2224 1 2222 101 0 00000ef8 00000000 NULL in
pagv ctty
k0x68373740 2308 1 399 101 0 67807efe 00000000 NULL in
pagv ctty
k0xac092cc0 2309 1 399 101 0 67807efe 00000000 NULL in
pagv ctty
k0x2f855740 2310 1 399 101 0 67807efe 00000000 NULL in
pagv ctty
k0x96171740 2339 1 2339 0 0 20084003 00000000 NULL in
pagv
k0x968fe240 2390 1 399 0 0 21800ef8 00000000 NULL in
pagv ctty
k0x43f8ecc0 2419 2339 2419 0 0 00004203 00000000 NULL in
pagv
k0x6c9ce240 2421 2339 2421 0 0 20004002 00000000 NULL in
pagv
k0xaed2ecc0 2422 1 2422 0 0 00000ef8 00000000 NULL in
pagv
k0xf87ad740 2423 1 2423 0 0 00000000 00000000 NULL in
pagv ctty
k0x968fecc0 2438 2421 2438 0 0 00004000 00000000 NULL in
pagv
k0x6c9cf740 2521 1037 1037 0 0 20084601 00000000 NULL in
pagv
k0x740cacc0 462976 358653 977 207 0 00000000 00000000 NULL in
pagv
k0xef6aa240 463922 463975 463964 0 0 00006efb 00000000 NULL
in pagv
k0x64254240 463936 463953 977 0 0 00000000 00000000 NULL in
pagv
k0x5d094240 463953 463954 977 0 0 00000000 00000000 NULL in
pagv
k0x485c8cc0 463954 463965 977 0 0 61087af8 00000000 NULL in
pagv
k0xb6860240 463962 977 977 0 0 00000000 00000000 NULL in
pagv
k0xef6ab740 463964 234 463964 0 0 00000000 00000000 NULL in
pagv
k0x66f3b740 463965 463962 977 0 0 60007af8 00000000 NULL in
pagv
k0xb6449740 463973 463953 977 0 0 00000000 00000000 NULL in
pagv
k0x44980240 463975 463964 463964 0 0 00000000 00000000 NULL
in pagv
k0x5bf5f740 464030 464061 463964 0 0 00000000 00000000 NULL
in pagv
k0x0a06b740 464046 464061 463964 0 0 00000000 00000000 NULL
in pagv
k0x33561740 464049 464061 463964 0 0 00000000 00000000 NULL
in pagv
k0x6c9cecc0 464061 463922 463964 0 0 00006eff 00000000 NULL
in pagv
k0x485c9740 464092 464061 463964 0 0 00000000 00000000 NULL
in pagv
k0x42541740 464097 464061 463964 0 0 00000000 00000000 NULL
in pagv
k0x0a06a240 464105 464061 463964 0 0 00000080 00000000 NULL
in pagv
k0xac640240 464334 28639 977 207 0 00000000 00000000 NULL in
pagv
k0x5d094cc0 464591 23627 977 207 0 00000000 00000000 NULL in
pagv
k0x485c8240 464767 27944 977 207 0 00000000 00000000 NULL in
pagv
k0x0fdd2240 464804 28370 977 207 0 00000000 00000000 NULL in
pagv
k0xb6861740 464813 24931 977 207 0 00000000 00000000 NULL in
pagv
k0x2f854cc0 464820 1037 1037 0 0 20084601 00000000 NULL in
pagv
k0x740ca240 464847 23683 977 207 0 00000000 00000000 NULL in
pagv
k0xb6448cc0 23450 1 977 207 0 00000ef8 00000000 NULL in
pagv
k0xe1006cc0 23627 1 977 207 0 60007af8 00000000 NULL in
pagv
k0x64254cc0 23683 1 977 207 0 60007af8 00000000 NULL in
pagv
k0xe1006240 24929 977 977 0 0 00000000 00000000 NULL in
pagv
k0x33560240 24931 24929 977 207 0 60007af8 00000000 NULL in
pagv
k0x24d24cc0 26624 977 977 0 0 00000000 00000000 NULL in
pagv
k0x44981740 26625 26624 977 207 0 00000ef8 00000000 NULL in
pagv
k0x24d25740 27942 977 977 0 0 00000000 00000000 NULL in
pagv
k0x64255740 27944 27942 977 207 0 60007af8 00000000 NULL in
pagv
k0xb6448240 28367 977 977 0 0 00000000 00000000 NULL in
pagv
k0x44980cc0 28370 28367 977 207 0 60007af8 00000000 NULL in
pagv
k0x0fdd3740 28639 1 977 207 0 60007af8 00000000 NULL in
pagv
k0xb6860cc0 358653 358659 977 207 0 60007af8 00000000 NULL in
pagv
k0xacb43740 358659 977 977 0 0 00000000 00000000 NULL in
pagv
k0x3369b740 256380 1 399 0 -20 20007e73 00000000 NULL in
pagv ctty
_kdbx_proc_end:
warning: Files compiled -g3: parameter values probably wrong
Audit subsystem disabled
No audit data to be saved
#
_crash_data_collection_finished:
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:50:36 EDT