E420R Reboot - Uncorrectable Memory Error

From: Tan Kian Chye (kianchye.tan@xatmi.com)
Date: Tue Dec 16 2003 - 21:09:51 EST


Hi,
One of our systems crahsed/rebooted today and I just wanted to make sure i
am heading in the right direction. This is the first time this has happened
in our environment.

Some suggestions in the WWW actually mentioned this could be problem in the
CPU, but i just wanted to post the info here to see if anyone had any
addidtional information on what i can do to track or perhaps fix this
problem.

Is this CPU or a Memory Problem ?? Maybe both ?

Below is the /var/adm/messages:
------------------------------------------------------------ ---------------
----------------
Dec 16 01:58:33 xa-ora-fin SUNW,UltraSPARC-II: [ID 424925 kern.warning]
WARNING: [AFT1] Uncorrectable Memory Error on CPU0 Data access at TL=0,
errID 0x00111411.65de423d
Dec 16 01:58:33 xa-ora-fin AFSR 0x00000000.80200000<PRIV,UE> AFAR
0x00000000.9ae64db8
Dec 16 01:58:33 xa-ora-fin AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00
Fault_PC 0x100cb160
Dec 16 01:58:33 xa-ora-fin UDBH 0x00b2 UDBH.ESYND 0xb2 UDBL 0x0203<UE>
UDBL.ESYND 0x03
Dec 16 01:58:33 xa-ora-fin UDBL Syndrome 0x3 Memory Module U1304 U0304 U1303
U0303
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 954695 kern.warning]
WARNING: [AFT1] errID 0x00111411.65de423d Syndrome x3 indicates that this
may not be a memory module problem
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin E$tag 0x00000000.0a40135c E$State: Shared
E$parity 0x05
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x00): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x08): 0x00000000.00000021
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x10): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x18): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x20): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x28): 0x00000000.0017c527
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x30): 0x00000000.0017cb5f
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2]
E$Data (0x38): 0x00000000.0000066a *Bad* PSYND=0x00ff
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 339554 kern.warning]
WARNING: [AFT1] CP event on CPU3 (caused Data access error on CPU0), errID
0x00111411.65de423d
Dec 16 01:58:34 xa-ora-fin AFSR 0x00000000.01000008<CP> AFAR
0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin AFSR.PSYND 0x0008(Score 95) AFSR.ETS 0x00
Dec 16 01:58:34 xa-ora-fin UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000
UDBL.ESYND 0x00
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin E$tag 0x00000000.1b40135c E$State: Owner E$parity
0x0d
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x00): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x08): 0x00000000.00000021
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x10): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x18): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x20): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x28): 0x00000000.0017c527
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x30): 0x00000000.0017cb5f
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2]
E$Data (0x38): 0x00000000.0000066a *Bad* PSYND=0x0008
Dec 16 01:58:34 xa-ora-fin unix: [ID 836849 kern.notice]
Dec 16 01:58:34 xa-ora-fin ^Mpanic[cpu0]/thread=30004623000:
Dec 16 01:58:34 xa-ora-fin unix: [ID 787147 kern.notice] [AFT1] errID
0x00111411.65de423d UE Error(s)
Dec 16 01:58:34 xa-ora-fin See previous message(s) for details
Dec 16 01:58:34 xa-ora-fin unix: [ID 100000 kern.notice]
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d400
SUNW,UltraSPARC-II:cpu_aflt_log+4e0 (2a10085d4be, 1, 10146ad8, 2a10085d648,
2a10085d50b, 10146b00)
Dec 16 01:58:34 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 000002a10085d710 0000000000000003 0000000000000010
Dec 16 01:58:34 xa-ora-fin %l4-7: 0000030003576088 00000300035761f0
000000000000000e 0000000000002000
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d650
SUNW,UltraSPARC-II:cpu_async_error+868 (104597f0, 2a10085d710, 80200000, 0,
650196480200000, 2a10085d8d0)
Dec 16 01:58:34 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
000000001040dae4 0000000000000032 0000000000000203 00000000000000b2
Dec 16 01:58:34 xa-ora-fin %l4-7: 000000009ae64d80 0000000000400000
0000000000400000 0000000000000001
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d820
unix:prom_rtt+0 (300010275c0, 0, 20, 0, 30002412040, 1)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000007 0000000000001400 00000044f0001606 000000001013e814
Dec 16 01:58:35 xa-ora-fin %l4-7: 0000000000000000 0000000000000000
0000000000000000 000002a10085d8d0
Dec 16 01:58:35 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d970
genunix:psig+23c (30004623110, 0, 68, e, 2, feb9aef4)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 0000030003576088 0000000000002000 0000030004620b00
Dec 16 01:58:35 xa-ora-fin %l4-7: 000000000000000e 0000000000000000
000000000000000e 000002a10085da10
Dec 16 01:58:35 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085da20
genunix:post_syscall+3ec (30004623000, 35, 1, ffbee6ac, 4, 0)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 000002a10085dba0 0000030004620b00 000000000000005b
Dec 16 01:58:35 xa-ora-fin %l4-7: 0000000000000000 0000030003576088
0000000000000004 0000000001850328
Dec 16 01:58:35 xa-ora-fin unix: [ID 100000 kern.notice]
Dec 16 01:58:33 xa-ora-fin SUNW,UltraSPARC-II: [ID 424925 kern.warning]
WARNING: [AFT1] Uncorrectable Memory Error on CPU0 Data access at TL=0,
errID 0x00111411.65de423d
Dec 16 01:58:33 xa-ora-fin AFSR 0x00000000.80200000<PRIV,UE> AFAR
0x00000000.9ae64db8
Dec 16 01:58:33 xa-ora-fin AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00
Fault_PC 0x100cb160
Dec 16 01:58:33 xa-ora-fin UDBH 0x00b2 UDBH.ESYND 0xb2 UDBL 0x0203<UE>
UDBL.ESYND 0x03
Dec 16 01:58:33 xa-ora-fin UDBL Syndrome 0x3 Memory Module U1304 U0304 U1303
U0303
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 954695 kern.warning]
WARNING: [AFT1] errID 0x00111411.65de423d Syndrome 0x3 indicates that this
may not be a memory module problem
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin E$tag 0x00000000.0a40135c E$State: Shared
E$parity 0x05
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x00): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x08): 0x00000000.00000021
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x10): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x18): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x20): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x28): 0x00000000.0017c527
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x30): 0x00000000.0017cb5f
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2]
E$Data (0x38): 0x00000000.0000066a *Bad* PSYND=0x00ff
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 339554 kern.warning]
WARNING: [AFT1] CP event on CPU3 (caused Data access error on CPU0), errID
0x00111411.65de423d
Dec 16 01:58:34 xa-ora-fin AFSR 0x00000000.01000008<CP> AFAR
0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin AFSR.PSYND 0x0008(Score 95) AFSR.ETS 0x00
Dec 16 01:58:34 xa-ora-fin UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000
UDBL.ESYND 0x00
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:33 xa-ora-fin SUNW,UltraSPARC-II: [ID 424925 kern.warning]
WARNING: [AFT1] Uncorrectable Memory Error on CPU0 Data access at TL=0,
errID 0x00111411.65de423d
Dec 16 01:58:33 xa-ora-fin AFSR 0x00000000.80200000<PRIV,UE> AFAR
0x00000000.9ae64db8
Dec 16 01:58:33 xa-ora-fin AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00
Fault_PC 0x100cb160
Dec 16 01:58:33 xa-ora-fin UDBH 0x00b2 UDBH.ESYND 0xb2 UDBL 0x0203<UE>
UDBL.ESYND 0x03
Dec 16 01:58:33 xa-ora-fin UDBL Syndrome 0x3 Memory Module U1304 U0304 U1303
U0303
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 954695 kern.warning]
WARNING: [AFT1] errID 0x00111411.65de423d Syndrome 0x3 indicates that this
may not be a memory module problem
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin E$tag 0x00000000.0a40135c E$State: Shared
E$parity 0x05
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x00): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x08): 0x00000000.00000021
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x10): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x18): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x20): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x28): 0x00000000.0017c527
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x30): 0x00000000.0017cb5f
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2]
E$Data (0x38): 0x00000000.0000066a *Bad* PSYND=0x00ff
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 339554 kern.warning]
WARNING: [AFT1] CP event on CPU3 (caused Data access error on CPU0), errID
0x00111411.65de423d
Dec 16 01:58:34 xa-ora-fin AFSR 0x00000000.01000008<CP> AFAR
0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin AFSR.PSYND 0x0008(Score 95) AFSR.ETS 0x00
Dec 16 01:58:34 xa-ora-fin UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000
UDBL.ESYND 0x00
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 999597 kern.info] [AFT2]
errID 0x00111411.65de423d PA=0x00000000.9ae64db8
Dec 16 01:58:34 xa-ora-fin E$tag 0x00000000.1b40135c E$State: Owner E$parity
0x0d
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x00): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x08): 0x00000000.00000021
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x10): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x18): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x20): 0x00000000.00000000
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x28): 0x00000000.0017c527
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2]
E$Data (0x30): 0x00000000.0017cb5f
Dec 16 01:58:34 xa-ora-fin SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2]
E$Data (0x38): 0x00000000.0000066a *Bad* PSYND=0x0008
Dec 16 01:58:34 xa-ora-fin unix: [ID 836849 kern.notice]
Dec 16 01:58:34 xa-ora-fin ^Mpanic[cpu0]/thread=30004623000:
Dec 16 01:58:34 xa-ora-fin unix: [ID 787147 kern.notice] [AFT1] errID
0x00111411.65de423d UE Error(s)
Dec 16 01:58:34 xa-ora-fin See previous message(s) for details
Dec 16 01:58:34 xa-ora-fin unix: [ID 100000 kern.notice]
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d400
SUNW,UltraSPARC-II:cpu_aflt_log+4e0 (2a10085d4be, 1, 10146ad8, 2a10085d648,
2a10085d50b, 10146b00)
Dec 16 01:58:34 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 000002a10085d710 0000000000000003 0000000000000010
Dec 16 01:58:34 xa-ora-fin %l4-7: 0000030003576088 00000300035761f0
000000000000000e 0000000000002000
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d650
SUNW,UltraSPARC-II:cpu_async_error+868 (104597f0, 2a10085d710, 80200000, 0,
650196480200000, 2a10085d8d0)
Dec 16 01:58:34 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
000000001040dae4 0000000000000032 0000000000000203 00000000000000b2
Dec 16 01:58:34 xa-ora-fin %l4-7: 000000009ae64d80 0000000000400000
0000000000400000 0000000000000001
Dec 16 01:58:34 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d820
unix:prom_rtt+0 (300010275c0, 0, 20, 0, 30002412040, 1)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000007 0000000000001400 00000044f0001606 000000001013e814
Dec 16 01:58:35 xa-ora-fin %l4-7: 0000000000000000 0000000000000000
0000000000000000 000002a10085d8d0
Dec 16 01:58:35 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085d970
genunix:psig+23c (30004623110, 0, 68, e, 2, feb9aef4)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 0000030003576088 0000000000002000 0000030004620b00
Dec 16 01:58:35 xa-ora-fin %l4-7: 000000000000000e 0000000000000000
000000000000000e 000002a10085da10
Dec 16 01:58:35 xa-ora-fin genunix: [ID 723222 kern.notice] 000002a10085da20
genunix:post_syscall+3ec (30004623000, 35, 1, ffbee6ac, 4, 0)
Dec 16 01:58:35 xa-ora-fin genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 000002a10085dba0 0000030004620b00 000000000000005b
Dec 16 01:58:35 xa-ora-fin %l4-7: 0000000000000000 0000030003576088
0000000000000004 0000000001850328
Dec 16 01:58:35 xa-ora-fin unix: [ID 100000 kern.notice]
Dec 16 01:58:35 xa-ora-fin genunix: [ID 672855 kern.notice] syncing file
systems...
Dec 16 01:58:36 xa-ora-fin genunix: [ID 904073 kern.notice] done
Dec 16 01:58:37 xa-ora-fin genunix: [ID 353387 kern.notice] dumping to
/dev/dsk/c0t0d0s1, offset 65536
Dec 16 01:59:39 xa-ora-fin genunix: [ID 409368 kern.notice] ^M100% done:
45617 pages dumped, compression ratio 2.92,
Dec 16 01:59:39 xa-ora-fin genunix: [ID 851671 kern.notice] dump succeeded
Dec 16 13:23:01 xa-ora-fin genunix: [ID 540533 kern.notice] ^MSunOS Release
5.8 Version Generic_108528-12 64-bit
Dec 16 13:23:01 xa-ora-fin genunix: [ID 913631 kern.notice] Copyright
1983-2001 Sun Microsystems, Inc. All rights reserved.
Dec 16 13:23:01 xa-ora-fin genunix: [ID 678236 kern.info] Ethernet address =
8:0:20:f0:26:89
Dec 16 13:23:01 xa-ora-fin swapgeneric: [ID 370176 kern.warning] WARNING:
forceload of drv/atf failed
Dec 16 13:23:01 xa-ora-fin swapgeneric: [ID 370176 kern.warning] WARNING:
forceload of drv/scsi failed
Dec 16 13:23:01 xa-ora-fin unix: [ID 389951 kern.info] mem = 4194304K
(0x100000000)
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:27:42 EDT