From: Raghavan, Vijay (vijay.raghavan@skanska.se)
Date: Tue Apr 01 2003 - 03:26:15 EST
I am adding more info and giving outputs of the exact problem
When I run lockstat sleep 5 the o/p is like this
# lockstat sleep 5
Adaptive mutex spin: 3736 events
Count indv cuml rcnt spin Lock Caller
----------------------------------------------------------------------------
---
187 5% 5% 1.00 7151 vx_sched_lk vx_worklist_thread+0x2c
119 3% 8% 1.00 4918 vx_sched_lk vx_worklist_thread+0x38
116 3% 11% 1.00 45 0x6d2c7af0 fisopen+0x8
103 3% 14% 1.00 48 0x6d2c7af0 closeandsetf+0x10
102 3% 17% 1.00 436 service_queue background+0xdc
85 2% 19% 1.00 235 service_queue qenable_locked+0x34
84 2% 21% 1.00 32 0x6d2c7af0 closeandsetf+0x218
76 2% 23% 1.00 52 0x6d2c7af0 getf+0x50
70 2% 25% 1.00 66 0x6d2c7af0 releasef+0x48
69 2% 27% 1.00 2693 0x61dc4200 stop+0x904
69 2% 29% 1.00 11 hblk8_lock
sfmmu_hblk_tofreelist+0x38
68 2% 31% 1.00 2880 0x61dc4200 issig_forreal+0x10
68 2% 33% 1.00 368 vx_worklist_lk
vx_worklist_process+0xb4
67 2% 34% 1.00 1012 0x61dc4200 stop+0x144
63 2% 36% 1.00 196 service_queue background+0x120
59 2% 38% 1.00 17 pr_pidlock pr_p_lock+0xc
Adaptive mutex block: 2173 events
Count indv cuml rcnt nsec Lock Caller
----------------------------------------------------------------------------
---
1112 51% 51% 1.00 17325 0x62920000 fca_pkt_complete+0x24
1026 47% 98% 1.00 17361 0x63d04000 fca_pkt_complete+0x24
6 0% 99% 1.00 85715 service_queue qenable_locked+0x34
2 0% 99% 1.00 537213 hblk8_lock
sfmmu_hblk_tofreelist+0x38
2 0% 99% 1.00 36609 0x6d2c7af0 releasef+0x48
2 0% 99% 1.00 33056 service_queue background+0x120
1 0% 99% 1.00 83733 0x7d4c16f4 putnext+0xc
1 0% 99% 1.00 35045 service_queue background+0xdc
1 0% 99% 1.00 129500 fpc_mutex+0x28 page_list_add+0xb8
1 0% 99% 1.00 32961 pse_mutex+0x1c0
page_try_reclaim_lock+0x18
1 0% 99% 1.00 22193 0x7c986350 background+0xa8
1 0% 99% 1.00 23825 0x73a84fd8 background+0xa8
1 0% 99% 1.00 115811 fpc_mutex+0x68
page_get_mnode_freelist+0xe
c
1 0% 99% 1.00 250112 hblk8_lock sfmmu_hblk_alloc+0xdc
1 0% 99% 1.00 802213 pidlock cfork+0x57c
Spin lock spin: 15506 events
Count indv cuml rcnt spin Lock Caller
----------------------------------------------------------------------------
---
829 5% 5% 1.00 808 cpu[9]+0x44 disp_getbest+0x4
794 5% 10% 1.00 89 cpu[12]+0x44 disp+0x78
678 4% 15% 1.00 132 cpu[9]+0x44 disp+0x78
660 4% 19% 1.00 74 cpu[17]+0x44 disp+0x78
561 4% 23% 1.00 95 cpu[28]+0x44 disp+0x78
556 4% 26% 1.00 925 cpu[7]+0x44 disp_getbest+0x4
555 4% 30% 1.00 98 cpu[8]+0x44 disp+0x78
555 4% 33% 1.00 149 cpu[13]+0x44 disp+0x78
548 4% 37% 1.00 88 cpu[25]+0x44 disp+0x78
548 4% 41% 1.00 886 cpu[8]+0x44 disp_getbest+0x4
541 3% 44% 1.00 121 cpu[24]+0x44 disp+0x78
Thread lock spin: 21 events
Count indv cuml rcnt spin Lock Caller
----------------------------------------------------------------------------
---
4 19% 19% 1.00 3744 lwpsleepq+0x7c4 prchoose+0x68
2 10% 29% 1.00 3267 lwpsleepq+0x7c4 ts_tick+0x8
2 10% 38% 1.00 1964 lwpsleepq+0xc44 prchoose+0x68
2 10% 48% 1.00 1900 lwpsleepq+0xc4 prchoose+0x68
1 5% 52% 1.00 2853 lwpsleepq+0x344 ts_tick+0x8
1 5% 57% 1.00 1876 lwpsleepq+0x844 ts_tick+0x8
1 5% 62% 1.00 3 shuttle_lock ts_update_list+0x60
1 5% 67% 1.00 313 lwpsleepq+0xf74 prchoose+0x68
1 5% 71% 1.00 213 sleepq_head+0x99c ts_tick+0x8
1 5% 76% 1.00 204 sleepq_head+0x40c prchoose+0x68
1 5% 81% 1.00 1429 lwpsleepq+0x2c4 prchoose+0x68
1 5% 86% 1.00 1385 sleepq_head+0x95c setrun+0x4
1 5% 90% 1.00 1715 sleepq_head+0x97c prchoose+0x68
1 5% 95% 1.00 3 cpu[25]+0x68 preempt+0x1c
1 5% 100% 1.00 2520 lwpsleepq+0x344 ts_update_list+0x60
----------------------------------------------------------------------------
---
R/W writer blocked by writer: 2 events
Count indv cuml rcnt nsec Lock Caller
----------------------------------------------------------------------------
---
2 100% 100% 1.00 219516694 0x6d4da238 ufs_remove+0x11c
----------------------------------------------------------------------------
---
R/W writer blocked by readers: 3 events
Count indv cuml rcnt nsec Lock Caller
----------------------------------------------------------------------------
---
1 33% 33% 1.00 139105 0x8047950c as_unmap+0x48
1 33% 67% 1.00 143115 0x6dd0615c as_unmap+0x48
1 33% 100% 1.00 150178 kas+0x1c as_unmap+0x48
----------------------------------------------------------------------------
---
R/W reader blocked by writer: 3 events
Count indv cuml rcnt nsec Lock Caller
----------------------------------------------------------------------------
---
1 33% 33% 1.00 111514 0x8047950c oprgetpsinfo+0x55c
1 33% 67% 1.00 175003 0x6dd0615c oprgetpsinfo+0x55c
1 33% 100% 1.00 5982315 0x6d4da238 ufs_dirlook+0xc8
----------------------------------------------------------------------------
---
Lockstat record failure: 2 events
Count indv cuml rcnt Lock Caller
----------------------------------------------------------------------------
---
2 100% 100% 0.00 lockstat_lock lockstat_add_to_hash
-------------------------------------------------------
> H/W platform : Ultra Enterprise 6500
> OS Version : SunOS ska101.data.skanska.se 5.6 Generic_105181-33 sun4u
>
> The problem is there is a serious mutex contention and the adaptive mutex
> spin is 945 events
>
> Any suggestions regarding tuning kernel parameters and it this also to do
> something with VxFS version.
>
> Any deatiled documentation or web sites explaining the problems and giving
> recommendations and solutions.
>
> Regards and will summarize.
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:26:05 EDT