From: jmendez (jmendez@sii.cl)
Date: Tue Apr 13 2004 - 12:28:44 EDT
Dears,
I have a Solaris System, E10K with 8 CPU of 400 MHZ 6 GB of RAM
and Oracle 7.34, 3 instances and a messagging local packages running.
I detected a Mutex contention, because the CPU (sys) > 20% and > CPU (usr)
besides mpstat (smtx) > 500. sometime CPU (sys)>60 % idle (0%)
How can I fix that problems or detect the fail ?
Is there some scheme or procedure that permit me to localize the problems.
troubleshooting scheme ?
I tried lockstat but the result sometime confused me ?
Please give me some Hints or documents that permit me to clarify the problems
waiting for some hints, recomendation, etc.
Regards,
Josh Mhndez
a little Sun Manager
I 'm attaching some information.
TOP:
last pid: 22463; load averages: 34.05, 34.29, 31.93
10:56:50
2329 processes:2292 sleeping, 27 running, 5 zombie, 5 on cpu
CPU states: 0.0% idle, 40.2% user, 59.8% kernel, 0.0% iowait, 0.0% swap
Memory: 6144M real, 191M free, 41M swap in use, 2638M swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
17223 root 1 0 0 7248K 7144K run 88:49 6.27% cai_UxOs_ProcCo
28824 root 1 0 0 3720K 2576K cpu11 2:30 1.74% top
25059 root 1 0 0 3624K 2568K sleep 105:54 1.65% top
8035 sqltmon 1 0 0 3736K 2584K run 2:03 1.54% top
23080 root 1 0 0 3680K 2576K sleep 15:42 1.28% top
8028 oracle 1 44 0 470M 464M sleep 1:09 1.03% oracle
3084 oracle 1 46 0 470M 460M sleep 119:44 1.03% oracle
1 root 1 59 0 792K 400K sleep 51:02 0.86% init
16473 root 11 59 0 11M 9680K sleep 33:19 0.44% caiUxOs
3329 eculink 1 59 0 4664K 4256K sleep 44:52 0.38% eculink31_2
9964 oracle 1 39 0 470M 464M sleep 0:06 0.38% oracle
27459 oracle 1 33 0 470M 464M sleep 1:17 0.36% oracle
27602 oracle 1 49 0 470M 464M sleep 0:51 0.32% oracle
28263 oracle 1 40 0 470M 464M sleep 0:48 0.28% oracle
27554 oracle 1 9 0 470M 464M sleep 0:01 0.28% oracle
Note:
ca is CA TNG, normally work just for 2 minutes collecting monitoring
information.
eculink is the local messaging application that work strong with sockets
schemes.
top are the desesperated administrator trying to detect something :) waiting
for your collaboration
MPSTAT
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 679 3 12188 499 100 1682 381 292 508 4 10719 36 47 4 13
2 742 1 3604 781 200 1928 419 308 574 6 13298 49 31 4 16
8 637 1 2402 634 100 2203 524 331 632 5 14021 46 34 6 14
9 727 1 3155 2120 1675 1806 436 291 579 4 11067 41 43 3 13
10 594 2 2304 602 100 2156 487 317 598 4 12831 41 39 3 17
11 555 2 2298 577 100 1963 465 291 528 2 11143 40 41 4 15
12 719 2 2664 661 100 2304 558 359 649 6 13672 45 35 5 15
14 667 2 3077 651 100 2418 539 374 676 7 13750 49 28 6 17
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 765 4 15096 735 100 2303 621 300 649 0 16339 54 36 2 8
2 618 2 12073 825 200 1840 464 275 564 0 13710 57 36 1 7
8 791 2 11168 677 100 2028 573 268 570 0 14517 56 37 1 7
9 525 2 13593 2254 1800 1678 424 278 558 0 11961 41 48 2 8
10 596 2 9118 655 100 2059 548 264 586 0 14025 51 42 1 6
11 739 1 4033 654 100 2026 539 273 600 0 12734 50 42 2 6
12 674 3 15864 713 100 2267 602 307 628 0 15451 56 37 2 5
14 570 4 18122 651 100 1989 539 273 571 0 13083 47 47 1 5
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 573 4 2945 512 100 2021 402 366 577 4 13698 51 26 6 18
2 543 1 2817 612 200 1384 237 253 426 7 10103 36 45 2 17
8 551 3 2588 558 100 2274 446 370 622 5 14352 50 27 4 19
9 413 2 1971 2156 1768 1507 302 269 483 4 8653 34 47 3 16
10 730 4 15993 496 100 1853 382 323 552 5 12320 41 36 5 18
11 588 4 3462 570 100 2140 456 350 624 5 12296 41 34 7 18
12 568 4 3324 581 100 2255 462 374 674 6 12673 48 31 5 16
14 414 3 2393 543 100 2080 424 354 610 4 11885 39 38 4 19
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 312 3 2522 517 100 1733 398 306 506 4 12326 45 46 2 8
2 392 6 3382 762 200 1903 391 341 534 8 16208 56 31 5 8
8 801 4 3697 581 100 2027 464 323 579 9 14307 55 33 4 7
9 513 4 2798 2385 1920 1731 380 308 583 6 11683 50 38 5 7
10 366 4 12360 515 100 1762 396 285 506 12 12463 39 51 5 5
11 419 5 2961 574 100 2174 452 362 619 8 15063 49 36 7 8
12 500 5 3127 680 100 2570 559 411 721 8 15136 52 31 5 12
14 681 5 3076 644 100 2327 532 404 682 8 15058 57 31 3 9
PRTMEM
Total memory: 5963 Megabytes
Kernel Memory: 343 Megabytes
Application: 497 Megabytes
Executable & libs: 250 Megabytes
File Cache: 4679 Megabytes
Free, file cache: 73 Megabytes
Free, free: 142 Megabytes
LOCKSTAT sleep 5
the documentation refer the bold hexadecimal number as a hashed locks
how can associate that with something coherent ?
Adaptive mutex spin: 27115 events
Count indv cuml rcnt spin Lock Caller
-----------------------------------------------------------------------------
-- 9807 36% 36% 1.00 58 0x70538008 semconv_lock+0x60 6474 24% 60% 1.00 64 0x70538008 cv_wait_sig+0x1b8 2054 8% 68% 1.00 46 0x70538008 cv_wait_sig+0x230 1757 6% 74% 1.00 33 service_queue background+0xdc 1331 5% 79% 1.00 30 service_queue background+0x120 1298 5% 84% 1.00 22 service_queue qenable_locked+0x34 934 3% 87% 1.00 97 0x724b2d30 qfestart+0x1f8 594 2% 89% 1.00 4 tod_lock uniqtime+0x14 184 1% 90% 1.00 3 hblk8_lock sfmmu_hblk_alloc+0xdc 167 1% 91% 1.00 76 0x724b2d30 qfeintr+0x2bc 126 0% 91% 1.00 3 hblk8_lock sfmmu_hblk_tofreelist+0x38 100 0% 92% 1.00 5 0x704e4018 untimeout+0x18 79 0% 92% 1.00 11 0x704e4018 callout_execute+0xcc 73 0% 92% 1.00 6 0x703234b0 sync_stream_buf+0x24 70 0% 92% 1.00 4 0x704e4018 timeout_common+0x4 68 0% 93% 1.00 6 0x7028731c putnext+0xc 63 0% 93% 1.00 30 service_queue queuerun+0x1cc 58 0% 93% 1.00 9 pr_pidlock pr_p_lock+0xc 56 0% 93% 1.00 15 0x7028473c queuerun+0x198 53 0% 93% 1.00 3 0x7249b220 ufs_lockfs_begin_getpage+0x38 53 0% 94% 1.00 4 0x70239560 untimeout+0x18 53 0% 94% 1.00 4 0x70289f90 putnext+0x88 52 0% 94% 1.00 3 0x70289f90 putnext+0x30 52 0% 94% 1.00 7 service_queue queuerun+0xc 47 0% 94% 1.00 3 0x70239560 timeout_common+0x4 45 0% 95% 1.00 39 0x7028473c background+0xa8 34 0% 95% 1.00 3 0x7028473c putq+0x38 33 0% 95% 1.00 10 0x70239560 callout_execute+0xcc 32 0% 95% 1.00 7 0x7028473c getq_noenab+0x18 31 0% 95% 1.00 2 0x7249b220 ufs_lockfs_end+0x24 30 0% 95% 1.00 12 0x7036b570 timeout_common+0x4 30 0% 95% 1.00 4265 pidlock pr_lookup_procdir+0x9c 26 0% 95% 1.00 2 0x741ed8b0 ufs_getpage+0x86c 25 0% 95% 1.00 2 0x702847c0 putnext+0x88 24 0% 96% 1.00 9 0x70310aa8 callout_execute+0xcc 24 0% 96% 1.00 3 0x702847c0 putnext+0x30 23 0% 96% 1.00 18 0x70388ab8 callout_execute+0xcc 22 0% 96% 1.00 10 0x7036b570 callout_execute+0xcc 20 0% 96% 1.00 8 0x70238020 callout_execute+0xcc 18 0% 96% 1.00 12 0x70388ab8 timeout_common+0x4 18 0% 96% 1.00 16 ph_mutex+0x38 page_lookup_nowait+0x148 17 0% 96% 1.00 11 0x70386038 timeout_common+0x4 17 0% 96% 1.00 9 0x703bb580 timeout_common+0x4 16 0% 96% 1.00 4 0x70238020 untimeout+0x18 16 0% 96% 1.00 6 0x7030e028 callout_execute+0xcc 16 0% 96% 1.00 16 pr_pidlock prinactive+0x38 14 0% 96% 1.00 3 0x70310aa8 untimeout+0x18 14 0% 96% 1.00 4 swapfs_lock swapfs_getvp+0x8 14 0% 96% 1.00 1004 pidlock pr_readdir_procdir+0xc4 13 0% 97% 1.00 6 0x7030e028 untimeout+0x18 13 0% 97% 1.00 7 0x70386038 callout_execute+0xcc 13 0% 97% 1.00 5 0x70310aa8 timeout_common+0x4 13 0% 97% 1.00 22 0x7036b570 untimeout+0x18 12 0% 97% 1.00 1 0x70238020 timeout_common+0x4 12 0% 97% 1.00 2 fnode_cache+0xb4 ufs_lockfs_end+0x24 11 0% 97% 1.00 2 0x7030e028 timeout_common+0x4 9 0% 97% 1.00 26 0x70387578 timeout_common+0x4 8 0% 97% 1.00 9 ph_mutex+0x38 page_create_va+0x698 8 0% 97% 1.00 2129 pidlock waitid+0x90 8 0% 97% 1.00 13 0x70387578 untimeout+0x18 8 0% 97% 1.00 2 0x7023aaa0 untimeout+0x18 8 0% 97% 1.00 1 fnode_cache+0xb4 ufs_lockfs_begin_getpage+0x38 8 0% 97% 1.00 1 0x7249b220 ufs_lockfs_begin+0x38 8 0% 97% 1.00 2 0x702847c0 entersq+0x10 7 0% 97% 1.00 22 service_queue remove_runlist+0x28 7 0% 97% 1.00 8 0x70388ab8 untimeout+0x18 7 0% 97% 1.00 10 0x7023aaa0 callout_execute+0xcc 6 0% 97% 1.00 1 0x7028473c runservice+0x50 6 0% 97% 1.00 73 pidlock thread_exit+0x58 5 0% 97% 1.00 11 0x703bb580 callout_execute+0xcc 5 0% 97% 1.00 6 0x7023aaa0 timeout_common+0x4 5 0% 97% 1.00 1 0x703bb580 untimeout+0x18 5 0% 97% 1.00 38 fpc_mutex+0x28 page_list_add+0xd0 5 0% 97% 1.00 1 anoninfo_lock anon_unresv+0x30 5 0% 97% 1.00 5376 pidlock thread_free+0x1b4 4 0% 97% 1.00 6 0x70410898 getblk_common+0x84 4 0% 97% 1.00 30 0x75062dc0 fifo_write+0x5c 4 0% 97% 1.00 19 0x7036a030 timeout_common+0x4 4 0% 97% 1.00 2 0x72a66620 vn_rele+0x24 4 0% 97% 1.00 21 0x72cdf280 bunyip_name_addcomponent+0x1d0 4 0% 97% 1.00 3 hblk1_lock sfmmu_hblk_tofreelist+0xb8 4 0% 97% 1.00 48 fpc_mutex+0x20 page_list_add+0xd0 4 0% 97% 1.00 1 anoninfo_lock anon_resvmem+0x2c 4 0% 97% 1.00 5 0x7030e028 callout_execute+0x4 4 0% 97% 1.00 7 0x70387578 callout_execute+0xcc 4 0% 97% 1.00 1 fnode_cache+0xb4 ufs_lockfs_begin+0x38 4 0% 97% 1.00 7261 pidlock cv_wait_sig_swap+0x1d8 4 0% 97% 1.00 16 0x706edf0c segmap_getmapflt+0x338 4 0% 97% 1.00 1 0x702847c0 leavesq+0xc 3 0% 97% 1.00 44 fpc_mutex+0x18 page_list_add+0xd0 3 0% 98% 1.00 45 0x70410898 brelse+0xcc 3 0% 98% 1.00 34 0x814c7510 background+0xa8 3 0% 98% 1.00 11 0x75062140 cv_wait_sig_swap+0x1d8 3 0% 98% 1.00 12 fpc_mutex+0x40 page_get_mnode_freelist+0x140 3 0% 98% 1.00 21 0x7643bcf0 background+0xa8 3 0% 98% 1.00 21 0x75062dc0 cv_wait_sig_swap+0x1d8 3 0% 98% 1.00 765 shm_lock shmget+0x10 3 0% 98% 1.00 2 0x70535ec0 vn_rele+0x24 3 0% 98% 1.00 31 fpc_mutex+0x68 page_get_mnode_freelist+0x140 3 0% 98% 1.00 21 0x8091c450 background+0xa8 3 0% 98% 1.00 24 0x706edf2c segmap_getmapflt+0x338 3 0% 98% 1.00 2 plocks+0x2a0 polldel+0x28 3 0% 98% 1.00 19 0x75ca3998 background+0xa8 3 0% 98% 1.00 40 fpc_mutex+0x48 page_list_add+0xd0 3 0% 98% 1.00 1 hblk1_lock sfmmu_hblk_alloc+0x1ac 3 0% 98% 1.00 5 0x78f5e950 background+0xa8 3 0% 98% 1.00 38 0x72ce1b60 bunyip_name_addcomponent+0x1d0 3 0% 98% 1.00 42 0x72cdab20 bunyip_name_addcomponent+0x1d0 3 0% 98% 1.00 3 0x7190d898 dnlc_lookup+0x9c 3 0% 98% 1.00 2097 pidlock strioctl+0x27cc 3 0% 98% 1.00 9 tcp_mi_lock tcp_snmp_get+0x1d4 3 0% 98% 1.00 13 0x71684bf0 times+0x8 3 0% 98% 1.00 70 volio_spin volkiodone+0x84 3 0% 98% 1.00 43 0x79962bd0 background+0xa8 3 0% 98% 1.00 9 0x70386038 untimeout+0x18 3 0% 98% 1.00 9722 pidlock thread_create+0x250 3 0% 98% 1.00 13 0x71682e50 times+0x8 3 0% 98% 1.00 20 0x76783810 background+0xa8 3 0% 98% 1.00 68 0x79d39000 background+0xa8 3 0% 98% 1.00 1 pse_mutex+0x298 page_trylock+0x18 3 0% 98% 1.00 6 0x70310aa8 callout_execute+0x4 3 0% 98% 1.00 3016 pidlock setpgrp+0xa8 3 0% 98% 1.00 8 0x703bb580 callout_thread+0x38 3 0% 98% 1.00 2 pse_mutex+0x190 page_trylock+0x18 2 0% 98% 1.00 23 door_knob door_call+0x358 2 0% 98% 1.00 49 sfmmu_page_lock+0x1d0 sfmmu_page_enter+0x6c 2 0% 98% 1.00 2 pse_mutex+0x180 page_unlock+0x1c 2 0% 98% 1.00 67 0x7fcc9ba0 fifo_close+0x70 2 0% 98% 1.00 52 0x7479e820 fifo_write+0x5c 2 0% 98% 1.00 14 anonhash_lock+0x98 anon_dup+0x7c 2 0% 98% 1.00 9 0x75062a00 fifo_read+0x24 2 0% 98% 1.00 64 0x75062a00 fifo_write+0x5c 2 0% 98% 1.00 26 0x7fad4550 background+0xa8 2 0% 98% 1.00 8 0x7036a030 callout_execute+0xcc 2 0% 98% 1.00 6 0x1068a620 sfmmu_tteload_acquire_hashbucket+0x88 2 0% 98% 1.00 29 ctr_mutex+0x8 page_ctr_add+0x30 2 0% 98% 1.00 19 0x70410898 getblk_common+0x184 2 0% 98% 1.00 76 0x7479e1e0 cv_wait_sig_swap+0x1d8 2 0% 98% 1.00 15 0x8136eac8 background+0xa8 2 0% 98% 1.00 24 0x7f3b8800 background+0xa8 2 0% 98% 1.00 11 0x76fb3170 background+0xa8 2 0% 98% 1.00 6 0x735742f8 background+0xa8 2 0% 98% 1.00 28 0x756218e8 background+0xa8 2 0% 98% 1.00 6 0x704e4018 callout_schedule_1+0x4 2 0% 98% 1.00 24 0x105cfb40 sfmmu_mlist_enter+0x84 2 0% 98% 1.00 2 ph_mutex+0x188 page_create_va+0x698 2 0% 98% 1.00 1390 pidlock sigqkill+0x60 2 0% 98% 1.00 76 fpc_mutex+0x68 page_list_add+0xd0 2 0% 98% 1.00 36 0x75062a00 fifo_close+0x280 2 0% 98% 1.00 59 0x75062a00 fifo_close+0x70 2 0% 98% 1.00 2 pse_mutex+0x388 page_unlock+0x1c 2 0% 98% 1.00 93 pse_mutex+0x78 page_unlock+0x1c 2 0% 98% 1.00 35 0x7efeb6b8 background+0xa8 2 0% 98% 1.00 7 0x706edf2c segmap_smapadd+0xc 2 0% 98% 1.00 30 ctr_mutex+0x70 page_ctr_add+0x30 2 0% 98% 1.00 16 0x7f376ff8 background+0xa8 2 0% 98% 1.00 7 0x71684680 times+0x8 2 0% 98% 1.00 25 0x71682d08 times+0x8 2 0% 98% 1.00 2 0x741ed8b0 ufs_getattr+0x14c 2 0% 98% 1.00 4 0x741ed8b0 rdip+0x104 2 0% 98% 1.00 12 0x71682918 times+0x8 2 0% 98% 1.00 17 0x7ef57bb0 background+0xa8 2 0% 98% 1.00 6 0x803e1cf0 background+0xa8 2 0% 98% 1.00 55 0x7f413408 background+0xa8 2 0% 98% 1.00 1 ftable_lock fifo_inactive+0xc 2 0% 98% 1.00 32 0x79169568 background+0xa8 2 0% 98% 1.00 39 0x7479e460 fifo_read+0x24 2 0% 98% 1.00 37 0x7479e460 fifo_write+0x5c 2 0% 98% 1.00 4 smap_mtx+0x90 segmap_getmapflt+0x2b8 2 0% 98% 1.00 1 anonhash_lock+0x98 anon_decref+0x48 2 0% 98% 1.00 1 pse_mutex+0x348 page_trylock+0x18 2 0% 98% 1.00 53 0x72ede060 fifo_write+0x5c 2 0% 98% 1.00 7 0x72a66620 dnlc_lookup+0x12c 2 0% 98% 1.00 16 0x72a66620 bunyip_name_addcomponent+0xb0 2 0% 98% 1.00 9 0x7190cb48 dnlc_lookup+0x9c 2 0% 98% 1.00 44 0x72a66620 traverse+0x8 2 0% 98% 1.00 27 mod_lock mod_release_stub+0x14 2 0% 98% 1.00 3749 pidlock exit+0x414 2 0% 98% 1.00 44 0x1068d1e8 hat_unload+0x16c 2 0% 98% 1.00 42 0x7479e1e0 fifo_write+0x5c 2 0% 98% 1.00 103 0x72e906c0 fifo_close+0x70 2 0% 98% 1.00 35 0x72f6ada8 background+0xa8 2 0% 98% 1.00 18 0x71684a50 times+0x8 2 0% 98% 1.00 2 0x75c27f70 background+0xa8 2 0% 98% 1.00 20 0x7557add8 background+0xa8 2 0% 98% 1.00 1 pse_mutex+0x30 page_try_reclaim_lock+0x18 2 0% 98% 1.00 59 0x71681658 times+0x8 2 0% 98% 1.00 26 0x803e01f0 background+0xa8 2 0% 98% 1.00 29 0x7f412608 background+0xa8 2 0% 98% 1.00 14 0x75bcd468 background+0xa8 2 0% 98% 1.00 15 0x72e90080 fifo_read+0x24 2 0% 98% 1.00 34 0x72e90080 fifo_write+0x5c 2 0% 98% 1.00 2 0x70239560 callout_execute+0x4 2 0% 98% 1.00 2 0x706edf0c segmap_smapadd+0xc 2 0% 98% 1.00 2 pse_mutex+0x48 page_unlock+0x1c 2 0% 98% 1.00 11490 pidlock getproc+0xd4 2 0% 98% 1.00 28 0x796620b0 background+0xa8 2 0% 98% 1.00 17 0x7fea8ab0 background+0xa8 2 0% 98% 1.00 5 0x71683a58 times+0x8 2 0% 98% 1.00 8 0x755f15e0 background+0xa8 2 0% 98% 1.00 12 0x71684128 xsetitimer+0xec 2 0% 98% 1.00 31 0x7a9788b0 background+0xa8 2 0% 98% 1.00 14312 pidlock exit+0x2f8 2 0% 98% 1.00 64 0x70388ab8 callout_thread+0x38 2 0% 98% 1.00 16 0x71683b08 xsetitimer+0xec 2 0% 98% 1.00 18 0x7036a030 untimeout+0x18 2 0% 98% 1.00 15 0x7036b570 callout_schedule_1+0x4 2 0% 98% 1.00 122 0x7fcc9ba0 fifo_write+0x5c 2 0% 98% 1.00 45 0x782426b8 background+0xa8 2 0% 99% 1.00 14 0x73c38f28 background+0xa8 2 0% 99% 1.00 26 0x71681cb0 times+0x8 2 0% 99% 1.00 3 0x7253ba30 ufs_getpage+0x86c 2 0% 99% 1.00 17 0x802b62d8 background+0xa8 2 0% 99% 1.00 62 0x7fcc92e0 fifo_close+0x70 2 0% 99% 1.00 9 0x7479e820 fifo_close+0x70 1 0% 99% 1.00 62 door_knob door_call+0xa4 1 0% 99% 1.00 22 door_knob door_return+0xa4 1 0% 99% 1.00 2 plocks+0x350 polldel+0x28 1 0% 99% 1.00 2 door_knob door_info+0x74 1 0% 99% 1.00 40 0x7fcc9ba0 fifo_close+0x280 1 0% 99% 1.00 1 0x70535ec0 lookuppn+0xc4 1 0% 99% 1.00 29 0x754a6a18 background+0xa8 1 0% 99% 1.00 17 0x105cf448 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 2 pse_mutex+0x3c0 page_try_reclaim_lock+0x18 1 0% 99% 1.00 31 0x71682ca8 times+0x8 1 0% 99% 1.00 85 0x71681388 times+0x8 1 0% 99% 1.00 125 0x7f62c120 background+0xa8 1 0% 99% 1.00 23 0x756992f0 background+0xa8 1 0% 99% 1.00 2 0x803418e0 background+0xa8 1 0% 99% 1.00 22 0x74920c20 background+0xa8 1 0% 99% 1.00 25 0x782db92c strrput+0x390 1 0% 99% 1.00 2 0x71684660 sigaction+0xc4 1 0% 99% 1.00 2 0x7279ad20 ufs_getpage+0x86c 1 0% 99% 1.00 2 0x782db92c strclean+0x10 1 0% 99% 1.00 20 0x7557aed8 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x3f8 page_unlock+0x1c 1 0% 99% 1.00 2 0x71680c80 xsetitimer+0xec 1 0% 99% 1.00 2 0x71683938 times+0x8 1 0% 99% 1.00 1 pse_mutex+0x288 page_try_reclaim_lock+0x18 1 0% 99% 1.00 19 smap_mtx+0x98 segmap_getmapflt+0x2b8 1 0% 99% 1.00 2 pse_mutex+0x1d0 page_unlock+0x1c 1 0% 99% 1.00 2 fpc_mutex+0x8 page_get_mnode_freelist+0x140 1 0% 99% 1.00 2 fpc_mutex+0x20 page_get_mnode_freelist+0x140 1 0% 99% 1.00 30 fpc_mutex+0x30 page_list_add+0xd0 1 0% 99% 1.00 1 pse_mutex+0x238 page_unlock+0x1c 1 0% 99% 1.00 48 0x71680308 times+0x8 1 0% 99% 1.00 1 0x73005e38 ufs_getpage+0x86c 1 0% 99% 1.00 2 anonhash_lock+0x18 anon_decref+0x48 1 0% 99% 1.00 37 anonhash_lock+0x118 anon_decref+0x48 1 0% 99% 1.00 2 0x73834610 bunyip_name_addcomponent+0xb0 1 0% 99% 1.00 1 0x75856920 queuerun+0x198 1 0% 99% 1.00 66 0x801247d0 background+0xa8 1 0% 99% 1.00 2 0x71681900 times+0x8 1 0% 99% 1.00 20 0x7567239c strrput+0x4c0 1 0% 99% 1.00 3 0x7567239c strclean+0x10 1 0% 99% 1.00 1 0x804c9608 background+0xa8 1 0% 99% 1.00 30 0x7af3e4d8 background+0xa8 1 0% 99% 1.00 29 0x71684bd0 xsetitimer+0xec 1 0% 99% 1.00 8 0x71684530 times+0x8 1 0% 99% 1.00 8 0x70386038 callout_schedule_1+0x4 1 0% 99% 1.00 48 0x80897248 background+0xa8 1 0% 99% 1.00 25 0x80eeba70 background+0xa8 1 0% 99% 1.00 15 0x750891c0 background+0xa8 1 0% 99% 1.00 5 0x7f412708 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x320 page_unlock+0x1c 1 0% 99% 1.00 3 0x716846a8 xsetitimer+0xec 1 0% 99% 1.00 2 pse_mutex+0x3e8 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x2f8 page_try_reclaim_lock+0x18 1 0% 99% 1.00 30 0x79903bc8 background+0xa8 1 0% 99% 1.00 1 pse_mutex+0x308 page_trylock+0x18 1 0% 99% 1.00 2 0x7410dbf0 putq+0x38 1 0% 99% 1.00 2 0x71682718 times+0x8 1 0% 99% 1.00 32 0x716801d8 times+0x8 1 0% 99% 1.00 36 0x7fc33468 background+0xa8 1 0% 99% 1.00 46 0x8050e628 background+0xa8 1 0% 99% 1.00 23 0x80e5dc68 background+0xa8 1 0% 99% 1.00 32 0x763e15e8 background+0xa8 1 0% 99% 1.00 106 0x80e5c968 background+0xa8 1 0% 99% 1.00 39 0x7382eb18 background+0xa8 1 0% 99% 1.00 4 0x71681b08 clock+0x2b0 1 0% 99% 1.00 12 0x71682490 times+0x8 1 0% 99% 1.00 2 0x105cd360 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 27 0x75ce45a0 background+0xa8 1 0% 99% 1.00 1 0x741ed4c0 ufs_getpage+0x86c 1 0% 99% 1.00 8 0x716840b8 xsetitimer+0xec 1 0% 99% 1.00 31 0x71682720 times+0x8 1 0% 99% 1.00 11 0x1068a620 hat_unlock+0xfc 1 0% 99% 1.00 36 0x106e8fb8 hat_unload+0x16c 1 0% 99% 1.00 8 0x72af1738 traverse+0xa8 1 0% 99% 1.00 3 ctr_mutex+0x8 page_ctr_sub+0x30 1 0% 99% 1.00 81 fpc_mutex+0x48 page_get_mnode_freelist+0x140 1 0% 99% 1.00 2 ctr_mutex+0x48 page_ctr_add+0x30 1 0% 99% 1.00 1 pse_mutex+0x1d0 page_trylock+0x18 1 0% 99% 1.00 2 fpc_mutex+0x30 page_get_mnode_freelist+0x140 1 0% 99% 1.00 28 fpc_mutex+0x28 page_get_mnode_freelist+0x140 1 0% 99% 1.00 2 0x105d5598 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 50 0x7fc8da70 background+0xa8 1 0% 99% 1.00 16 0x71682328 times+0x8 1 0% 99% 1.00 2 0x71682328 xsetitimer+0xec 1 0% 99% 1.00 25 0x75062140 fifo_write+0x5c 1 0% 99% 1.00 21 0x7502b7b0 background+0xa8 1 0% 99% 1.00 4 0x75ca3198 background+0xa8 1 0% 99% 1.00 57 0x787327f0 background+0xa8 1 0% 99% 1.00 19 0x79b3d3e8 background+0xa8 1 0% 99% 1.00 1 0x71683ba8 times+0x8 1 0% 99% 1.00 130 0x71684ef0 times+0x8 1 0% 99% 1.00 2 0x71682908 times+0x8 1 0% 99% 1.00 42 0x75062000 fifo_close+0x280 1 0% 99% 1.00 1 0x73834a00 vn_rele+0x24 1 0% 99% 1.00 45 0x71683518 times+0x8 1 0% 99% 1.00 22 0x70239560 callout_schedule_1+0x4 1 0% 99% 1.00 22 0x71681980 times+0x8 1 0% 99% 1.00 16 0x716840b8 times+0x8 1 0% 99% 1.00 2 0x70289f90 qwriter_inner+0x8 1 0% 99% 1.00 52 0x7f0767c8 background+0xa8 1 0% 99% 1.00 62 0x716846f0 times+0x8 1 0% 99% 1.00 37 0x71680ad8 times+0x8 1 0% 99% 1.00 6 0x70238020 callout_execute+0x4 1 0% 99% 1.00 28 0x7023d020 dmpstrategy+0x58 1 0% 99% 1.00 32 0x7efeb0b8 background+0xa8 1 0% 99% 1.00 16 0x71681cb0 xsetitimer+0xec 1 0% 99% 1.00 12 0x724642c0 fifo_close+0x70 1 0% 99% 1.00 10 0x75062dc0 fifo_close+0x280 1 0% 99% 1.00 2 0x7289fc58 background+0xa8 1 0% 99% 1.00 27 0x812a64b8 background+0xa8 1 0% 99% 1.00 30 0x7a25c750 background+0xa8 1 0% 99% 1.00 2 0x756b39f8 background+0xa8 1 0% 99% 1.00 2 0x75bcd268 background+0xa8 1 0% 99% 1.00 64 0x716826c8 xsetitimer+0xec 1 0% 99% 1.00 21 0x71682ef8 times+0x8 1 0% 99% 1.00 43 0x71684030 times+0x8 1 0% 99% 1.00 2 0x71684e00 issig_forreal+0x18 1 0% 99% 1.00 39 0x75062140 fifo_close+0x280 1 0% 99% 1.00 30 fpc_mutex+0x78 page_list_add+0xd0 1 0% 99% 1.00 2 0x105d4488 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 2 volio_spin voliod_loop+0x174 1 0% 99% 1.00 1 anonhash_lock+0x1c8 anon_decref+0x48 1 0% 99% 1.00 10 0x7fcc96a0 cv_wait_sig_swap+0x1d8 1 0% 99% 1.00 3 0x71681898 sigaction+0xc4 1 0% 99% 1.00 22 0x7f181e10 leavesq+0xc 1 0% 99% 1.00 13 0x7191c898 dnlc_lookup+0x9c 1 0% 99% 1.00 1 0x72a66690 ufs_getattr+0x14c 1 0% 99% 1.00 15 0x105d3c48 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 51 0x719690d8 dnlc_lookup+0x9c 1 0% 99% 1.00 1 pse_mutex+0x1b0 page_trylock+0x18 1 0% 99% 1.00 2 0x7036a030 callout_thread+0x38 1 0% 99% 1.00 15 0x80565730 background+0xa8 1 0% 99% 1.00 53 0x750063a0 background+0xa8 1 0% 99% 1.00 52 0x8139d6d8 background+0xa8 1 0% 99% 1.00 55 0x7f181de8 background+0xa8 1 0% 99% 1.00 8 0x75057ab8 background+0xa8 1 0% 99% 1.00 2 0x7502b2b0 background+0xa8 1 0% 99% 1.00 18 0x71684530 xsetitimer+0xec 1 0% 99% 1.00 18 0x7479fd60 cv_wait_sig_swap+0x1d8 1 0% 99% 1.00 158 0x106a41e0 sfmmu_shadow_hcreate+0xf8 1 0% 99% 1.00 2 pse_mutex+0x2a0 page_try_reclaim_lock+0x18 1 0% 99% 1.00 2 0x71681980 xsetitimer+0xec 1 0% 99% 1.00 2 pse_mutex+0x1d8 page_unlock+0x1c 1 0% 99% 1.00 3 0x70388ab8 callout_schedule_1+0x4 1 0% 99% 1.00 2 pse_mutex+0x20 page_unlock+0x1c 1 0% 99% 1.00 6 pse_mutex+0x30 page_unlock+0x1c 1 0% 99% 1.00 1 pse_mutex+0x150 page_unlock+0x1c 1 0% 99% 1.00 13 0x7479f4a0 fifo_close+0x70 1 0% 99% 1.00 22 0x724642c0 fifo_read+0x24 1 0% 99% 1.00 3 0x724642c0 fifo_write+0x5c 1 0% 99% 1.00 1 pse_mutex+0x1c8 page_try_reclaim_lock+0x18 1 0% 99% 1.00 9 0x791db670 background+0xa8 1 0% 99% 1.00 10 0x79600fa8 background+0xa8 1 0% 99% 1.00 20 0x8142f500 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x170 page_unlock+0x1c 1 0% 99% 1.00 2 0x71684be8 sigaction+0xc4 1 0% 99% 1.00 22 0x7be5b240 background+0xa8 1 0% 99% 1.00 7 pse_mutex+0x228 page_unlock+0x1c 1 0% 99% 1.00 1 0x716850b0 times+0x8 1 0% 99% 1.00 59 0x716850b0 prinactive+0x64 1 0% 99% 1.00 1 pse_mutex+0xf0 page_trylock+0x18 1 0% 99% 1.00 2 pse_mutex+0xf8 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x170 page_trylock+0x18 1 0% 99% 1.00 2 pse_mutex+0x338 page_unlock+0x1c 1 0% 99% 1.00 2 pse_mutex+0xc8 page_try_reclaim_lock+0x18 1 0% 99% 1.00 29 0x71680cc0 xsetitimer+0xec 1 0% 99% 1.00 8 0x71684168 pr_read_psinfo+0x38 1 0% 99% 1.00 42 0x71681678 times+0x8 1 0% 99% 1.00 2 0x105cda60 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 2 pr_pidlock prfree+0x20 1 0% 99% 1.00 2 0x105d13a0 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 17 pidlock strioctl+0x2a6c 1 0% 99% 1.00 1 0x75831718 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x358 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x230 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x388 page_try_reclaim_lock+0x18 1 0% 99% 1.00 1 pse_mutex+0x148 page_hashout+0x160 1 0% 99% 1.00 7 0x71682450 prinactive+0x64 1 0% 99% 1.00 2 pse_mutex+0x78 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x1a0 page_try_reclaim_lock+0x18 1 0% 99% 1.00 2 pse_mutex+0xe0 page_unlock+0x1c 1 0% 99% 1.00 31 0x75ea88b0 background+0xa8 1 0% 99% 1.00 113 0x80897648 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x2a8 page_hashout+0x160 1 0% 99% 1.00 2 pse_mutex+0x198 page_unlock+0x1c 1 0% 99% 1.00 2 pse_mutex+0x338 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x118 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x398 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x3f0 page_trylock+0x18 1 0% 99% 1.00 2 pse_mutex+0x280 page_try_reclaim_lock+0x18 1 0% 99% 1.00 2 pse_mutex+0x380 page_trylock+0x18 1 0% 99% 1.00 8 0x80ffed98 background+0xa8 1 0% 99% 1.00 2 pse_mutex+0x290 page_trylock+0x18 1 0% 99% 1.00 2 pse_mutex+0x70 page_unlock+0x1c 1 0% 99% 1.00 2 pse_mutex+0x138 page_unlock+0x1c 1 0% 99% 1.00 2 pse_mutex+0x3f8 page_trylock+0x18 1 0% 99% 1.00 1 pse_mutex+0x50 page_trylock+0x18 1 0% 99% 1.00 68 0x760150c0 background+0xa8 1 0% 99% 1.00 38 0x7af3fad8 background+0xa8 1 0% 99% 1.00 8 0x80660d10 background+0xa8 1 0% 99% 1.00 2 0x105d0120 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 7 0x105d7c40 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 1 pse_mutex+0x288 page_trylock+0x18 1 0% 99% 1.00 2 0x7fcc96a0 fifo_close+0x70 1 0% 99% 1.00 2 0x7c6ff0cc strclean+0x10 1 0% 99% 1.00 3 0x7908c660 background+0xa8 1 0% 99% 1.00 21 0x750892c0 background+0xa8 1 0% 99% 1.00 63 0x105d7a88 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 29 0x716833a8 times+0x8 1 0% 99% 1.00 2 fpc_mutex+0x58 page_get_mnode_freelist+0x140 1 0% 99% 1.00 2 smap_mtx+0x90 segmap_release+0xb0 1 0% 99% 1.00 2 0x105cddc8 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 2 0x105d0260 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 1 0x73ee5b28 dnlc_lookup+0x12c 1 0% 99% 1.00 17 0x105d6db8 sfmmu_mlist_enter+0x84 1 0% 99% 1.00 1 0x71682d08 clock+0x2b0 1 0% 99% 1.00 2 0x7253ba30 ufs_addmap+0x20 1 0% 99% 1.00 46 0x750104a8 background+0xa8 1 0% 99% 1.00 20 0x78e4c640 background+0xa8 1 0% 99% 1.00 1 0x71683a60 times+0x8 1 0% 99% 1.00 13 ctr_mutex+0x70 page_ctr_sub+0x30 1 0% 99% 1.00 2 plocks+0x280 polldel+0x28 1 0% 99% 1.00 9 0x71682650 xsetitimer+0xec 1 0% 99% 1.00 22 0x72cdd950 bunyip_name_addcomponent+0x1d0 1 0% 99% 1.00 47 0x7f3766f8 background+0xa8 1 0% 99% 1.00 19 0x7f86d830 background+0xa8 1 0% 99% 1.00 13 0x796639b0 background+0xa8 1 0% 99% 1.00 13 0x703bb580 callout_schedule_1+0x4 1 0% 99% 1.00 10 0x7f412608 queuerun+0x198 1 0% 99% 1.00 8 fpc_mutex+0x60 page_list_add+0xd0 1 0% 99% 1.00 33 0x79b3d5e8 background+0xa8 1 0% 99% 1.00 59 0x7f91e540 background+0xa8 1 0% 99% 1.00 14 0x71684dc8 times+0x8 1 0% 99% 1.00 116 0x71681218 times+0x8 1 0% 99% 1.00 18 0x716851c0 clock+0x2b0 1 0% 99% 1.00 48 0x71684390 clock+0x2b0 1 0% 99% 1.00 9 0x75856220 queuerun+0x198 1 0% 99% 1.00 79 0x716824a0 times+0x8 1 0% 99% 1.00 16 0x7fe72fa8 background+0xa8 1 0% 99% 1.00 56 0x71681ba0 times+0x8 1 0% 99% 1.00 42 0x78d78f38 background+0xa8 1 0% 99% 1.00 21 0x716850a0 times+0x8 1 0% 99% 1.00 57 0x7ca677e8 queuerun+0x198 1 0% 99% 1.00 2 0x7fffc5c0 queuerun+0x198 1 0% 99% 1.00 51 0x761bfbd0 background+0xa8 1 0% 99% 1.00 95 0x716840d0 times+0x8 1 0% 99% 1.00 2 0x80ffedc0 rwnext+0x248 1 0% 99% 1.00 2 0x7f86c530 background+0xa8 1 0% 99% 1.00 19 0x75063cc0 cv_wait_sig_swap+0x1d8 1 0% 99% 1.00 5 0x7023aaa0 callout_execute+0x4 1 0% 99% 1.00 2 0x807dc368 drain_syncq+0x250 1 0% 99% 1.00 20 0x75062dc0 fifo_read+0x24 1 0% 99% 1.00 93 0x716843a0 times+0x8 1 0% 99% 1.00 100 0x789e7b10 background+0xa8 1 0% 99% 1.00 34 0x716824a0 xsetitimer+0xec 1 0% 99% 1.00 1 plocks+0x380 polldel+0x28 1 0% 99% 1.00 2 fpc_mutex+0x70 page_get_mnode_freelist+0x140 1 0% 99% 1.00 23 0x71681f78 clock+0x2b0 1 0% 99% 1.00 2 pse_mutex+0x300 page_hashout+0x160 1 0% 100% 1.00 13 0x71684050 times+0x8 1 0% 100% 1.00 70 0x72ede060 fifo_close+0x280 1 0% 100% 1.00 93 0x75062c80 fifo_close+0x280 1 0% 100% 1.00 47 0x75062c80 cv_wait_sig_swap+0x1d8 1 0% 100% 1.00 26 0x71681d60 times+0x8 1 0% 100% 1.00 98 0x7479e460 fifo_close+0x280 1 0% 100% 1.00 116 0x7479e460 fifo_close+0x70 1 0% 100% 1.00 137 0x7fcc9ba0 cv_wait_sig_swap+0x1d8 1 0% 100% 1.00 2 0x7fc0a43c strclean+0x10 1 0% 100% 1.00 14 0x71684e00 clock+0x2b0 1 0% 100% 1.00 1 pse_mutex+0x188 page_trylock+0x18 1 0% 100% 1.00 52 0x7830a6c0 background+0xa8 1 0% 100% 1.00 36 ctr_mutex+0x20 page_ctr_add+0x30 1 0% 100% 1.00 2 0x71684bc8 times+0x8 1 0% 100% 1.00 3 0x716833e8 times+0x8 1 0% 100% 1.00 2 0x79a77de0 background+0xa8 1 0% 100% 1.00 2 0x73896148 ufs_getpage+0x86c 1 0% 100% 1.00 63 0x71683010 times+0x8 1 0% 100% 1.00 2 0x73e72690 ufs_getpage+0x86c 1 0% 100% 1.00 52 0x764d84f8 background+0xa8 1 0% 100% 1.00 46 0x74b67410 background+0xa8 1 0% 100% 1.00 34 0x71682fd8 times+0x8 1 0% 100% 1.00 49 0x72ede060 fifo_read+0x24 1 0% 100% 1.00 23 0x71683c90 xsetitimer+0xec 1 0% 100% 1.00 1783 0x7303ac18 pollwakeup_safe+0x1ac 1 0% 100% 1.00 2 plocks+0x380 pollwakeup_safe+0x3c 1 0% 100% 1.00 2 0x803e1cf0 queuerun+0x198 1 0% 100% 1.00 58 0x79a772e0 background+0xa8 1 0% 100% 1.00 27 ph_mutex+0xf0 page_create_va+0x698 1 0% 100% 1.00 9 0x706edf6c segmap_smapadd+0xc 1 0% 100% 1.00 6 0x71684ef0 xsetitimer+0xec 1 0% 100% 1.00 61 0x71681280 times+0x8 1 0% 100% 1.00 2 0x73834760 vn_rele+0x24 1 0% 100% 1.00 2 0x738347d0 ufs_getattr+0x14c 1 0% 100% 1.00 9 shash_mtx+0x148 segmap_getmapflt+0x108 1 0% 100% 1.00 46 0x10685868 hat_unlock+0xfc 1 0% 100% 1.00 86 fpc_mutex+0x8 page_list_add+0xd0 1 0% 100% 1.00 8 ph_mutex+0x1a8 page_lookup+0x230 1 0% 100% 1.00 2 sfmmu_page_lock+0x1f8 sfmmu_page_enter+0x6c 1 0% 100% 1.00 27 anonhash_lock+0xa8 anon_decref+0x48 1 0% 100% 1.00 1 pcf+0x308 page_create_va+0x948 1 0% 100% 1.00 1 0x105d7dc0 sfmmu_mlist_enter+0x84 1 0% 100% 1.00 2 fpc_mutex+0x38 page_get_mnode_freelist+0x140 1 0% 100% 1.00 81 0x799f20d8 background+0xa8 1 0% 100% 1.00 45 0x71680d70 times+0x8 1 0% 100% 1.00 1 sfmmu_page_lock+0x20 sfmmu_page_enter+0x6c 1 0% 100% 1.00 19 0x71683b08 times+0x8 1 0% 100% 1.00 46 0x71682628 xsetitimer+0x160 1 0% 100% 1.00 54 0x81264ab0 background+0xa8 1 0% 100% 1.00 2 0x70310aa8 callout_schedule_1+0x4 1 0% 100% 1.00 62 0x71684500 times+0x8 1 0% 100% 1.00 45 0x7023aaa0 callout_schedule_1+0x4 1 0% 100% 1.00 10 0x75ca3098 background+0xa8 1 0% 100% 1.00 33 0x71682a38 times+0x8 1 0% 100% 1.00 31 0x7a177740 background+0xa8 1 0% 100% 1.00 74 0x71682dd8 times+0x8 1 0% 100% 1.00 36 0x105d3af0 sfmmu_mlist_enter+0x84 1 0% 100% 1.00 1 0x75062000 fifo_close+0x70 1 0% 100% 1.00 40 0x71684dc8 xsetitimer+0xec 1 0% 100% 1.00 34 0x784264d0 background+0xa8 1 0% 100% 1.00 420 0x1068e808 sfmmu_hblk_alloc+0x274 1 0% 100% 1.00 4 0x1068e808 hat_unload+0x16c 1 0% 100% 1.00 6 0x105d0658 sfmmu_mlist_enter+0x84 1 0% 100% 1.00 2 anonhash_lock+0x1e0 anon_decref+0x48 1 0% 100% 1.00 5 0x785210e0 background+0xa8 1 0% 100% 1.00 45 0x75c26170 background+0xa8 1 0% 100% 1.00 29 0x813ca4e8 background+0xa8 1 0% 100% 1.00 49 0x71681258 times+0x8 1 0% 100% 1.00 2 0x71682bc8 times+0x8 1 0% 100% 1.00 24 0x71680ed0 times+0x8 1 0% 100% 1.00 1 0x7028443c background+0xa8 1 0% 100% 1.00 3 0x71684e00 xsetitimer+0xec 1 0% 100% 1.00 2 0x75c32a90 background+0xa8 1 0% 100% 1.00 40 0x716841b8 times+0x8 1 0% 100% 1.00 2 0x71b355b8 socal_cq_enque+0x40 1 0% 100% 1.00 26 0x7b38ebf8 background+0xa8 1 0% 100% 1.00 27 0x71682598 times+0x8 1 0% 100% 1.00 20 0x71682598 xsetitimer+0xec 1 0% 100% 1.00 57 0x799636d0 background+0xa8 1 0% 100% 1.00 2 volio_spin voldiskiodone+0x2e0 1 0% 100% 1.00 36 0x803fc800 background+0xa8 1 0% 100% 1.00 8 0x756996f0 background+0xa8 1 0% 100% 1.00 2 0x71b02010 iommu_dma_bindhdl+0x250 1 0% 100% 1.00 17 0x7e414a50 background+0xa8 1 0% 100% 1.00 2 pio_mutex+0x2a8 page_io_unlock+0x18 1 0% 100% 1.00 11 pse_mutex+0x2a8 page_trylock+0x18 1 0% 100% 1.00 59 0x71683058 xsetitimer+0xec 1 0% 100% 1.00 13 0x105cf4f0 sfmmu_mlist_enter+0x84 1 0% 100% 1.00 8 0x79bb50f0 background+0xa8 1 0% 100% 1.00 2 0x79d9c808 background+0xa8 1 0% 100% 1.00 2 0x71684298 times+0x8 1 0% 100% 1.00 8 0x7f3b8900 background+0xa8 1 0% 100% 1.00 22 0x8141b56c putnext+0xc 1 0% 100% 1.00 32 0x7ac77560 background+0xa8 1 0% 100% 1.00 61 0x7f503a10 background+0xa8 1 0% 100% 1.00 2 0x71684388 xsetitimer+0xec 1 0% 100% 1.00 20 pidlock strioctl+0x29cc 1 0% 100% 1.00 20 0x7036b570 callout_thread+0x38 1 0% 100% 1.00 193 0x71684128 xsetitimer+0x160 1 0% 100% 1.00 1 0x71684128 pr_p_lock+0x38 1 0% 100% 1.00 2 0x71684cb8 times+0x8 1 0% 100% 1.00 2 0x71682e10 nanosleep+0x128 1 0% 100% 1.00 90 volio_spin voliod_iohandle+0x88 1 0% 100% 1.00 47 0x71683b08 pr_p_lock+0x38 1 0% 100% 1.00 8 0x71680e40 times+0x8 1 0% 100% 1.00 42 0x7f3764f8 background+0xa8 1 0% 100% 1.00 37 0x71681c80 times+0x8 1 0% 100% 1.00 6 plocks+0x340 prpoll+0xe0 1 0% 100% 1.00 204 0x7fcc97e0 fifo_close+0x70 1 0% 100% 1.00 1 sfmmu_page_lock+0xf0 sfmmu_page_enter+0x6c 1 0% 100% 1.00 75 0x737e0210 background+0xa8 1 0% 100% 1.00 12 0x756215e8 background+0xa8 1 0% 100% 1.00 47 0x76de2c58 background+0xa8 1 0% 100% 1.00 12 0x71682720 clock+0x2b0 1 0% 100% 1.00 2 0x71684228 times+0x8 1 0% 100% 1.00 2 pse_mutex+0xc8 page_trylock+0x18 1 0% 100% 1.00 17 0x7557fd38 background+0xa8 1 0% 100% 1.00 20 0x71681e38 times+0x8 1 0% 100% 1.00 2 0x105cd780 sfmmu_mlist_enter+0x84 1 0% 100% 1.00 2 pse_mutex+0x148 page_unlock+0x1c 1 0% 100% 1.00 45 pidlock sigcld_repost+0x48 1 0% 100% 1.00 48 0x716826c8 times+0x8 1 0% 100% 1.00 2 0x71684668 clock+0x2b0 1 0% 100% 1.00 2 0x706edf4c segmap_smapadd+0xc 1 0% 100% 1.00 44 0x7e520668 background+0xa8 1 0% 100% 1.00 6 ctr_mutex+0x60 page_ctr_sub+0x30 1 0% 100% 1.00 2 0x81392ba4 strrput+0x4c0 1 0% 100% 1.00 20 0x81392ba4 strclean+0x10 1 0% 100% 1.00 29 0x71684f60 xsetitimer+0xec 1 0% 100% 1.00 8 0x71681b48 xsetitimer+0xec 1 0% 100% 1.00 2 pse_mutex+0x100 page_try_reclaim_lock+0x18 1 0% 100% 1.00 57 pidlock sess_create+0x48 1 0% 100% 1.00 49 0x71683a58 xsetitimer+0xec 1 0% 100% 1.00 2 0x7453fba0 background+0xa8 1 0% 100% 1.00 21 0x71682710 times+0x8 1 0% 100% 1.00 2 plocks+0x160 polldel+0x28 ----------------------------------------------------------------------------- -- Adaptive mutex block: 766 events Count indv cuml rcnt nsec Lock Caller ----------------------------------------------------------------------------- -- 342 45% 45% 1.00 1406000 0x70538008 cv_wait_sig+0x1b8 215 28% 73% 1.00 1396551 0x70538008 semconv_lock+0x60 43 6% 78% 1.00 446313 0x724b2d30 qfestart+0x1f8 25 3% 82% 1.00 95639 service_queue qenable_locked+0x34 24 3% 85% 1.00 128170 service_queue background+0x120 21 3% 87% 1.00 123177 service_queue background+0xdc 20 3% 90% 1.00 73733 0x724b2d30 qfeintr+0x2bc 7 1% 91% 1.00 98611 volio_spin voliod_loop+0x174 6 1% 92% 1.00 316864 0x70538008 cv_wait_sig+0x230 6 1% 93% 1.00 2229996 tod_lock uniqtime+0x14 5 1% 93% 1.00 544026 0x7028731c putnext+0xc 5 1% 94% 1.00 3553192 pidlock pr_lookup_procdir+0x9c 4 1% 94% 1.00 596840 hblk8_lock sfmmu_hblk_alloc+0xdc 3 0% 95% 1.00 670927 0x704e4018 untimeout+0x18 2 0% 95% 1.00 765259 hblk8_lock sfmmu_hblk_tofreelist+0x38 2 0% 95% 1.00 159438 0x70387578 timeout_common+0x4 2 0% 96% 1.00 29227 0x7028473c putq+0x38 2 0% 96% 1.00 47421 0x70239560 timeout_common+0x4 2 0% 96% 1.00 138366 0x70388ab8 timeout_common+0x4 1 0% 96% 1.00 57997 0x79662fb0 background+0xa8 1 0% 96% 1.00 81407 0x702847c0 putnext+0x30 1 0% 96% 1.00 251520 0x75bcd468 background+0xa8 1 0% 97% 1.00 116846 0x7a05a95c cv_wait_sig+0x1b8 1 0% 97% 1.00 132330 0x75258ed0 background+0xa8 1 0% 97% 1.00 30498 0x71682628 clock+0x2b0 1 0% 97% 1.00 19374 0x71684540 clock+0x2b0 1 0% 97% 1.00 35064 0x78e5c38c cv_wait_sig+0x1b8 1 0% 97% 1.00 96629 0x7249b220 ufs_lockfs_end+0x24 1 0% 97% 1.00 163760 0x799634d0 background+0xa8 1 0% 98% 1.00 681127 0x7190d898 dnlc_lookup+0x9c 1 0% 98% 1.00 302395 0x76783c10 background+0xa8 1 0% 98% 1.00 22604 0x81392ba4 cv_wait_sig+0x1b8 1 0% 98% 1.00 193856716 kstat_chain_lock read_kstat_data+0x34 1 0% 98% 1.00 92620 ph_mutex+0x38 page_create_va+0x698 1 0% 98% 1.00 26383 0x70388ab8 callout_schedule_1+0x4 1 0% 98% 1.00 1210276 0x7fc0b3fc cv_wait_sig+0x1b8 1 0% 98% 1.00 36509 0x71684170 clock+0x2b0 1 0% 99% 1.00 55299 0x756cff00 background+0xa8 1 0% 99% 1.00 177998 0x70239560 untimeout+0x18 1 0% 99% 1.00 22047 0x8141b86c cv_wait_sig+0x1b8 1 0% 99% 1.00 125510 0x702847c0 leavesq+0xc 1 0% 99% 1.00 135051 0x76bb9c40 background+0xa8 1 0% 99% 1.00 60529 0x7249b220 ufs_lockfs_begin_getpage+0x38 1 0% 99% 1.00 85796 0x784a57d8 background+0xa8 1 0% 99% 1.00 24589 0x70289f90 putnext+0x88 1 0% 100% 1.00 73274 lwpchanlock+0x118 lwp_sema_wait+0x3e8 1 0% 100% 1.00 167944 0x75b66460 background+0xa8 1 0% 100% 1.00 128238 0x7479e820 cv_wait_sig_swap+0x1d8 1 0% 100% 1.00 62379 0x704e4018 timeout_common+0x4 ----------------------------------------------------------------------------- -- Spin lock spin: 5559 events Count indv cuml rcnt spin Lock Caller ----------------------------------------------------------------------------- -- 1063 19% 19% 1.00 546 sleepq_head+0x79c cv_signal+0x24 272 5% 24% 1.00 438 cpu[0]+0x44 disp_getbest+0x4 261 5% 29% 1.00 191 cpu[0]+0x44 setbackdq+0x188 254 5% 33% 1.00 463 cpu[12]+0x44 disp_getbest+0x4 246 4% 38% 1.00 484 cpu[11]+0x44 disp_getbest+0x4 243 4% 42% 1.00 503 cpu[14]+0x44 disp_getbest+0x4 230 4% 46% 1.00 501 cpu[10]+0x44 disp_getbest+0x4 218 4% 50% 1.00 461 cpu[8]+0x44 disp_getbest+0x4 209 4% 54% 1.00 513 cpu[9]+0x44 disp_getbest+0x4 170 3% 57% 1.00 185 0x72b9ff7b mutex_adaptive_release+0x78 151 3% 60% 1.00 173 cpu[14]+0x44 setbackdq+0x188 149 3% 62% 1.00 629 0x72b9ff3b mutex_adaptive_release+0x78 141 3% 65% 1.00 206 cpu[8]+0x44 setbackdq+0x188 140 3% 67% 1.00 513 cpu[2]+0x44 disp_getbest+0x4 135 2% 70% 1.00 147 cpu[12]+0x44 setbackdq+0x188 123 2% 72% 1.00 164 cpu[11]+0x44 setbackdq+0x188 123 2% 74% 1.00 170 cpu[10]+0x44 setbackdq+0x188 117 2% 76% 1.00 737 0x72b9ff3b mutex_vector_enter+0x208 104 2% 78% 1.00 165 cpu[0]+0x44 disp+0x78 94 2% 80% 1.00 165 0x72b9ff7b mutex_vector_enter+0x208 92 2% 82% 1.00 199 cpu[14]+0x44 setfrontdq+0x128 79 1% 83% 1.00 246 cpu[14]+0x44 disp+0x78 77 1% 84% 1.00 228 cpu[0]+0x44 setfrontdq+0x128 76 1% 86% 1.00 151 cpu[11]+0x44 disp+0x78 76 1% 87% 1.00 179 cpu[9]+0x44 setbackdq+0x188 69 1% 88% 1.00 164 cpu[8]+0x44 disp+0x78 69 1% 90% 1.00 263 cpu[12]+0x44 setfrontdq+0x128 65 1% 91% 1.00 192 cpu[12]+0x44 disp+0x78 61 1% 92% 1.00 243 cpu[11]+0x44 setfrontdq+0x128 57 1% 93% 1.00 182 cpu[10]+0x44 disp+0x78 55 1% 94% 1.00 228 cpu[8]+0x44 setfrontdq+0x128 53 1% 95% 1.00 267 cpu[10]+0x44 setfrontdq+0x128 51 1% 96% 1.00 144 cpu[9]+0x44 disp+0x78 49 1% 97% 1.00 236 sleepq_head+0x79c cv_block+0x70 45 1% 97% 1.00 276 cpu[9]+0x44 setfrontdq+0x128 32 1% 98% 1.00 200 cpu[2]+0x44 setfrontdq+0x128 28 1% 99% 1.00 100 0x7053800e mutex_vector_enter+0x13c 21 0% 99% 1.00 218 cpu[2]+0x44 disp+0x78 20 0% 99% 1.00 254 cpu[2]+0x44 setbackdq+0x188 12 0% 99% 1.00 181 service_queue+0x6 mutex_vector_enter+0x13c 3 0% 100% 1.00 85 sleepq_head+0x2ac sema_v+0x10 3 0% 100% 1.00 38 hres_lock+0x3 hr_clock_lock+0x10 2 0% 100% 1.00 111 hblk8_lock+0x6 mutex_vector_enter+0x13c 2 0% 100% 1.00 324 sleepq_head+0x2ac sema_tryp+0x14 1 0% 100% 1.00 1 sleepq_head+0xdfc cv_broadcast+0x24 1 0% 100% 1.00 198 0x72b9ffdb mutex_adaptive_release+0x78 1 0% 100% 1.00 896 sleepq_head+0x5d4 cv_block+0x70 1 0% 100% 1.00 49 sleepq_head+0xb7c cv_broadcast+0x24 1 0% 100% 1.00 231 sleepq_head+0x9dc cv_block+0x70 1 0% 100% 1.00 1047 sleepq_head+0x7d4 cv_block+0x70 1 0% 100% 1.00 412 sleepq_head+0x22c cv_signal+0x24 1 0% 100% 1.00 614 sleepq_head+0x30c cv_block+0x70 1 0% 100% 1.00 41 0x724b2d36 mutex_vector_enter+0x13c 1 0% 100% 1.00 676 sleepq_head+0x42c sema_v+0x10 1 0% 100% 1.00 263 sleepq_head+0xea4 cv_broadcast+0x24 1 0% 100% 1.00 868 sleepq_head+0xcdc cv_block+0x70 1 0% 100% 1.00 157 sleepq_head+0xf44 cv_block+0x70 1 0% 100% 1.00 1 sleepq_head+0xec4 cv_broadcast+0x24 1 0% 100% 1.00 309 sleepq_head+0x994 cv_block+0x70 1 0% 100% 1.00 292 sleepq_head+0x44c cv_block+0x70 1 0% 100% 1.00 635 0x7600dfb4 rw_exit_wakeup+0x94 1 0% 100% 1.00 23 sleepq_head+0x1bc cv_block+0x70 1 0% 100% 1.00 4 0x72b9ff7b mutex_vector_enter+0x174 ----------------------------------------------------------------------------- -- Thread lock spin: 83 events Count indv cuml rcnt spin Lock Caller ----------------------------------------------------------------------------- -- 29 35% 35% 1.00 302 sleepq_head+0x79c setrun+0x4 5 6% 41% 1.00 212 0x72b9ff3b ts_tick+0x8 3 4% 45% 1.00 167 cpu[0]+0x44 ts_tick+0x8 2 2% 47% 1.00 36 sleepq_head+0x79c pi_fix_epri+0x4c 2 2% 49% 1.00 38 cpu[8]+0x68 cv_wait_sig+0xb0 2 2% 52% 1.00 13 cpu[11]+0x68 preempt+0x1c 1 1% 53% 1.00 71 cpu[9]+0x68 sigtoproc+0x2c4 1 1% 54% 1.00 294 sleepq_head+0xd4c setrun+0x4 1 1% 55% 1.00 217 sleepq_head+0xabc ts_tick+0x8 1 1% 57% 1.00 256 sleepq_head+0xafc ts_tick+0x8 1 1% 58% 1.00 238 sleepq_head+0x624 setrun+0x4 1 1% 59% 1.00 89 cpu[9]+0x68 preempt+0x1c 1 1% 60% 1.00 84 cpu[12]+0x44 sigtoproc+0x2c4 1 1% 61% 1.00 132 sleepq_head+0xd74 ts_tick+0x8 1 1% 63% 1.00 282 sleepq_head+0x4ac ts_tick+0x8 1 1% 64% 1.00 80 cpu[9]+0x44 ts_tick+0x8 1 1% 65% 1.00 26 sleepq_head+0xda4 ts_update_list+0x60 1 1% 66% 1.00 6 cpu[0]+0x68 cv_wait_sig+0xb0 1 1% 67% 1.00 115 cpu[8]+0x44 sigtoproc+0x2c4 1 1% 69% 1.00 161 sleepq_head+0xddc ts_tick+0x8 1 1% 70% 1.00 5 sleepq_head+0xbfc sigtoproc+0x2c4 1 1% 71% 1.00 20 cpu[11]+0x44 sigtoproc+0x2c4 1 1% 72% 1.00 293 cpu[12]+0x44 prchoose+0x68 1 1% 73% 1.00 241 cpu[14]+0x44 ts_tick+0x8 1 1% 75% 1.00 10 cpu[14]+0x44 sigtoproc+0x2c4 1 1% 76% 1.00 22 cpu[9]+0x44 sigtoproc+0x2c4 1 1% 77% 1.00 225 cpu[0]+0x44 sigtoproc+0x2c4 1 1% 78% 1.00 50 cpu[11]+0x44 ts_tick+0x8 1 1% 80% 1.00 178 sleepq_head+0xfc4 ts_tick+0x8 1 1% 81% 1.00 243 sleepq_head+0x42c ts_tick+0x8 1 1% 82% 1.00 86 sleepq_head+0x6c ts_tick+0x8 1 1% 83% 1.00 406 sleepq_head+0xcdc ts_tick+0x8 1 1% 84% 1.00 133 sleepq_head+0xde4 ts_tick+0x8 1 1% 86% 1.00 5 sleepq_head+0xcec ts_update_list+0x60 1 1% 87% 1.00 77 cpu[0]+0x68 ts_tick+0x8 1 1% 88% 1.00 63 sleepq_head+0x9cc setrun+0x4 1 1% 89% 1.00 114 sleepq_head+0xdc4 ts_tick+0x8 1 1% 90% 1.00 108 sleepq_head+0x4cc ts_tick+0x8 1 1% 92% 1.00 41 cpu[12]+0x68 preempt+0x1c 1 1% 93% 1.00 520 sleepq_head+0xcdc setrun+0x4 1 1% 94% 1.00 183 sleepq_head+0x8bc ts_tick+0x8 1 1% 95% 1.00 129 sleepq_head+0xc5c ts_tick+0x8 1 1% 96% 1.00 13 cpu[10]+0x68 cv_wait_sig+0xb0 1 1% 98% 1.00 382 sleepq_head+0x634 prchoose+0x68 1 1% 99% 1.00 117 sleepq_head+0xd4c ts_update_list+0x60 1 1% 100% 1.00 294 sleepq_head+0x3bc ts_tick+0x8 ----------------------------------------------------------------------------- -- R/W writer blocked by writer: 11 events Count indv cuml rcnt nsec Lock Caller ----------------------------------------------------------------------------- -- 5 45% 45% 1.00 26679356 0x73bbbf48 write+0x84 2 18% 64% 1.00 8311176 0x759428d8 pwrite64+0x170 2 18% 82% 1.00 3529571 0x7db33b68 pwrite64+0x170 1 9% 91% 1.00 14441149 0x73a72790 write+0x84 1 9% 100% 1.00 12546186 0x73a72640 write+0x84 ----------------------------------------------------------------------------- -- R/W writer blocked by readers: 1 events Count indv cuml rcnt nsec Lock Caller ----------------------------------------------------------------------------- -- 1 100% 100% 1.00 162874 0x7600dfac as_map+0x6c ----------------------------------------------------------------------------- -- R/W reader blocked by writer: 2 events Count indv cuml rcnt nsec Lock Caller ----------------------------------------------------------------------------- -- 1 50% 50% 1.00 11314336 0x73bbbf48 pread+0xdc 1 50% 100% 1.00 14818874 0x73a72640 pread+0xdc ----------------------------------------------------------------------------- -- Lockstat record failure: 1 events Count indv cuml rcnt Lock Caller ----------------------------------------------------------------------------- -- 1 100% 100% 0.00 lockstat_lock lockstat_add_to_hash ----------------------------------------------------------------------------- -- I have volume manager 2.5 my system file is: set msgsys:msginfo_msgmap=258 set msgsys:msginfo_msgmax=32768 set msgsys:msginfo_msgmnb=65535 set msgsys:msginfo_msgmni=128 set msgsys:msginfo_msgtql=256 set msgsys:msginfo_msgseg=7168 set msgsys:msginfo_msgssz=32 set shmsys:shminfo_shmmin=1 *set shmsys:shminfo_shmmax=268435456 set shmsys:shminfo_shmmax=2684354560 set shmsys:shminfo_shmseg=200 set shmsys:shminfo_shmmni=1000 set semsys:seminfo_semmap=258 *set semsys:seminfo_semmni=2000 set semsys:seminfo_semmni=1000 set semsys:seminfo_semmns=10000 *set semsys:seminfo_semmns=4500 set semsys:seminfo_semmnu=1000 *set semsys:seminfo_semmsl=600 set semsys:seminfo_semmsl=4500 *set semsys:seminfo_semopm=20 set semsys:seminfo_semopm=200 set semsys:seminfo_semvmx=32767 set semsys:seminfo_semaem=16384 set semsys:seminfo_semume=100 forceload: sys/msgsys forceload: sys/shmsys forceload: sys/semsys * set pln:pln_enable_detach_suspend=1 set soc:soc_enable_detach_suspend=1 * SUNWecsr end * Number of Users JM3 set pt_cnt=1024 set npty=1024 set rlim_fd_max=8096 * jm3 set qfe:qfe_adv_autoneg_cap=0 set qfe:qfe_adv_100T4_cap=0 set qfe:qfe_adv_100fdx_cap=1 set qfe:qfe_adv_100hdx_cap=0 set qfe:qfe_adv_10fdx_cap=0 set qfe:qfe_adv_10hdx_cap=0 * Priority Paging. jm3 set priority_paging=1 set fastscan=16000 set handspreadpages=16000 set maxpgio=65536 * DR jm3. set dr_max_mem=16384 #set dr-max-mem=16384 # Network queue JM3 set sq_max_size=0x30 * Monitoring RAM State JM3 set report_ce_console=1 * VM FS Cached JM3 set bufhwm=56000 set ncsize=36000 set ufs_ninode=36000 * vxvm_START (do not remove) forceload: drv/vxdmp forceload: drv/vxio forceload: drv/vxspec forceload: drv/ssd forceload: drv/sf forceload: drv/SUNW,socal forceload: drv/sbus forceload: drv/sd forceload: drv/pln forceload: drv/soc forceload: drv/socal rootdev:/pseudo/vxio@0:0 set vxio:vol_rootdev_is_volume=1 * vxvm_END (do not remove) _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:28:27 EDT