From: Michael Hase (michael@six.de)
Date: Wed Jun 19 2002 - 12:32:36 EDT
Hi all,
lately our E250 with 2gb running solaris 7 (kernel 106541-16) runs a
bit slowly. The box is quite loaded with oracle, mysql, some
apache/php processes, samba and nfs serving. We noticed the slowdown
when we loaded some webobjects java apps onto the box eating about
300-700mb of ram (dependent on number of app instances), Oracle is
configured at about 200mb (just development). About 36 days uptime.
Some days ago we started virtual-adrian.se and it sometimes reports
slow disks, thats ok with our workload. But since we loaded those
webobjects apps it reports ram shortage quite often, although there
should be some memory left if we sum up user space ram usage.
Today I learned about memtool and I think there are some strange
numbers:
# prtmem
Total memory: 1970 Megabytes
Kernel Memory: 642 Megabytes
Application: 1070 Megabytes
Executable & libs: 83 Megabytes
File Cache: 99 Megabytes
Free, file cache: 56 Megabytes
Free, free: 18 Megabytes
Imho the solaris kernel should not occupy 642 Megabytes, at least not
on a 2gb box. Thats nearly 1/3 of all available ram.
echo kmastat | crash
gives the following suspicious line (full listing below):
kmem_alloc_1120 1120 29818 336826 394182656 672801271 0
thats 375mb occupied by those 1120 bytes buffers. I don't know any
solaris kernel internals, but this seems a little high. The other
numbers look reasonable, at least to me.
Now the questions: could this be a kernel memory leak? how can we
avoid this excessive kernel memory usage?
Any ideas?
cheers,
Michael
#######################
full kmastat output:
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> buf buf buf memory #allocations
cache name size avail total in use succeed fail
---------- ----- ----- ----- -------- ------- ----
kmem_magazine_1 16 366 508 8192 67995 0
kmem_magazine_3 32 61 12954 417792 5538839 0
kmem_magazine_7 64 26 3683 237568 760079 0
kmem_magazine_15 128 16 14427 1875968 3319402 0
kmem_magazine_31 256 0 0 0 221222 0
kmem_magazine_47 384 6 2394 933888 183153 0
kmem_magazine_63 512 0 0 0 0 0
kmem_magazine_95 768 0 0 0 0 0
kmem_magazine_143 1152 0 0 0 0 0
kmem_slab_cache 56 11 60706 3915776 1696826 0
kmem_bufctl_cache 32 157 423164 13647872 12545919 0
kmem_bufctl_audit_cache 184 36 420 81920 1408 0
kmem_pagectl_cache 32 144 384 24576 9015965 0
kmem_alloc_8 8 29097 112887 909312 145742503 0
kmem_alloc_16 16 30697 68072 1097728 1047792191 0
kmem_alloc_24 24 8733 30510 737280 81619755 0
kmem_alloc_32 32 23798 32512 1048576 19331195 0
kmem_alloc_40 40 6133 8932 360448 15042022 0
kmem_alloc_48 48 879 3718 180224 159083731 0
kmem_alloc_56 56 29625 336690 19021824 14595757 0
kmem_alloc_64 64 1048 2413 155648 51054654 0
kmem_alloc_80 80 1334 8787 712704 3753086 0
kmem_alloc_96 96 51 672 65536 24059729 0
kmem_alloc_112 112 42 11952 1359872 342585 0
kmem_alloc_128 128 83 1386 180224 3315378 0
kmem_alloc_144 144 98 1008 147456 1227184 0
kmem_alloc_160 160 1065 1900 311296 4735680 0
kmem_alloc_176 176 35 414 73728 4263434 0
kmem_alloc_192 192 61 168 32768 236947 0
kmem_alloc_208 208 132 702 147456 9454212 0
kmem_alloc_224 224 680 3348 761856 73313274 0
kmem_alloc_240 240 43 363 90112 12690590 0
kmem_alloc_256 256 32 558 147456 76549118 0
kmem_alloc_320 320 33 3050 999424 990020 0
kmem_alloc_384 384 27 84 32768 910314 0
kmem_alloc_448 448 35 198 90112 43036726 0
kmem_alloc_512 512 36 1184 606208 1655431 0
kmem_alloc_544 544 57 585 319488 91207 0
kmem_alloc_640 640 21 144 98304 1237883 0
kmem_alloc_768 768 28 330 270336 43729954 0
kmem_alloc_864 864 31 126 114688 491808 0
kmem_alloc_992 992 2559 3576 3661824 3930334 0
kmem_alloc_1120 1120 29818 336826 394182656 672801271 0
kmem_alloc_1312 1312 23 204 278528 2023901 0
kmem_alloc_1600 1600 26 140 229376 5331088 0
kmem_alloc_2048 2048 33 504 1032192 9362258 0
kmem_alloc_2688 2688 19 129 352256 14062959 0
kmem_alloc_4096 4096 14 106 434176 8667862 0
kmem_alloc_8192 8192 39 5664 46399488 25752042 0
kmem_alloc_12288 12288 5 26 319488 377752 0
kmem_alloc_16384 16384 14 109 1785856 701443 0
streams_mblk 64 4355 17526 1130496 468428213 0
streams_dblk_32 128 184 1071 139264 57177533 0
streams_dblk_64 160 4935 8100 1327104 384398119 0
streams_dblk_96 192 2395 2772 540672 65328518 0
streams_dblk_128 224 1131 1728 393216 43683228 0
streams_dblk_160 256 217 217 57344 209094825 0
streams_dblk_256 352 1776 2599 925696 70899234 0
streams_dblk_544 640 118 120 81920 15356496 0
streams_dblk_1024 1120 315 315 368640 44590805 0
streams_dblk_1536 1632 180 180 294912 58227674 0
streams_dblk_1920 2016 143 208 425984 772633527 0
streams_dblk_2592 2688 30 30 81920 213352249 0
streams_dblk_3968 4064 4 4 16384 1107555 0
streams_dblk_8064 8160 13 13 106496 20577879 0
streams_dblk_12160 12256 18 18 221184 17126978 0
streams_dblk_esb 96 84 84 8192 9817397 0
task_cache 32 36 254 8192 218 0
taskq_cache 120 63 67 8192 4 0
bp_map_16384 16384 4 4 65536 2659743 0
bp_map_32768 32768 9 9 294912 3506855 0
bp_map_49152 49152 6 6 294912 278149 0
bp_map_65536 65536 0 0 0 0 0
bp_map_81920 81920 0 0 0 0 0
bp_map_98304 98304 0 0 0 0 0
bp_map_114688 114688 0 0 0 0 0
bp_map_131072 131072 0 0 0 0 0
sfmmuid_cache 48 77 338 16384 443983 0
sfmmu8_cache 312 1419 30784 9699328 91096 0
sfmmu1_cache 88 727 10028 892928 204980 0
ism_blk_cache 200 32 62 16384 1502 0
ism_ment_cache 32 224 254 8192 1506 0
seg_cache 56 2159 14500 819200 13591190 0
ddi_callback_cache 40 0 0 0 0 0
log_event_cache 1032 14 15 16384 1 0
thread_cache 624 92 1068 729088 316117 0
lwp_cache 872 95 1071 974848 86569 0
turnstile_cache 64 220 1270 81920 596199 0
cred_cache 96 15773 78624 7667712 69686697 0
file_cache 56 1502 4205 237568 185141270 0
stream_head_cache 328 975 1968 671744 12005589 0
queue_cache 456 2928 5916 2850816 29257727 0
syncq_cache 120 54 134 16384 3329 0
qband_cache 64 125 127 8192 2 0
linkinfo_cache 48 129 169 8192 823 0
strevent_cache 48 145 169 8192 3690630 0
as_cache 120 75 335 40960 447659 0
seg_skiplist_cache 32 0 0 0 0 0
anon_cache 48 5655 184886 8962048 17848500 0
anonmap_cache 48 942 12168 589824 5008251 0
segvn_cache 88 2136 14444 1286144 10951981 0
flk_edges 48 100 169 8192 530225 0
fdb_cache 104 0 0 0 0 0
physio_buf_cache 224 36 36 8192 36881224 0
snode_cache 248 54 736 188416 3966996 0
ufs_inode_cache 440 75753 106884 48644096 214865530 0
directio_buf_cache 248 32 32 8192 168425 0
kmem_alloc_le_1024 1024 4 8 8192 4 0
glm0_cache 384 267 273 106496 12919643 0
glm1_cache 384 19 21 8192 44255162 0
glm2_cache 384 309 315 122880 72864594 0
glm3_cache 384 162 168 65536 48159433 0
error reading stats for kmem_alloc_le_256
sock_cache 416 952 1577 679936 2282732 0
sock_unix_cache 416 43 57 24576 9015235 0
process_cache 2624 18 279 761856 261063 0
fnode_cache 256 21 31 8192 4893 0
pipe_cache 480 34 64 32768 86626 0
kmem_alloc_le_8192 8192 0 1 8192 1 0
lufs save 32 254 254 8192 7442335 0
lufs bufs 232 35 35 8192 7720912 0
authkern_cache 72 106 113 8192 53790942 0
authloopback_cache 72 113 113 8192 28199 0
authdes_cache_handle 72 0 0 0 0 0
rnode_cache 616 13 70005 44113920 3541074 0
nfs_access_cache 40 17724 131950 5324800 12566788 0
lm_vnode 168 47 48 8192 23 0
lm_xprt 32 253 254 8192 1 0
lm_sysid 136 57 59 8192 10 0
lm_client 64 124 127 8192 37 0
lm_async 32 0 0 0 0 0
lm_sleep 88 0 0 0 0 0
lm_config 80 100 101 8192 6 0
exi_cache_handle 40 155 203 8192 2225 0
fctl_cache 112 0 0 0 0 0
cachefs_cnode_cache 1256 0 0 0 0 0
cachefs_async_request 56 0 0 0 0 0
cachefs_fscache 960 0 0 0 0 0
cachefs_filegrp 112 0 0 0 0 0
cachefs_cache_t 448 0 0 0 0 0
bunyip_name_cache 112 31 16416 1867776 16385 0
bunyip_mem_cache 64 1502 1524 98304 1515 0
---------- ----- ----- ----- -------- ------- ----
permanent - - - 393216 4100 0
oversize - - - 28205056 8778445 0
---------- ----- ----- ----- -------- ------- ----
Total - - - 672407552 1669516373 0
>
-- Michael Hase Six Offene Systeme GmbH michael@six.de Sielminger Str. 63 http://www.six.de 70771 Leinfelden-Echterdingen phone +49 711 99091 62 Germany _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:24:29 EDT