From: JOSEPH KREMBLAS (jkremblas@REDHEARTGIFTS.COM)
Date: Wed Jan 14 2004 - 14:02:26 EST
I understand. Concerning adding additional paging space to external disks, I
would be alert to what happens when you reboot the node. If the external
disks are not available and you've enabled them for automatic activation at
boot time, this could present some problems. Depending on the version of AIX
you have, you can dynamically add paging space on the fly without a reboot.
As for "SAP keeps all the paging space and dose [sic] not release it," this
depends on a variety of things. First, you must ask yourself, "Is early page
space allocation enabled before SAP starts?" Because of IBM AIX's
nonconformity to the POSIX 1003.1 standard, IBM introduced the "early page
space allocation" algorithm (PSALLOC=early), as typically used in all UNIX
platforms implementations, by default, to prevent AIX from going on "witch
hunts" to dispose of threads (or processes) by issuing SIGDANGER and SIGKILL
system calls and later punish them for using "late page space allocation,"
which meant that maybe there was enough page space or maybe there wasn't to
service all the threads. The important thing to remember here about early
and late page space allocation is this:
early page space allocation = immune to "witch hunts" and SIGDANGER and
SIGKILL calls conducted by AIX
late page space allocation = you will be subject to SIGDANGER and
SIGKILL calls by AIX
Also, understand that one of the key designs of AIX was the concept of a
dynamic kernel that was intertwined very closely with VMM to implement
what's called demand paging. Most, if not all, of the AIX kernel is in
virtual memory in order to free up segments for other processes. This is the
area that made AIX so successfull--the concept of a dynamic kernel and a
dynamic VMM.
Even when SAP "keeps all the paging space" this may not be true, because it
seems this way especially if she is using the "late page space algorithm".
Remember, that whenever a "page fault" occurs in AIX, the "page stealer" is
activated.
page fault = virtual-->real memory address does not exist
When the "page stealer" is activated, he first determines if this is the
first request. If it is, the an "initial page fault" algorithm is executed
that allocates virtual memory (page space) and real memory. He allocates the
page space memory so that when he goes on a "witch hunt" later on in the
process of things, he'll copy the real memory segment to the page space to
free up real memory so he doesn't have to do this later. This explains the
"why" of why applications start so slowly the first time and when restarted
it starts much faster. If the "page stealer" determines that this isn't the
first request, he goes on a "witch hunt" to free real memory pages in one of
two ways. First, he consults a "least recently used (LRU)" list which is
nothing more than a table of real memory pages that haven't been used very
much. If the page is "dirty"--meaning data has been modified in
memory--he'll dump this to page space, which causes a page-out (po) on
vmstat output. The page-out is only for "dirty" pages. "Dirty" pages are
nonpersistent working pages, a.k.a. nonfile pages, not persistent (file)
pages used by LVM filesystems and NFS filesystems. This is important to
note. You'll never see file pages in the page-out column on vmstat output,
because file pages go through the LVM via VMM to the filesystem cache. If
the page is "clean"--meaning that the page in real memory hasn't been
recently used, he simply purges it from real memory to free up the free page
list (fr column in vmstat output) and no page-out occurs, because the image
is already in page space.
So as processes come and go, a page fault is always activated if the process
is not in the process table. The page stealer does his work to make room for
the process and this process continues, ad infinitum.
po = dirty, nonpersistent working segments (includes private data
segments that are supposed to stay in memory; the AIX justice system says
different)
fr = page steals by page stealer; also page reclaims
pi = pages from page space needed again in real memory
So, in essence, and by default, the pages in memory by SAP will most likely
end up in the page space file, because the page stealer will make sure of it
provided there is enough page space.
I could talk about thrashing and those kinds of things, but I don't want to
bore the group. I would certainly consider adjusting minperm and maxperm
using the /usr/samples/kernel/vmtune command. Typically, maxperm is set to
about 80% by default and minperm is set at about 20%. I would definitely
start by reducing these numbers for your SAP application. I would also
consider adjusting the sequential read-ahead values as well (-r and -R on
vmtune output). Additionally, the -W (for random write-behind) and -c (for
sequential write-behind) should be adjusted as well. As for the schedtune
command, ensure that -h is set to 0. This area relates to the AIX scheduler
and on systems with greater than 128MB of memory, this value should be 0 to
effectively turn this guy off. Even on systems with less than 128MB of
memory I would turn this off. The default is 6. The -h in schedtune is used
to calculate whether the system is thrashing. The formula is:
Thrashing = po/fr > 1/h
or
Thrashing = po*h > fr
where h=-h from schedtune; po and fr = output columns from vmstat
command.
If Thrashing=1, then useless system. You are probably experiencing sever
memory starvation with lots of po's.
I hope this helps.
--joseph
-----Original Message-----
From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of
Wilson, Jeff
Sent: Wednesday, January 14, 2004 8:59 AM
To: aix-l@Princeton.EDU
Subject: Re: Paging space on a fastT
We are running the paging monster app "SAP on DB2". I have never seen an
app eat paging space like this one. If we bounce SAP we get it all back but
until we do that SAP keeps all the paging space and dose not release it. As
always the recommend 2 X physical memory but with the new SAP installs they
want 3 X physical memory which is ridicules but I have no choice in the
matter so I have to add it even if they don't need it.
Jeff Wilson
303-737-5399
-----Original Message-----
From: JOSEPH KREMBLAS [mailto:jkremblas@REDHEARTGIFTS.COM]
Sent: Wednesday, January 14, 2004 8:34 AM
To: aix-l@Princeton.EDU
Subject: Re: Paging space on a fastT
Jeff,
Before throwing more paging space at the system, have you ran vmstat and
iostat to see what the system is doing? Can you provide a vmstat listing and
iostat listing? Are you running Oracle? Any other databases? Also, provide
the output of vmtune and schedtune. The group may be able to tell what's
going on performance wise and make some recommedations.
--joseph
-----Original Message-----
From: IBM AIX Discussion List [mailto:aix-l@Princeton.EDU] On Behalf Of
Wilson, Jeff
Sent: Wednesday, January 14, 2004 8:27 AM
To: aix-l@Princeton.EDU
Subject: Paging space on a fastT
I was wondering if anyone has had any experience putting paging space on
attached disks like a fastT 900. I am out of space on my internal OS disks
and need more paging space and was wondering if there are any problems I
should be aware of before I do this. Thanks for you help.
Jeff Wilson
System Administrator
Great-West Life & Annuity Ins. Co.
Desk: (303) 737-5399
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 22:17:30 EDT