From: JESSE CARROLL (jesse-carroll@usa.net)
Date: Tue Aug 07 2007 - 17:01:07 EDT
(original question below)
To date I've received excellent suggestions from William Hathaway, Ric
Anderson and Mark Round. Mark was correct in that the interrupts where from
one of the NIC's. Ric had an excellent suggestion to use 'prstat -P <cpu>',
which showed which processes where running on a particular CPU. This showed
that the OS was rotating the processes around. William Hathaway takes the
prize though, with his suggestion to use 'intrstat'. This definitely was one
of the NIC's. My guess is that the NIC's somehow are tied to a cpu/core for
interrupt handling as 'intrstat' showed that a particular NIC was
consistently
responsible for the high(er) rates of interrupts for a particular CPU.
As Michael Hase pointed out, the system was fairly idle during the sample
period. However, earlier today the system was experiencing a weird high rate
swapped out light-weight processes, though the system didn't appear to be
under a significant load. Still looking for that, but its a hit-or-miss
thing.
JC
------ Original Message ------
Received: Tue, 07 Aug 2007 01:06:05 PM EDT
From: "JESSE CARROLL" <jesse-carroll@usa.net>
To: <sunmanagers@sunmanagers.org>
Subject: mpstat on T2000
While looking at a possible performance issue on a T2000 I noticed that cpu
27
was experiencing a much higher rate of interrupts and interrupts on threads
than the other 31 cores. (The mpstat output is below) I looked at a couple of
other T2000's and saw very similar behavior. I can't seem to find a
particular
process consistently running on cpu 27 and all system run different
applications. Is cpu 27 somehow used differently (e.g. floating point or
specific interrupt handling) than the other ones?
mpstat output
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 22 0 995 314 140 143 1 1 1 0 306 9 1 0 90
1 16 0 760 50 0 95 1 1 1 0 222 7 1 0 92
2 10 0 620 34 0 65 0 1 1 0 130 4 0 0 95
3 8 0 575 31 0 59 0 1 1 0 115 3 0 0 97
4 13 0 687 44 0 86 0 1 1 0 171 5 1 0 95
5 13 0 654 44 0 85 0 1 1 0 164 5 1 0 95
6 12 0 630 45 1 84 0 1 1 0 161 4 1 0 95
7 9 0 595 41 0 80 0 1 1 0 146 3 0 0 97
8 14 0 667 44 0 85 0 1 1 0 168 5 1 0 95
9 11 0 618 41 0 80 0 1 1 0 151 3 0 0 96
10 12 0 616 43 0 83 0 1 1 0 157 4 0 0 95
11 12 0 612 43 0 84 0 1 1 0 160 5 0 0 95
12 12 0 609 42 0 83 0 1 1 0 157 3 1 0 96
13 12 0 651 44 0 86 0 1 1 0 162 4 1 0 95
14 12 0 634 44 0 86 0 1 1 0 163 5 1 0 95
15 11 0 606 42 0 83 0 1 1 0 152 4 0 0 96
16 13 0 661 43 0 83 0 1 1 0 166 4 1 0 95
17 12 0 623 43 0 83 0 1 1 0 160 5 0 0 95
18 12 0 602 42 0 82 0 1 1 0 155 4 0 0 95
19 11 0 582 42 0 82 0 1 1 0 154 4 0 0 95
20 11 0 652 43 0 84 0 1 1 0 156 3 1 0 96
21 13 0 650 45 0 87 0 1 1 0 161 5 1 0 95
22 13 0 629 45 0 87 0 1 1 0 160 5 1 0 95
23 9 0 575 41 0 81 0 1 1 0 145 2 0 0 97
24 13 0 647 43 0 84 0 1 1 0 162 4 1 0 95
25 12 0 611 42 0 83 0 1 1 0 155 4 1 0 95
26 12 0 599 42 0 82 0 1 1 0 154 5 1 0 95
27 10 0 1064 578 535 83 0 1 3 0 155 3 1 0 95
28 11 0 662 50 8 81 0 1 1 0 157 3 1 0 96
29 12 0 632 43 0 83 0 1 1 0 158 5 1 0 95
30 12 0 606 48 6 83 0 1 1 0 157 4 1 0 95
31 12 0 616 43 0 84 0 1 1 0 160 5 1 0 95
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 1 0 207 310 149 112 1 2 0 0 147 4 1 0 95
1 43 0 376 33 0 71 0 2 1 0 94 1 3 0 96
2 38 0 111 33 0 67 0 2 0 0 285 2 0 0 98
3 15 0 96 29 0 61 0 2 0 0 38 0 0 0 99
4 1 0 49 20 0 44 0 1 1 0 15 0 0 0 100
5 0 0 11 21 0 44 0 1 0 0 11 0 0 0 100
6 0 0 18 15 2 33 0 1 0 0 58 0 0 0 100
7 1 0 60 22 0 45 0 2 1 0 189 0 0 0 99
8 68 0 170 73 0 151 1 2 3 0 503 4 0 0 96
9 27 0 104 13 0 25 0 1 0 0 64 1 0 0 99
10 26 0 87 21 0 44 0 1 0 0 264 1 0 0 99
11 0 0 6 10 0 24 0 1 0 0 0 0 0 0 100
12 16 0 40 40 0 80 0 2 1 0 17 0 0 0 100
13 0 0 44 40 0 84 0 2 1 0 18 0 0 0 100
14 15 0 52 14 0 29 0 2 1 0 115 0 0 0 100
15 43 0 122 31 0 64 0 1 1 0 301 2 0 0 98
16 43 0 90 17 0 35 0 2 0 0 301 2 0 0 98
17 0 0 3 11 0 22 0 2 1 0 7 0 0 0 100
18 0 0 18 28 0 58 0 1 0 0 2 0 0 0 100
19 15 0 11252 16 0 33 1 1 3 0 816 2 4 0 94
20 42 0 56 20 0 69 0 1 2 0 261 0 1 0 99
21 0 0 61 34 0 67 0 1 0 0 12 0 0 0 100
22 16 0 11 21 0 40 0 1 0 0 58 0 0 0 100
23 0 0 5 10 0 20 0 1 1 0 1 0 0 0 100
24 0 0 9 84 0 168 1 2 2 0 152 8 1 0 92
25 0 0 13 14 0 30 0 2 0 0 93 0 0 0 100
26 1 0 53 66 0 139 0 2 4 0 20 1 0 0 99
27 0 0 332 497 471 54 0 2 8 0 2 0 1 0 99
28 42 0 85 18 2 30 0 1 0 0 265 2 1 0 98
29 0 0 1 4 0 8 0 1 0 0 13 0 0 0 100
30 1 0 1 10 5 8 0 1 0 0 56 0 0 0 100
31 0 0 36 80 0 153 4 1 2 0 206 64 1 0 36
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:42:11 EDT