Wednesday, July 30, 2008

Solaris sar %sys cpu high

How to find out which pid is consuming %usr kernel cpu resources
#prstat -m
eg.
$ prstat -m
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
3828 hpfimhas 24 75 1.1 0.0 0.0 0.0 0.0 0.0 0 0 87K 0 prstat/1
3812 root 0.1 0.1 0.0 0.0 0.0 0.0 100 0.0 9 0 224 0 sshd/1
3820 hpfimhas 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 38 0 128 0 sshd/1
3822 hpfimhas 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 110 0 sh/1
6105 instbas 0.0 0.0 0.0 0.0 0.0 73 27 0.0 10 0 14 0 java/48
6006 lmcbejo 0.0 0.0 0.0 0.0 0.0 87 13 0.0 23 1 16 0 java/30
13769 instbas 0.0 0.0 0.0 0.0 0.0 91 9.4 0.0 4 0 9 0 java/128
5974 instbas 0.0 0.0 0.0 0.0 0.0 84 16 0.0 16 0 10 0 java/57

For solaris 10 can use dtrace, see example below

On with the show: let's say you're looking at mpstat(1) output on your
multiuser server. You might see something like this:

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
12 1 27 3504 338 206 765 27 114 65 1 337 9 19 41 31
13 1 19 5725 98 68 723 22 108 120 0 692 3 17 20 61
14 0 57 3873 224 192 670 22 75 86 1 805 7 10 35 48
15 36 7 1551 42 28 689 6 68 59 0 132 2 9 35 54
16 14 7 7209 504 457 1031 37 125 244 0 459 6 30 4 60
17 5 5 4960 150 108 817 37 98 154 0 375 6 26 6 62
18 5 6 6085 1687 1661 741 60 76 248 0 434 3 33 0 64
19 0 15 10037 72 41 876 23 100 291 1 454 2 19 9 71
20 12 5 5890 746 711 992 32 122 216 2 960 10 33 4 53
21 60 5 1567 729 713 467 15 80 59 0 376 2 35 10 53
22 0 6 4378 315 291 751 17 84 142 1 312 3 16 1 80
23 0 6 12119 33 3 874 20 82 384 1 513 4 24 11 62

And well you may wonder (as perhaps you often have) -- what the hell is
causing all of those cross calls, anyway? (Cross calls appear in the "xcal"
column; see mpstat(1).)

Using DTrace, investigating this is a snap:

# dtrace -n xcalls'{@[execname] = count()}'
dtrace: description 'xcalls' matched 4 probes
[ letting this run for a few seconds ]
^C

mozilla-bin 1
lockd 1
in.mpathd 2
nsrmmd 5
grep 6
chmod 6
cat 6
nwadmin 13
ls 24
in.tftpd 28
nsrindexd 34
fsflush 38
cut 42
find 42
mkdir 66
rm 76
ipop3d 78
scp 79
inetd 96
dtrace 111
nawk 118
imapd-simmonmt 126
rmdir 132
sshd 138
rpc.rstatd 159
mv 398
save 1292
gzip 1315
get_all2 1678
sched 1712
nfsd 3709
tar 27054 <----- high usage