pio (Process I/O)

I originally wrote this program for Solaris when I realized top (or prstat) couldn't sort on I/O (history). Now I have it for other OS'es. For Solaris, read on until the end of purple background. For Windows, click here. For HP-UX, you can jump here but I recommend you glance through the Solaris section first. For Linux, click here.

SOLARIS

pio (for Process I/O) on Solaris is a program that shows how much character I/O a process has read or written. topio can be used to show the most I/O-intensive processes. This is what top or prstat command cannot do. Jump to the most useful command if you're impatient.

For instance, the following command shows that your current shell has read and written 10994 characters so far (we'll talk about other columns later).

$ pio -p $$ PID InpBlk OutpBlk RWChar MjPgFlt Comm 392 0 33 10994 0 -ksh

You can look at a process continuously (-H removes header)

$ pio -p 554 PID InpBlk OutpBlk RWChar MjPgFlt Comm 554 991 0 116099 922 find / $ while true; do > pio -Hp 554 > sleep 1 > done 554 2266 0 243623 2095 find / 554 2339 0 251878 2166 find / 554 2408 0 258030 2229 find / ^C$

or look at all processes (note that pio doesn't need SUID permission to look at other users' processes, and -A prevents header from being printed)

$ pio -A #columns are pid, InpBlk, OutpBlk, RWChar, MjPgFlt, Comm 0 280 1 0 93 sched 0 280 1 0 93 sched 0 280 1 0 93 sched 1 125 1 68121 92 /etc/init - 2 0 0 0 0 pageout 3 0 210 0 0 fsflush 342 12 4 8691 11 /usr/lib/saf/sac -t 300 ...

The next useful thing to do is write a program to sort the RWChar column. I wrote a Perl script specifically for this purpose, appropriately named topio.

$ topio ** WARNING: Running topio without -d may not be ** ** what you want. Type topio -h for help. ** PID InpBlk OutpBlk RWChar MjPgFlt Command 338 530 4 2394164 408 /usr/openwin/bin/Xsun :0 -nobanner -defdepth 24 -auth /var/dt/A:0-oHaGLa 368 96 5 2027102 86 /usr/lib/ssh/sshd 363 188 0 834232 163 dtgreet -display :0 348 196 0 679939 147 /usr/sfw/sbin/snmpd 388 0 4 291163 0 /usr/lib/ssh/sshd 214 7 0 128703 7 /usr/sbin/inetd -s 350 50 1 84051 40 /usr/dt/bin/dtlogin -daemon 1 125 1 68121 92 /etc/init - 247 0 3 57536 0 /usr/lib/utmpd 314 8 0 34965 4 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf ^C$

Probably the most useful of pio and topio is the -d option of topio, which sorts based on the delta or difference of the process Read/Write Characters between two consecutive runs. (While the examples above are run on my laptop, the screen shot below is captured on a server so the numbers differ.)

$ topio -d -s2 -n5 #display 5 top Delta-I/O processes every 2 seconds --PID-------RWChar-----DltRWC-----MjPgFlt-DltMPF Command------------------------ 8872 64025286 1835008 7 0 ora_dbw0_ORATEST 5945 289675626 56832 62 0 ora_lgwr_ORATRN 5947 497918441 49152 324 0 ora_ckpt_ORATRN 5773 3917910706 28672 47 0 ora_lgwr_INTTST 5943 3609392512 16384 1 0 ora_dbw0_ORATRN --PID-------RWChar-----DltRWC-----MjPgFlt-DltMPF Command------------------------ 8874 3130856681 2108416 112 0 ora_lgwr_ORATEST 18528 11064724 831589 0 0 oracleORATEST 5945 289729898 54272 62 0 ora_lgwr_ORATRN 5775 1537267122 49152 223 0 ora_ckpt_INTTST 8876 302708422 16384 241 0 ora_ckpt_ORATEST --PID-------RWChar-----DltRWC-----MjPgFlt-DltMPF Command------------------------ 8872 68752070 4726784 7 0 ora_dbw0_ORATEST 8874 3132257001 1400320 112 0 ora_lgwr_ORATEST 18528 11361185 296461 0 0 oracleORATEST 18526 48811015 178640 113 0 oracleORATEST 5775 1537381810 114688 223 0 ora_ckpt_INTTST ^C$

Process 5773 has the highest absolute I/O's under RWChar column according to topio output (without -d, not shown here). But its delta I/O, difference of absolute I/O's between two consecutive runs, only shows up near the top occasionally. This process happens to be an Oracle background process LGWR which writes to the redo logfiles of INTTST database. This LGWR process at the time we captured wrote 28672 bytes to logfiles in a 2 second period. (LGWR does not read, unless the database is being recovered from crash.) If you're only checking Oracle processes' I/O, you may want to supplement this information with that offered by Oracle's tools such as the statistics collected in Oracle v$sess_io view. (Unfortunately v$sess_io doesn't record physical writes.)

Download source code pio.c and type gcc -o pio pio.c. Also download topio and read the line below #!. Put pio and topio in the same directory and chmod to make executable. If you wish to run topio from directories other than where they are, change $PIO in topio to the absolute path. The current version additionally probes the process major page fault in the hope that true disk I/O excluding page cache I/O can be deduced. Note for x86 Solaris users: gcc 3 has problems with some headers. Use gcc 2.95 instead, unless you want to fix the header files.

How does it work? Before Solaris 10,^note1 there're two ways to get the I/O count of a process on Solaris. Brendan Gregg's psio Perl program uses the prex utility to probe into kernel and filter on a specific process. My pio, originally written by looking at Jim Mauro and Richard McDougall's msacct^note2 published in Appendix C of Solaris Internals, fetches the I/O count from /proc filesystem. (I'm not using microstate accounting as in Jim's program, which is essential in CPU costing but would pose some performance overhead.) Basically, pio gets process I/O statistics from /proc/pid/usage, specifically the fields pr_inblock, pr_oublock and pr_ioch of struct prusage, as explained on pp.314-5 of Solaris Internals and proc(4) man page. You may wonder how much precious information is collected by our UNIX box without ever being used! That's right. If you don't write programs like this to fetch the data, they're collected and simply thrown away.

What the numbers mean pr_inblock and pr_oublock are generally not very useful. According to Adrian Cockcroft, "inblock and outblock [sic] counters are uninteresting as they only refer to filesystem metadata for the old-style buffer cache". Indeed, beginning with Solaris 2, the old buffer cache is largely replaced by page cache and is only used to store metadata. So if you see occasional number jump in InpBlk and OutpBlk, it is, for instance, because the allocated file blocks needs to be extended/shrunk to accommodate more/less data, so the inode is updated. What I observed is, when a process continuously does I/O, RWChar keeps increasing. InpBlk and OutpBlk remain the same for some time and suddenly jump, remain the same for a while again and jump again. But the ratio of this jump in blocks to the number of characters incremented in RWChar is not consistent for each file. That's why the 2nd and 3rd columns of pio output don't look important to me.

The statistic pr_ioch or Read/Write Characters lumps reads and writes together and there's no way to separate them. The only workaround I can think of is something like

#trace read/write syscalls, redirect stderr (which truss outputs to) to Perl filter, #which prints syscall return value, i.e. number of chars read/written truss -t read,pread -p pid 2>&1 | perl -nle '/= (\d+)$/; print $1' truss -t write,pwrite -p pid 2>&1 | perl -nle '/= (\d+)$/; print $1'
You can do some math there to sum up the return values for a period of time. If scatter-gather I/O is used, you may want to add readv and writev to -t.

Another problem with pio is that RWChar includes all kinds of I/O, i.e. disk I/O as well as terminal and network I/O. If somebody has left the top program running for a long time (because he doesn't know the lighter-weight prstat!), topio may show this top process has accumulated a lot of RWChar, and possibly a lot of delta I/O in topio -d output, particularly if top was launched with a short interval (like top -s1). You can test this problem of pio and topio with a tight loop of echo "some characters" without a sleep in the loop. The current version of my program incorporates major page fault statistic in order to hopefully uniquely identify real disk I/O. Fortunately people often use topio to monitor daemon processes including Oracle server processes. So terminal I/O is completely off. But disk I/O and network I/O (if any) are still mixed.

____________________
^note1 Solaris 10 has the powerful DTrace facility which can be used to provide process I/O statistics.

^note2 Jim Mauro's msacct uses printf("%ld".. for process usage. I changed it to printf("%lu".. in pio.c. Otherwise numbers greater than 2 billion would show as negative. They're defined as unsigned long anyway.

HP-UX

Assuming you have quickly read the Solaris section, I only highlight a few points here. pio on HP-UX tells you how many read and write operations a process has performed. topio sorts all processes by either reads or writes.

$ pio -p $$
PID     InpOps  OutpOps MjPgFlt Comm
25240   8       16      0       sh

$ topio -n3 -s2 -kW #display 3 top Delta-Write processes every 2 seconds
--PID ProcName--------- -----Reads ---DltR -----Writs ---DltW -----PFlts ---DltF
   52 vxfsd                    225       0     322383       6          0       0
 1700 midaemon                   0       0          0       0          0       0
13730 ia64_corehw                0       0          0       0          0       0
--PID ProcName--------- -----Reads ---DltR -----Writs ---DltW -----PFlts ---DltF
   52 vxfsd                    225       0     322388       5          0       0
 1429 java                     191       0       8080       1        794       0
 1700 midaemon                   0       0          0       0          0       0

While the Solaris version lumps read and write characters together, the HP-UX version separately counts input and output, and it counts read and write operations, not number of characters. In addition, the HP-UX version no longer needs -d to sort on deltas.

Download source code pio.c and type cc -D_PSTAT64 -o pio pio.c. Also download topio and read the line below #!. Put pio and topio in the same directory and chmod to make them executable. If you wish to run topio from directories other than where they are, change $PIO in topio to the absolute path. [Dec 2008, Alexander Beyn comments "on HPUX 11.00, I had to #define _RUSAGE_EXTENDED before sys/pstat.h was included, otherwise pst_inblock and pst_oublock were not part of the pst_status structure...It looks like HP-UX 11.11 (released in 2000) and newer expose those fields without _RUSAGE_EXTENDED."]

How does it work? pio fetches I/O statistics from pstat, specifically pst_inblock and pst_oublock fields of struct pst_status. You can see these fields in /usr/include/sys/pstat/pm_pstat_body.h (thanks to Don Morris and Christof Meerwald on the newsgroup). Note that judging by the names, you would think they represent number of input and output blocks, just like pr_inblock and pr_oublock on Solaris. But the header file comment says they are block input and output operations.

Linux

Historically, Linux didn't have process I/O usage recorded either in the proc filesystem or by the getrusage call (getrusage has the fields for the current process but they're never populated). On those older Linux boxes, if we want process I/O count, we can have a kernel module to catch read(2) and write(2) syscalls or their variants; I think this is exactly what AT Consultancy's atop has been doing all these years. Alternatively, we can enable block I/O debugging as done by iodump. Also see this thread. Finally, SystemTap's uid-iotop could be used as well. However, in my opinion, all those have been outdated by kernel 2.6.18-164's introduction of /proc/pid/io.

Assume you have /proc/pid/io:
Download pio and topio into the same directory, type chmod to make them executable. If you wish to run topio from a directory other than where they are now, change $PIO in topio to the absolute path. Run ./topio -h first. A screen shot of actual use is shown below. (You may have to run as root now because from some version of Linux on, /proc/*/io is no longer other-readable.)

$ ./topio -s3 -krb -n4 #display top 4 delta-read-bytes (DltRBts) processes every 3 seconds
--PID ProcName----- -----RdChars DltRChs ------WtChrs DltWChs -------Rds -DltRds ------Wts DltW ------RdBytes DltRBts -----WtByts DtWB ---CWBts DltCWB
22537 ora_lmon_orac    732113332     560       206773       0    8914128       7      7935    0  144191709184  114688      413696    0        0      0
22555 ora_ckpt_orac   1317082923    1482        95358       0    8779307      10       123    0   36745311232   32768       32768    0        0      0
11629 /u01/crs/orac  16103068023   17851   8661523718    8092   50064836      59  30159419   34   14550792704   10752  7267489280 4608 52371046   8192
12599 /u01/crs/orac  79179130109  743582  57139477524  561498   56850567     404  16021147  154    3476266496    4096 10401308672 4915 11809955   8192

See Linux documentation for detailed description of these counters I use, exposed in /proc/pid/io. Guillaume Chazarain already implemented iotop based on these IO counters. My age-old topio serves as an alternative tool.

The only other major OS I care about is AIX. According to this discussion. AIX has nmon that can sort on process I/O. Indeed it works as expected (press t then 5 to sort on I/O). They also have nmon for Linux. But on Linux, it actually sorts on delta page faults, not really I/O's.