iostat is ok from the shell but I want to be able to pull stats over snmp. iostat shows disk stats that have accumulated since the system was booted. You get averages by setting a report interval; iostat will generate a new report every <interval> seconds. Each additional report only covers the time since the last report. When I want to see the stats, I want to see them right away - not have to wait for five minutes to see a report showing averages for the last 5 minutes. The reports from iostat need to be stored somewhere but redirecting to a file will make for a very large file sooner or later. There's also no clean way to grab only the relevant lines from the bottom of the file if it contains all reports that have been produced so far.

iostat has problems with device mapper names on some systems. E.g. a KVM virtual machine that uses virtio: the virtblk device takes the device id 253, which iostat expects to belong to device-mapper.

What's needed is a daemon that can keep iostat running in the background and save the latest report to a file.

This script will run iostat and have it report statistics every 60 seconds (or what ever you specify). The file /var/lib/iostat-poller/stats contains statistics from the most recent report. The report has been filtered a little to make it easier for other programs to digest.

iostat-poller:

#!/bin/bash

# Copyright Kieran Whitbread 2010
# This file is released under the following license: 
# http://opensource.org/licenses/lgpl-2.1.php GNU Lesser General Public License


DEFAULT_INTERVAL=60
DEV_ID_MAJOR=$(grep "device-mapper" /proc/devices | cut -d\  -f 1)
PID_FILE=
STAT_FILE=/var/lib/iostat-poller/stats
TMP_FILE="/tmp/$(basename "$0").$$"

# call this with a partition device name as the argument (e.g. hda1, dm-4, sdb2)
# converts device name for Device Mapper devices to their friendly name
# calling iostats -N only sets the device mapper name if device-mapper major id is 253
# (check in /proc/devices)
function _device_map() {
        DEV_ID_MINOR="${1##dm-}"
        if [ "$1" = "$DEV_ID_MINOR" ]
        then
                # Not a device mapper name, so just echo it back and exit
                echo "$1"
                return 
        fi 
        echo $(ls -lQ "/dev/mapper" | egrep "^b.*${DEV_ID_MAJOR}, +${DEV_ID_MINOR}" | sed -e 's/^[^"]*"\(.*\)"$/\1/')
}
function _trap() {
	if [ "$PID_FILE" ]
	then 
		rm "$PID_FILE" 2>&1 > /dev/null
	fi
	rm -f "${TMP_FILE}" 2>&1 > /dev/null
	exit
}


trap _trap TERM INT HUP EXIT KILL

if [ ! -w "$STAT_FILE" ]
then
	echo "Cannot write to $STAT_FILE" 1>&2 
	exit 1
fi

case "$1" in
	--pidfile=*)
		PID_FILE="${1#--pidfile=}"
		shift
		echo $$ > "$PID_FILE"
	;;
	[0-9])
		INTERVAL=$1
	;;
esac

touch "$TMP_FILE"

exec <&-
exec 2> /dev/null
exec > /dev/null

if [ -z "$INTERVAL" ]
then
	INTERVAL=$DEFAULT_INTERVAL
fi

iostat -dmx $INTERVAL | while read LINE
do
	if [ "${LINE:0:8}" = "Device: " ]
	then
		continue
	fi
	if [ "$LINE" ]
	then
		DEV=$(echo "${LINE}" | cut -d\  -f 1)
		DEV_NAME=$(_device_map "${DEV}")
		LINE=$(echo "${LINE}" | sed -e "s/^${DEV} /${DEV_NAME} /")
		STATS="${STATS}\n${LINE}"
	else
		# iostat out put for each device can span more than one line.
		echo -e "${STATS}" | sed -e 'N;s/\(^[^ ]*\)\(\n\)\(.*\)$/\1\3/'  | sed -e 's/  */ /g' > "${TMP_FILE}"
		STATS=
		mv "${TMP_FILE}" "${STAT_FILE}"
	fi
done

The script wants to run in the background like a daemon but it's written in bash - you have to help it out by sending it to the background when you start it. It disconnects from STDIN so you don't have to worry about it hanging up when you close your terminal.

To start the script like this (but see below, you probably want to run this with a specific system user account):

# /usr/local/sbin/iostat-poller &

Here's a little script to help view the stats collected with iostat-poller.sh iostat-devices.sh:

#!/bin/bash

STAT_FILE=/var/lib/iostat-poller/stats

if [ -f "$STAT_FILE" ]
then
	cat "$STAT_FILE"
fi

You'll need to make the var directory to store the stats in

# mkdir /var/lib/iostat-poller

and a user and init script so you can run it as a daemon

 # useradd -r -d /var/lib/iostat-poller -s /bin/false iostat
# chown iostat /var/lib/iostat-poller

/etc/init.d/iostat-poller:

#! /bin/bash
#
# iostat-poller      Start/Stop the poller script
#
# chkconfig: 345 71 40
# description: iostat-poller is a script that runs iostat continously and 
#              filters then stores the most recent report
# processname: iostat-poller

# Source function library.
. /etc/init.d/functions
path="/usr/local/sbin/iostat-poller"
prog="$(basename ${path})"
niceness=+5
user=iostat
pidfile="/var/lib/iostat-poller/iostat-poller.pid"
RETVAL=0

start() {
	echo -n $"Starting $prog: "
	pid=$(cat $pidfile 2>/dev/null)
	if [ -n "$pid" ]
	then 
		failure $"$prog startup"
		echo
		return 1
	fi
	daemon --pidfile=$pidfile --user=$user $niceness "$path --pidfile=$pidfile &"
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
	return $RETVAL
}

stop() {
	echo -n $"Stopping $prog: "
	pid=$(cat $pidfile 2>/dev/null)
	if checkpid $pid
	then
		CHILDREN="$(ps --ppid $pid h o pid 2>/dev/null)"
		kill -KILL $CHILDREN > /dev/null 2>&1
		rm -f /var/lock/subsys/$prog
		checkpid $pid && success $"$prog shutdown" || failure $"$prog shutdown"
		RETVAL=$?
	else
		failure $"$prog shutdown"
		RETVAL=1
	fi
	echo
	return $RETVAL
}	
restart() {
  	stop
	start
}	

case "$1" in
  start)
  	start
	;;
  stop)
  	stop
	;;
  restart)
  	restart
	;;
  status)
	status -p $pidfile $path
	;;
  condrestart)
  	[ -f /var/lock/subsys/$prog ] && restart || :
	;;
  *)
	echo $"Usage: $0 {start|stop|status|restart|condrestart}"
	exit 1
esac

exit $?

To enable the service:

# chkconfig --add iostat-poller
# service start iostat-poller