«
»

I like zabbix… but it annoys me that it’s ops per second, and bytes per second data is broken for hard drives in the linux 2.6 kernel. So I created a work around :)

Add the following code to your zabbix_agentd.conf file (/etc/zabbix/zabbix_agentd.conf by default), and restart the zabbix agent:

UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

The names are pretty descriptive, but you can find more information on how to decipher /proc/diskstats here: http://www.kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/iostats.txt

After you add the UserParameter’s to your config file, you can check that they work by using zabbix_get from your zabbix server:

zabbix_get -s HostName -p 10050 -k custom.vfs.dev.write.ops[sdb]

It should return a large number, and not something like: ZBX_NOTSUPPORTED. If it comes back not supported, make sure you restarted the agent after adding the userparameters. Then try running the commands outside of zabbix. Note that if you run the commands outsie of zabbix, you’ll need to replace the first $1 with a disk name, and remove one of the $ characters from the awk command.

Once you’ve verfied everything is in working order, you can add it as an item to your zabbix installation:

You can download my exported Template here , which will create a template called Template_app_FileServer. It will give you:

Disk Read/Write: Bytes/sec (or KB, MB per second)

Disk Read/Write: Ops/second

Disk Read/Write: Milliseconds spent reading or writing

Disk:IO currently executing

Disk:IO ms: time spent performing IO

If you’d rather just create your own items, and not bother loading mine, I have attached some screenshots of the config screens, so you can re create it with the right settings:

Bytes/sec chart

Bytes/Sec config

Operations/sec config

active IO config

As always, if you find this page helpful or have any questions, please let me know! :)

This entry was posted on 3 October 2007 at 11:33 pm and is filed under Tech, Zabbix. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

26 Comments

  1. trentster says:

    Thanks for this, its very helpful, and I will use it on many agents. I also enjoyed the Centos 5 article. Keep up the great work, looking forward to many new excellent Zabbix articles.

    12 October 2007, 3:45 pm

  2. colin says:

    Agreed, this article was very helpful. Thanks for documenting this !

    12 November 2007, 1:28 pm

  3. Peter says:

    Hey,

    realy nice howto/workarround!

    For those who get
    awk: 1: unexpected character 0xe2
    awk: line 2: missing } near end of file
    awk: 1: unexpected character 0xe2
    awk: line 2: missing } near end of file

    in /var/log/zabbix-agent/zabbix_agentd.log

    Use
    UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
    UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
    UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
    UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
    UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
    UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
    UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
    UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

    With ‘ instead of ’.

    greets from Germany,

    Peter

    6 January 2008, 2:23 pm

  4. Steve Ley says:

    Fantastic information and template. Thanks for sharing!

    19 March 2008, 9:34 am

  5. Matthias says:

    Nice workaround! Any Ideas for FreeBSD Users??

    7 May 2008, 7:55 am

  6. Stelios says:

    I’m always getting a ZBX_NOTSUPPORTED even though I’ve double checked everything. Any ideas?

    5 June 2008, 8:44 am

  7. Mark says:

    Stelios, make sure your shell script returns some text instead of just a return code.

    You can test with something like zabbix_agentd -t vm.memory.size[free]

    Substitute with your custom test name, of course.

    9 June 2008, 7:24 am

  8. Danie says:

    This will be fantastic is I can get it to work though.
    The error I get when creating the templates is:
    Type of received value [8 17 sdb1 654553 24426018 1870561 14964480] is not sutable for [custom.vfs.dev.read.ops[sdb1]@
    Can anyone perhaps help me with this?
    The templates are 100% created as above.

    3 September 2008, 4:01 am

  9. Koos says:

    Danie, I ran into the same problem.
    Just make sure that you have 2 x dollar signs in your awk statements.
    Otherwise the agent thinks the Zabbix server sent the variables.

    Thanks for the article, saved me loads of time.

    7 November 2008, 12:07 pm

  10. psd says:

    Hi,

    I am using your system above, when I fill up a filesystem on sda, using dd, the graph does not seem to move to show that the number of writes has increased on the system? any ideas pls

    3 December 2008, 6:24 am

  11. катринна says:

    Привет всем!!!
    Зря ты так

    2 February 2009, 4:44 am

  12. Monitoring PostgreSQL TPS with Zabbix | gtuhl: startup technology says:

    [...] a ton of stuff including tracking CPU utilization and disk IO (awesome IO monitoring thanks to this page) on all servers, detecting outtages and automatically fixing them, monitoring and charting the [...]

    21 February 2009, 6:20 am

  13. Кирилл Александров says:

    Даа… Какой кошмар!

    9 April 2009, 9:07 pm

  14. ldunston says:

    Awesome contribution. I had to change the single quotes around the awk print statements to get it to work.

    Also take a peek at /proc/diskstats to make sure you’re looking at the right disk. I’m using lvm on vmware so dm-0 and dm-1 were my target disks.

    23 April 2009, 1:15 pm

  15. brian says:

    Was digging around for info on IO stats on Linux using Zabbix and stumbled across this entry. Even though I’m late to the game (this post is kind of old ;) ), this worked great for me. Thanks for taking the time to blog on this.

    24 September 2009, 7:32 am

  16. Andrea says:

    great post!

    I currently use zabbix to monitor a really large environment (more than 320 servers)

    I’ve founded a wonderful plugin that is more than a plugin and the others monitoring systems don’t have nothing of similar, and nothing that go inside oracle so deeply.

    In the hope that someone found useful my comment

    http://www.smartmarmot.com

    here you are going to find Orabbix opensource and released under GPL3

    2 June 2010, 5:54 am

  17. Rubén Ortiz says:

    Hi

    I have test the command out of Zabbix and I have notices that, at least, in my Centos 5.5 x_64 box, I have to change `and´ for ” in commands to obtain numbers, exemple:

    cat /proc/diskstats |grep sda |head -1|awk ‘{print $4}’
    116817480

    Are you agree with this change?

    thanks

    19 November 2010, 3:59 am

  18. Hirsty says:

    http://blog.thepoint.com/2008/06/30/today-in-eaction-news-063008/

    29 November 2010, 7:08 am

  19. malex says:

    Here is a equivalent command implemented with a single awk call:

    awk ‘$3 == “sda” { print $4; exit }’ /proc/diskstats

    1 December 2010, 8:57 am

  20. Paul C says:

    using – UserParameter=custom.vfs.dev.io.ms

    you can work out %Util by taking this field and setting the item for delta/sec then multiply by .1

    theory is if the diskstats shows it spends 1000ms doing IO during a second ( 1000ms ) then it’s @ 100%.

    this gives the same value that iostat gives for %Util, so I think my logic is sound.

    6 January 2011, 10:25 am

  21. jar says:

    Thanks a lot man. It really helped me. I changed the template to use sda and sdd, but apart of that, perfect!

    2 March 2011, 4:50 am

  22. Zach Bailey says:

    This is extremely helpful for setting up disk i/o monitoring! Thanks!

    The only problem I had like some of the others was that it appears your blog converted regular ASCII quotes ‘ to smart quotes ‘’ which can throw a snag into things if you copy/paste right into the zabbix config without testing first.

    13 April 2011, 8:07 pm

  23. BludGeonT says:

    Years after this post it still applies even to the newest version of Zabbix. It is working beautifully and thank you again for the easy solution of getting the graphing to work.

    Zabbix version: 1.8.5
    OS Ubuntu 11.4

    8 August 2011, 4:20 pm

  24. Ankit says:

    Hello sir,
    I’m still facing error (Item [{HOSTNAME}:custom.vfs.dev.read.sectors[sdc]] does not exists) when import this zabbix_FileServer_template.xml. :(

    25 November 2011, 10:54 pm

  25. Ankit says:

    Thanks :) finally its done and giving an outstanding performance. Thanks a lot for sharing.

    26 November 2011, 2:14 am

  26. Elvar says:

    What about for Ubuntu 11.10 systems that use cciss such as below…

    Filesystem Size Used Avail Use% Mounted on
    /dev/cciss/c0d0p3 908G 3.4G 859G 1% /
    udev 5.9G 4.0K 5.9G 1% /dev
    tmpfs 2.4G 312K 2.4G 1% /run
    none 5.0M 0 5.0M 0% /run/lock
    none 5.9G 4.0K 5.9G 1% /run/shm
    /dev/cciss/c0d0p1 1.9G 61M 1.7G 4% /boot

    Regards,

    9 December 2011, 8:52 am

Leave a Reply