MuckNet

The title of this post is about zabbix, because thats my NMS of choice, but this method would apply to just about any monitoring application out there. Cacti, Nagios, big brother, opennms, etc. It also is not limited to postfix. Sendmail would work with a minor modification (look for stat=Sent, instead of status=sent).

OK, so what is the problem? Sendmail is archaic. Postfix tries to be backwards compatible with archaic. Neither of them keep a running counter of total messages sent. So in order to keep an eye on how many mail messages are coming out of your system (or not coming out of your system as the case may unfortunately be), you have to come up with some hokey way of grabbing those stats from the log file. I spent an hour or so searching for someones already written script, thinking SOMEONE has to have done this. And while I found a lot of people asking the question, I didn’t find any tidy, wrapped up answers.

There are lots of solutions to comb your logs nightly and produce a graph or report, but I wanted real time data, and I wanted the data in my NMS, so I could do my own graphing and alerting. One thing I absolutely hate is having 18 applications capable of alerting me of various things. So I cobbled together this perl script, which will check the last X minutes of a logfile for a string, and return it to you. You can then pipe it through a ‘wc -l’ to count how many times it occured. You can put it in cron, and pipe it through mail, to send you a copy, whatever you want.

Requirements:

ReadBackwards Perl Module - You can install from source if your servers don’t have internet access (IMHO – production servers should never have outbound internet access, inbound only — I’ll save that lecture for another blog post. Maybe.). If you have internet access just run:

perl -MCPAN -e ‘install File::ReadBackwards’

/var/log/maillog permissions – (or whatever log file you’re looking at). Make sure the user you’re reading the log files with can read that log file by default your unprivileged zabbix user cannot read /var/log/maillog. You can fix this any number of ways. chmod. sudo. changing file ownership. I chose to change file ownership. this seems the cleanest, and I don’t view it as a security risk for my maillog to be readable by all users on the system. Before you ask – Yes, your logrotate process will continue to propogate those new permissions, because it gets its cue’s, from the file it is rotating

chmod a+r /var/log/maillog

The script itself is below, but I will also attach it to this post, because wordpress will surely screw the formatting or characters up.. so please download the attached perl script here, before complaining it won’t compile: count_pf_mail.pl.txt

#!/usr/bin/perl
use File::ReadBackwards;
use POSIX "strftime";

my $items;
my $time = $ARGV[0];
my $logfile = "/var/log/maillog";
my $search_string = "status=sent";
my $goback = POSIX::strftime( "%b %d %H:%M:%S", localtime( time() - $time * 60 ) );

$readback = File::ReadBackwards->new( $logfile ) or
                        die "unable to open '$logfile' $!" ;

while ( defined( my $line = $readback->readline() ) ) {

    last if $line lt $goback;

    if ($line =~ /$search_string/) {
        print $line;
    }

}

The only argument the script takes is number of minutes to go back in the log file.

[doug@app5 ~]# /usr/local/bin/count_pf_mail.pl 5

Mar 20 11:12:01 app5 postfix/smtp[31232]: D864A77EBA: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.27, delays=0.04/0/0.08/0.14, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 n20si3574089pof.6)
Mar 20 11:12:02 app5 postfix/smtp[31223]: DAD4B77EB3: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.22, delays=0.02/0.01/0.08/0.11, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 q18si3562811pog.12)

…

Is this totally accurate?

Absolutely not. Its close, but its not perfect. It assumes that your monitoring time period will be at EXACTLY the right interval. That won’t ever happen, so its possible that some mail will be counted twice, and some will not be counted at all. However, its good enough for comparison purposes, and its better than nothing In my ideal world, this script would feed into a database/file of already counted message ID’s, to make sure that the same message ID isn’t counted twice. But This works good enough for me

OK, so how do I make this work with zabbix?

zabbix_agentd.conf:

The only UserParameter you’re really worried about for this particular usage is the last one, but if you download my zabbix template below, you’ll want the other 2 also

UserParameter=mailq,/usr/bin/mailq | grep Request | awk '{print $5}'
UserParameter=established_conns[*], netstat -an | grep :$1 | grep ESTABLISHED |wc -l
UserParameter=count_pf_mail[*], /usr/local/bin/count_pf_mail.pl $1 |wc -l

Creating a zabbix item:

Choosing a longer polling period will make the results more accurate over time. I’ve been experimenting with 3 minutes, 5 minutes, etc. I wouldn’t suggest anything less than 3 minutes. The following example uses a 3 minute polling interval, and displays it as a “per minute” statistic. You could modify to be per second, per hour, per 5 minutes if you want. Its all just math. I’ve highlighted the parameters that need to be changed if you want to change the polling period.

Key: count_pf_mail[3]

Type of Information: Float

Multiplier (Use Custom): .333

Update Interval: 180

If you wanted to do 5 minute polling intervals, and still display it as a per minute statistic:

Key: count_pf_mail[5]

Type of Information: Float

Multiplier (Use Custom): .2

Update Interval: 300

Zabbix Graph showing the data from my (not yet active) postfix cluster:

You can also download my postfix template here: template_app_postfix.xml.txt

It will work with the mailq and established_conns Userparameter above. It is by no means completed or final, but hopefully you’ll find it useful anyway. Everything here could be modified pretty easily to work with sendmail also.

As always if you find this post helpful, let me know! Comments, suggestions, and corrections are always welcome!

Zabbix uses libcurl (libraries, not binaries) to do its Web Scenarios. Web scenarios are very powerful, and allow you to emulate a user experience. Using a Zabbix web scenario, you can emulate logging into your site, accepting the cookie, clicking on something unique (Report showing 10 Last purchases for example), then verify that you get either a particular HTTP code, or that certain text shows up in the response. Way cool stuff. Its got a few kinks to be worked out, however. One very frustrating one is that these web scenarios are not template aware yet… But the zabbix team is working on it, and its going to be a part of a future release. One minor, but significant thing for several of my environments is that the web scenario will error out if the SSL certificate CN (www.yoursite.com) does not match the URL you accessed the web server with.

But Doug, thats bad practice for the CN to not match the url!

I know! However, in most environments its not uncommon for the internal DNS name to NOT match the external DNS name. For example, the CN name for your SSL cert will be www.yoursite.com, but internally you have 10 app servers responding as www. You refer to them as app1.sjc.yoursite.com, app2.phx.yoursite.com.

By default curl (and therefore zabbix) will error out. With the following:

Failed on “HTTPS TEST app1.sjc.yoursite.com” [1 of 1] Error: SSL peer certificate was not ok

I’ve written a patch for the zabbix_server binary, which will instruct libcurl to not error out, and life is peachy! You need to unpack the zabbix source, apply the patch, recompile, and install the new binary. The patch, and steps are below:

I’ve attached the patch to this post, I’d suggest downloading it, instead of copying and pasting, but if you’d like to here it is:

--- src/zabbix_server/httppoller/httptest.c     2007-08-20 12:22:22.000000000 -0700
 +++ src/zabbix_server/httppoller/httptest.c.dp  2007-11-13 17:53:54.000000000 -0800
 @@ -318,6 +318,15 @@ static void        process_httptest(DB_HTTPTEST
 return;
 }

+       /* Process certs whose hostnames do not match the queried hostname. */
 +       if(CURLE_OK != (err = curl_easy_setopt(easyhandle,CURLOPT_SSL_VERIFYHOST , 0)))
 +       {
 +               zabbix_log(LOG_LEVEL_ERR, "Cannot set CURLOPT_SSL_VERIFYHOST [%s]",
 +                       curl_easy_strerror(err));
 +               (void)curl_easy_cleanup(easyhandle);
 +               return;
 +       }
 +
 lastfailedstep=0;
 httptest->time = 0;
 result = DBselect("select httpstepid,httptestid,no,name,url,timeout,posts,required,status_codes from httpstep where httptest
 id=" ZBX_FS_UI64 " order by no",

Link to Patch: libcurl disable ssl verifyhost

Instructions for installing patch:

Shut down zabbix_server process

/etc/init.d/zabbix_server stop

If you already have your zabbix source unpacked, you can skip the first tar step I’ve checked the patch with Zabbix 1.4.1 and 1.4.2

tar -zxvf zabbix-1.4.2.tar.gz

cd zabbix-1.4.2

wget http://www.muck.net/wp-content/uploads/2007/11/libcurl_disable_ssl_verifyhost.patch

patch src/zabbix_server/httppoller/httptest.c libcurl_ssl_verifyhost.patch

Then build zabbix_server as normal, for example:

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

Restart zabbix_server

/etc/init.d/zabbix_server start

Your Internal SSL Web Scenarios should now work! That was easy wasn’t it?

As always, I appreciate any feedback, and would love to hear if this helped you, or if you have any questions!

I’ve been saying I was going to do it for the last 6 months. I finally did it this weekend. I built a demo version of zabbix that I can give full admin rights to. If you’re not familiar with zabbix you can visit my zabbix page , or the zabbix website .

The problem with the old zabbix demo was that it was the only copy. If someone messed it up, I had to manually go in and restore the DB. It didn’t have a self recovery method. In the end, it meant that I was limited to handing out read only access in most cases. That in turn meant it wasn’t used very often by folks, and because it wasn’t used very often, I neglected it.

However! I have now put zabbix into its own VM universe. I took a snapshot of a “clean” zabbix installation, which is known to be good. Every 24 hours, I wipe out the VM instance, put a new one in place, restore the Zabbix DB, and timeshift all the data so it looks like the data is contiguous, even though in all likelyhood its several weeks old. The process results in about 10 minutes of downtime, and another 20 minutes of data processing in the background. Any data displayed from Midnight PST through Midnight PST the following night, is live, accurate, real data, not timeshifted data.

I’m pretty happy with it so far. I still want to add a lot more items and hosts to the demo, but I think its a great start. As far as I know this is the only Zabbix Demo, so I’d love to hear from you what you’d like to see in the demo. More graphs? Screens? Web Monitoring? oracle monitoring? mysql monitoring? Zabbix Tutorials? Walkthroughs? Clever uses for the Demo? Any and all comments appreciated.

Feel free to add new hosts, or even install a zabbix agent on your side and monitor your own server (until midnight that is, when the DB will be reset). I get a lot of hits on my site every day from people searching google for zabbix demo’s. Hopefully this will raise awareness of zabbix and get more people using this excellent tool.

Of course, I can also help you with your zabbix deployment. I can do a custom install for your environment, or I can ship you a pre configured zabbix box that you just plug in and go. If you’d like talk about my zabbix services, please indicate so on the Zabbix Demo form below. Hopefully the form you have to fill out to get access to the zabbix demo isn’t too painful. If you really dislike me for making you fill out a form to get the login credentials, feel free to call me names in every field except the email field. That one happens to be accurate in order to get the login credentials!

Stay tuned over the next several weeks for updates and new features.

Zabbix Demo

1. Your Name (required)
2. Email (valid email required)
3. Website (required)
4. Location (City,State,Country) (required)
5. Phone Number (required)
6. Zabbix Demo usage Please select one...Comparing to other products Evaluating zabbix upgrade Looking for ideas Looking to replace current monitoring system Looking for first monitoring system Other Leave me alone and just let me get to the demo! (required)
7. Please select any software packages you're currently using
8. zabbix mrtg OpenNMS Cacti Nagios BigBrother BigSister Syslog-ng Other Custom
9. Would you like to be contacted by MuckNet?Please select one...Yes No Leave me alone, and let me get to the demo! (required)
10. Comments or questions?

cforms contact form by delicious:days

If you’re not familiar with linux or open source tools, finding all the dependencies, downloading the source, compiling source, creating the db, etc can be a daunting task. So I’ve created this cut and paste walk through to help you through those steps. Almost everything here is cut and paste, except for hostname, and password information You’ll need to provide those on your own. I’ve done my best to make this as accurate as possible. I hate walkthroughs that just aren’t accurate! CentOS was installed choosing zero options, with as base of an installation as it would let me. I used the 2.6.18-8 kernel. If you have any questions, or find any errors, please let me know. And of course as usual, if you find it helpful, also, please let me know

I wrote these instructions using 1.4.1 as the example, but theres no reason why 1.4.2 shouldn’t work the same way

Update, 11/4/2007: 1.4.2 seems to install its binaries under prefix/sbin instead of prefix/bin, which is different than 1.4.1 which was used for this document. I’ve also noticed that when copying and pasting from this guide some of the whitespace, apostrophies, and dashes (’ – ) seem to get distorted upon pasting. Its correct in the source, but when its displayed something is munged up. When I figure out what it is, I’ll fix it. In the meantime if you get a syntax error, try retyping what I’ve put on this page instead of copying and pasting. And if you know why its happening, let me know!

Install all the necessary pieces. I started with a very base installation of CentOS 5.

yum -y install ntp php php-bcmath php-gd php-mysql httpd mysql gcc mysql-server mysql-devel net-snmp net-snmp-utils net-snmp-devel net-snmp-libs curl-devel mak

Start up the time server. its important for the time between your devices to be in sync.

/etc/init.d/ntpd start

Download fPing, and install it:

wget http://dag.wieers.com/rpm/packages/fping/fping-2.4-1.b2.2.el5.rf.i386.rpm

rpm -Uvh fping-2.4-1.b2.2.el5.rf.i386.rpm

chmod 7555 /usr/sbin/fping

Create Zabbix user.

useradd zabbix

Download zabbix and untar it.

wget http://superb-east.dl.sourceforge.net/sourceforge/zabbix/zabbix-1.4.1.tar.gz

tar -xzvf zabbix-1.4.1.tar.gz

Start MySQL, and change the root password.

/etc/init.d/mysqld start

/usr/bin/mysqladmin -u root password YourFancyNewRootPassword

Connect to the DB using your newly created root password. Create the zabbix DB, and assign a new user (zabbixmysqluser) with privileges to that DB. You may want to change “zabbixmysqlpassword” to something else. But it should not be the same as any other “critical” password because it will be stored plain text in a config file.

mysql -u root -p

mysql> CREATE DATABASE zabbix;

mysql> GRANT DROP,INDEX,CREATE,SELECT,INSERT,UPDATE,ALTER,DELETE ON zabbix.* TO zabbixmysqluser@localhost IDENTIFIED BY ‘zabbixmysqlpassword’;

mysql> quit;

Create the DB Schema

cd zabbix-1.4.1

cat create/schema/mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/data.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/images_mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

make clean

Compile the agent. I chose to compile it staticly, so it can be copied easily to other hosts.

./configure –enable-agent –prefix=/usr/local/zabbix –enable-static

make install

Add the zabbix server and agent ports to your /etc/services file.

echo ‘zabbix_agent 10050/tcp’ >> /etc/services

echo ‘zabbix_trap 10051/tcp’ >> /etc/services

Copy the sample configs to /etc/zabbix for server and agentd.

mkdir /etc/zabbix

cp misc/conf/zabbix_agentd.conf /etc/zabbix

cp misc/conf/zabbix_server.conf /etc/zabbix

in /etc/zabbix/zabbix_server.conf, modify:

DBUser=zabbixmysqluser

BPassword=zabbixmysqlpassword

DBSocket=/var/lib/mysql/mysql.sock

FpingLocation=/usr/sbin/fping

in /etc/zabbix/zabbix_agentd.conf, modify:

Server=127.0.0.1,Your.Zabbix.Server.IP

Hostname=EnterAUniqueHostNameForEachAgent

cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
cp misc/init.d/redhat/zabbix_server_ctl /etc/init.d/zabbix_server

in /etc/init.d/zabbix_agentd AND /etc/init.d/zabbix_server:

BASEDIR=/usr/local/zabbix

in /etc/init.d/zabbix_agentd (Note the # hash marks, they are necessary), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Agentd

in /etc/init.d/zabbix_server (again, note the # Hash marks, they are required), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Server

Configure automatic starting and stopping of services.

chkconfig –level 345 zabbix_server on

chkconfig –level 345 zabbix_agentd on

chkconfig –level 345 httpd on

chkconfig –level 345 mysqld on

chkconfig –level 0123456 iptables off

/etc/init.d/iptables stop

Note: I turn the iptables firewall OFF because my box is behind a firewall. You should consult with your network folks before turning off the firewall. At the very least you should poke holes for port 80, 10050, and 10051 in the firewall.

cp -r frontends/php /var/www/html/zabbix

in /etc/php.ini, modify:

max_execution_time = 300

date.timezone = America/Los_Angeles

Note: Obviously you should substitute your own time zone. For a list of all valid timezones, click here

/etc/init.d/httpd start

chmod 777 /var/www/html/zabbix/conf

Launch http://your.servers.name/zabbix inyour browser. You should be prompted with a setup screen. Click through the user agreement, and when you get to the Pre requisites screen, make sure you have a green OK next to everything.

When you’ve finished walking through the web interface setup:

chmod 755 /var/www/html/zabbix/conf

mv /var/www/html/zabbix/setup.php /var/www/html/zabbix/setup.php.bak

/etc/init.d/zabbix_agentd start

/etc/init.d/zabbix_server start

You can now login to your zabbix installation by going to http://your.server.name/zabbix, using the username “admin”, with no password. To monitor your zabbix server, you can go to the Configuration Tab, and choose the “hosts” sub Tab. Select the “Zabbix Server” host, by putting a checkmark next to it. and choose the “Activate Selected” button below. Wait a minute or two, then select the “Monitoring” tab, and then the “latest data” sub tab. You should start seeing performance stats appear!

For Reference, your binaries are under /usr/local/zabbix/bin, and your configuration files are in /etc/zabbix.

I’m not a big fan of their default template, I think the naming sucks. Look for a future article talking about renaming zabbix items. But this should be enough to get you started! You can find the answers to most of your questions in the Zabbix manual, available here: http://www.zabbix.com/documentation.php . You can also find lots of answers in the zabbix forums .

I like zabbix… but it annoys me that it’s ops per second, and bytes per second data is broken for hard drives in the linux 2.6 kernel. So I created a work around

Add the following code to your zabbix_agentd.conf file (/etc/zabbix/zabbix_agentd.conf by default), and restart the zabbix agent:

UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

The names are pretty descriptive, but you can find more information on how to decipher /proc/diskstats here: http://www.kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/iostats.txt

After you add the UserParameter’s to your config file, you can check that they work by using zabbix_get from your zabbix server:

zabbix_get -s HostName -p 10050 -k custom.vfs.dev.write.ops[sdb]

It should return a large number, and not something like: ZBX_NOTSUPPORTED. If it comes back not supported, make sure you restarted the agent after adding the userparameters. Then try running the commands outside of zabbix. Note that if you run the commands outsie of zabbix, you’ll need to replace the first $1 with a disk name, and remove one of the $ characters from the awk command.

Once you’ve verfied everything is in working order, you can add it as an item to your zabbix installation:

You can download my exported Template here , which will create a template called Template_app_FileServer. It will give you:

Disk Read/Write: Bytes/sec (or KB, MB per second)

Disk Read/Write: Ops/second

Disk Read/Write: Milliseconds spent reading or writing

Disk:IO currently executing

Disk:IO ms: time spent performing IO

If you’d rather just create your own items, and not bother loading mine, I have attached some screenshots of the config screens, so you can re create it with the right settings:

As always, if you find this page helpful or have any questions, please let me know!