MuckNet

The title of this post is about zabbix, because thats my NMS of choice, but this method would apply to just about any monitoring application out there. Cacti, Nagios, big brother, opennms, etc. It also is not limited to postfix. Sendmail would work with a minor modification (look for stat=Sent, instead of status=sent).

OK, so what is the problem? Sendmail is archaic. Postfix tries to be backwards compatible with archaic. Neither of them keep a running counter of total messages sent. So in order to keep an eye on how many mail messages are coming out of your system (or not coming out of your system as the case may unfortunately be), you have to come up with some hokey way of grabbing those stats from the log file. I spent an hour or so searching for someones already written script, thinking SOMEONE has to have done this. And while I found a lot of people asking the question, I didn’t find any tidy, wrapped up answers.

There are lots of solutions to comb your logs nightly and produce a graph or report, but I wanted real time data, and I wanted the data in my NMS, so I could do my own graphing and alerting. One thing I absolutely hate is having 18 applications capable of alerting me of various things. So I cobbled together this perl script, which will check the last X minutes of a logfile for a string, and return it to you. You can then pipe it through a ‘wc -l’ to count how many times it occured. You can put it in cron, and pipe it through mail, to send you a copy, whatever you want.

Requirements:

ReadBackwards Perl Module - You can install from source if your servers don’t have internet access (IMHO – production servers should never have outbound internet access, inbound only — I’ll save that lecture for another blog post. Maybe.). If you have internet access just run:

perl -MCPAN -e ‘install File::ReadBackwards’

/var/log/maillog permissions – (or whatever log file you’re looking at). Make sure the user you’re reading the log files with can read that log file by default your unprivileged zabbix user cannot read /var/log/maillog. You can fix this any number of ways. chmod. sudo. changing file ownership. I chose to change file ownership. this seems the cleanest, and I don’t view it as a security risk for my maillog to be readable by all users on the system. Before you ask – Yes, your logrotate process will continue to propogate those new permissions, because it gets its cue’s, from the file it is rotating

chmod a+r /var/log/maillog

The script itself is below, but I will also attach it to this post, because wordpress will surely screw the formatting or characters up.. so please download the attached perl script here, before complaining it won’t compile: count_pf_mail.pl.txt

#!/usr/bin/perl
use File::ReadBackwards;
use POSIX "strftime";

my $items;
my $time = $ARGV[0];
my $logfile = "/var/log/maillog";
my $search_string = "status=sent";
my $goback = POSIX::strftime( "%b %d %H:%M:%S", localtime( time() - $time * 60 ) );

$readback = File::ReadBackwards->new( $logfile ) or
                        die "unable to open '$logfile' $!" ;

while ( defined( my $line = $readback->readline() ) ) {

    last if $line lt $goback;

    if ($line =~ /$search_string/) {
        print $line;
    }

}

The only argument the script takes is number of minutes to go back in the log file.

[doug@app5 ~]# /usr/local/bin/count_pf_mail.pl 5

Mar 20 11:12:01 app5 postfix/smtp[31232]: D864A77EBA: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.27, delays=0.04/0/0.08/0.14, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 n20si3574089pof.6)
Mar 20 11:12:02 app5 postfix/smtp[31223]: DAD4B77EB3: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.22, delays=0.02/0.01/0.08/0.11, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 q18si3562811pog.12)

…

Is this totally accurate?

Absolutely not. Its close, but its not perfect. It assumes that your monitoring time period will be at EXACTLY the right interval. That won’t ever happen, so its possible that some mail will be counted twice, and some will not be counted at all. However, its good enough for comparison purposes, and its better than nothing In my ideal world, this script would feed into a database/file of already counted message ID’s, to make sure that the same message ID isn’t counted twice. But This works good enough for me

OK, so how do I make this work with zabbix?

zabbix_agentd.conf:

The only UserParameter you’re really worried about for this particular usage is the last one, but if you download my zabbix template below, you’ll want the other 2 also

UserParameter=mailq,/usr/bin/mailq | grep Request | awk '{print $5}'
UserParameter=established_conns[*], netstat -an | grep :$1 | grep ESTABLISHED |wc -l
UserParameter=count_pf_mail[*], /usr/local/bin/count_pf_mail.pl $1 |wc -l

Creating a zabbix item:

Choosing a longer polling period will make the results more accurate over time. I’ve been experimenting with 3 minutes, 5 minutes, etc. I wouldn’t suggest anything less than 3 minutes. The following example uses a 3 minute polling interval, and displays it as a “per minute” statistic. You could modify to be per second, per hour, per 5 minutes if you want. Its all just math. I’ve highlighted the parameters that need to be changed if you want to change the polling period.

Key: count_pf_mail[3]

Type of Information: Float

Multiplier (Use Custom): .333

Update Interval: 180

If you wanted to do 5 minute polling intervals, and still display it as a per minute statistic:

Key: count_pf_mail[5]

Type of Information: Float

Multiplier (Use Custom): .2

Update Interval: 300

Zabbix Graph showing the data from my (not yet active) postfix cluster:

You can also download my postfix template here: template_app_postfix.xml.txt

It will work with the mailq and established_conns Userparameter above. It is by no means completed or final, but hopefully you’ll find it useful anyway. Everything here could be modified pretty easily to work with sendmail also.

As always if you find this post helpful, let me know! Comments, suggestions, and corrections are always welcome!