Some older cisco IOS routers don’t have the new time zone information. Below is an example of my time related configuration, including NTP and logging options. Configure this and you will no longer be lost when looking at logs!

My timezone is obviously Pacific, but you can insert your own. CDT, EDT, CST, EST, etc

service timestamps debug datetime localtime
service timestamps log datetime localtime
clock timezone PST -8
clock summer-time PDT recurring 2 Sun Mar 2:00 2 Sun Nov 2:00
ntp logging
ntp update-calendar
ntp server 4.us.pool.ntp.org
ntp server 2.us.pool.ntp.org
ntp server 1.us.pool.ntp.org
ntp server 3.us.pool.ntp.org
ntp server 0.us.pool.ntp.org

OK, so what is the problem? Sendmail is archaic. Postfix tries to be backwards compatible with archaic. Neither of them keep a running counter of total messages sent. So in order to keep an eye on how many mail messages are coming out of your system (or not coming out of your system as the case may unfortunately be), you have to come up with some hokey way of grabbing those stats from the log file. I spent an hour or so searching for someones already written script, thinking SOMEONE has to have done this. And while I found a lot of people asking the question, I didn’t find any tidy, wrapped up answers.

There are lots of solutions to comb your logs nightly and produce a graph or report, but I wanted real time data, and I wanted the data in my NMS, so I could do my own graphing and alerting. One thing I absolutely hate is having 18 applications capable of alerting me of various things. So I cobbled together this perl script, which will check the last X minutes of a logfile for a string, and return it to you. You can then pipe it through a ‘wc -l’ to count how many times it occured. You can put it in cron, and pipe it through mail, to send you a copy, whatever you want.

Requirements:

ReadBackwards Perl Module - You can install from source if your servers don’t have internet access (IMHO – production servers should never have outbound internet access, inbound only — I’ll save that lecture for another blog post. Maybe.). If you have internet access just run:

perl -MCPAN -e ‘install File::ReadBackwards’

/var/log/maillog permissions – (or whatever log file you’re looking at). Make sure the user you’re reading the log files with can read that log file by default your unprivileged zabbix user cannot read /var/log/maillog. You can fix this any number of ways. chmod. sudo. changing file ownership. I chose to change file ownership. this seems the cleanest, and I don’t view it as a security risk for my maillog to be readable by all users on the system. Before you ask – Yes, your logrotate process will continue to propogate those new permissions, because it gets its cue’s, from the file it is rotating

chmod a+r /var/log/maillog

The script itself is below, but I will also attach it to this post, because wordpress will surely screw the formatting or characters up.. so please download the attached perl script here, before complaining it won’t compile: count_pf_mail.pl.txt

#!/usr/bin/perl
use File::ReadBackwards;
use POSIX "strftime";

my $items;
my $time = $ARGV[0];
my $logfile = "/var/log/maillog";
my $search_string = "status=sent";
my $goback = POSIX::strftime( "%b %d %H:%M:%S", localtime( time() - $time * 60 ) );

$readback = File::ReadBackwards->new( $logfile ) or
                        die "unable to open '$logfile' $!" ;

while ( defined( my $line = $readback->readline() ) ) {

    last if $line lt $goback;

    if ($line =~ /$search_string/) {
        print $line;
    }

}

The only argument the script takes is number of minutes to go back in the log file.

[doug@app5 ~]# /usr/local/bin/count_pf_mail.pl 5

Mar 20 11:12:01 app5 postfix/smtp[31232]: D864A77EBA: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.27, delays=0.04/0/0.08/0.14, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 n20si3574089pof.6)
Mar 20 11:12:02 app5 postfix/smtp[31223]: DAD4B77EB3: to=, relay=gmail-smtp-in.l.google.com[72.14.253.27]:25, delay=0.22, delays=0.02/0.01/0.08/0.11, dsn=2.0.0, status=sent (250 2.0.0 OK 1206036799 q18si3562811pog.12)

…

Is this totally accurate?

Absolutely not. Its close, but its not perfect. It assumes that your monitoring time period will be at EXACTLY the right interval. That won’t ever happen, so its possible that some mail will be counted twice, and some will not be counted at all. However, its good enough for comparison purposes, and its better than nothing In my ideal world, this script would feed into a database/file of already counted message ID’s, to make sure that the same message ID isn’t counted twice. But This works good enough for me

OK, so how do I make this work with zabbix?

zabbix_agentd.conf:

The only UserParameter you’re really worried about for this particular usage is the last one, but if you download my zabbix template below, you’ll want the other 2 also

UserParameter=mailq,/usr/bin/mailq | grep Request | awk '{print $5}'
UserParameter=established_conns[*], netstat -an | grep :$1 | grep ESTABLISHED |wc -l
UserParameter=count_pf_mail[*], /usr/local/bin/count_pf_mail.pl $1 |wc -l

Creating a zabbix item:

Choosing a longer polling period will make the results more accurate over time. I’ve been experimenting with 3 minutes, 5 minutes, etc. I wouldn’t suggest anything less than 3 minutes. The following example uses a 3 minute polling interval, and displays it as a “per minute” statistic. You could modify to be per second, per hour, per 5 minutes if you want. Its all just math. I’ve highlighted the parameters that need to be changed if you want to change the polling period.

Key: count_pf_mail[3]

Type of Information: Float

Multiplier (Use Custom): .333

Update Interval: 180

If you wanted to do 5 minute polling intervals, and still display it as a per minute statistic:

Key: count_pf_mail[5]

Type of Information: Float

Multiplier (Use Custom): .2

Update Interval: 300

Zabbix Graph showing the data from my (not yet active) postfix cluster:

You can also download my postfix template here: template_app_postfix.xml.txt

It will work with the mailq and established_conns Userparameter above. It is by no means completed or final, but hopefully you’ll find it useful anyway. Everything here could be modified pretty easily to work with sendmail also.

As always if you find this post helpful, let me know! Comments, suggestions, and corrections are always welcome!

First, remove the password:

1) Unplug the Switch

2) Plug the switch back in, and be immediately ready to:

3) Hit b, to enter into the boot monitor

4) Type:

no password

boot system flash primary

5) Foundry will boot up, and you can ‘enable’ without being prompted for a password

To reset your ServerIronXL to factory defaults:

6) After enabling, type:

erase start

NOTE: This is permanent! There is no going back! Make sure this is what you want to do! You are resetting to factory defaults (nothing!)

7) Reboot, and enjoy your clean slate

As always, if you find this helpful, please let me know!

But Doug, thats bad practice for the CN to not match the url!

I know! However, in most environments its not uncommon for the internal DNS name to NOT match the external DNS name. For example, the CN name for your SSL cert will be www.yoursite.com, but internally you have 10 app servers responding as www. You refer to them as app1.sjc.yoursite.com, app2.phx.yoursite.com.

By default curl (and therefore zabbix) will error out. With the following:

Failed on “HTTPS TEST app1.sjc.yoursite.com” [1 of 1] Error: SSL peer certificate was not ok

I’ve written a patch for the zabbix_server binary, which will instruct libcurl to not error out, and life is peachy! You need to unpack the zabbix source, apply the patch, recompile, and install the new binary. The patch, and steps are below:

I’ve attached the patch to this post, I’d suggest downloading it, instead of copying and pasting, but if you’d like to here it is:

--- src/zabbix_server/httppoller/httptest.c     2007-08-20 12:22:22.000000000 -0700
 +++ src/zabbix_server/httppoller/httptest.c.dp  2007-11-13 17:53:54.000000000 -0800
 @@ -318,6 +318,15 @@ static void        process_httptest(DB_HTTPTEST
 return;
 }

+       /* Process certs whose hostnames do not match the queried hostname. */
 +       if(CURLE_OK != (err = curl_easy_setopt(easyhandle,CURLOPT_SSL_VERIFYHOST , 0)))
 +       {
 +               zabbix_log(LOG_LEVEL_ERR, "Cannot set CURLOPT_SSL_VERIFYHOST [%s]",
 +                       curl_easy_strerror(err));
 +               (void)curl_easy_cleanup(easyhandle);
 +               return;
 +       }
 +
 lastfailedstep=0;
 httptest->time = 0;
 result = DBselect("select httpstepid,httptestid,no,name,url,timeout,posts,required,status_codes from httpstep where httptest
 id=" ZBX_FS_UI64 " order by no",

Link to Patch: libcurl disable ssl verifyhost

Instructions for installing patch:

Shut down zabbix_server process

/etc/init.d/zabbix_server stop

If you already have your zabbix source unpacked, you can skip the first tar step I’ve checked the patch with Zabbix 1.4.1 and 1.4.2

tar -zxvf zabbix-1.4.2.tar.gz

cd zabbix-1.4.2

wget http://www.muck.net/wp-content/uploads/2007/11/libcurl_disable_ssl_verifyhost.patch

patch src/zabbix_server/httppoller/httptest.c libcurl_ssl_verifyhost.patch

Then build zabbix_server as normal, for example:

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

Restart zabbix_server

/etc/init.d/zabbix_server start

Your Internal SSL Web Scenarios should now work! That was easy wasn’t it?

As always, I appreciate any feedback, and would love to hear if this helped you, or if you have any questions!

The problem with the old zabbix demo was that it was the only copy. If someone messed it up, I had to manually go in and restore the DB. It didn’t have a self recovery method. In the end, it meant that I was limited to handing out read only access in most cases. That in turn meant it wasn’t used very often by folks, and because it wasn’t used very often, I neglected it.

However! I have now put zabbix into its own VM universe. I took a snapshot of a “clean” zabbix installation, which is known to be good. Every 24 hours, I wipe out the VM instance, put a new one in place, restore the Zabbix DB, and timeshift all the data so it looks like the data is contiguous, even though in all likelyhood its several weeks old. The process results in about 10 minutes of downtime, and another 20 minutes of data processing in the background. Any data displayed from Midnight PST through Midnight PST the following night, is live, accurate, real data, not timeshifted data.

I’m pretty happy with it so far. I still want to add a lot more items and hosts to the demo, but I think its a great start. As far as I know this is the only Zabbix Demo, so I’d love to hear from you what you’d like to see in the demo. More graphs? Screens? Web Monitoring? oracle monitoring? mysql monitoring? Zabbix Tutorials? Walkthroughs? Clever uses for the Demo? Any and all comments appreciated.

Feel free to add new hosts, or even install a zabbix agent on your side and monitor your own server (until midnight that is, when the DB will be reset). I get a lot of hits on my site every day from people searching google for zabbix demo’s. Hopefully this will raise awareness of zabbix and get more people using this excellent tool.

Of course, I can also help you with your zabbix deployment. I can do a custom install for your environment, or I can ship you a pre configured zabbix box that you just plug in and go. If you’d like talk about my zabbix services, please indicate so on the Zabbix Demo form below. Hopefully the form you have to fill out to get access to the zabbix demo isn’t too painful. If you really dislike me for making you fill out a form to get the login credentials, feel free to call me names in every field except the email field. That one happens to be accurate in order to get the login credentials!

Stay tuned over the next several weeks for updates and new features.

cforms contact form by delicious:days

Update 12/6/2007: There is a bug with more recent updates of RHEL5 (Similar or the same as this bug: https://bugzilla.redhat.com/show_bug.cgi?id=228430). The bug will cause the original “yum install” command to fail with the following:

Error: No Package Matching glibc.i686

To prevent that, install glibc first, then install the rest of the stuff you want:

yum install glibc

yum install kernel-devel gcc kernel-headers openssl gcc-c++ openssl-devel boost-devel

Update (2/5/2008): I added boost-devel to the install list, because of an error I encountered installing on CentOS 4.

checking boost/shared_ptr.hpp usability… no
checking boost/shared_ptr.hpp presence… no
checking for boost/shared_ptr.hpp… no
configure: error:
Can’t find boost/shared_.h – add the boost include dir to CPPFLAGS and
rerun configure, eg:
export CPPFLAGS=-I/usr/local/include

Download latest fuse (2.7.1 at this time 10/2007)

wget http://internap.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.1.tar.gz

Download latest rlog (1.3.7 at this time)

wget http://freshmeat.net/redir/rlog/47815/url_tgz/rlog-1.3.7.tgz

Download latest encFS (1.3.2-1 at this time)

wget http://freshmeat.net/redir/encfs/45687/url_tgz/encfs-1.3.2-1.tgz

Rlog:

tar -xvzf rlog-1.3.7.tgz

cd rlog-1.3.7

./configure

make

make install

Fuse: lather, rinse

tar -xvzf fuse-2.7.1.tar.gz

cd fuse-2.7.1

./configure

make

make install

encF: and repeat

tar -xzvf encfs-1.3.2-1.tgz

cd encfs-1.3.2

./configure

make

make install

Start fuse:

/etc/init.d/fuse start

Fix init script for CentOS, replace the startup information at the top with:

### BEGIN INIT INFO
# chkconfig: 2345 90 10
# description:       Load the fuse module and mount the fuse control
#       filesystem.
### END INIT INFO

Run:

chkconfig –add fuse

Create an Encrypted Filesystem (Its not really a filesystem… but I digress) as a test:

encfs /usb/disk1/.crypt-raw /usb/disk1/crypt-mount

It really is that easy. Good luck!

I wrote these instructions using 1.4.1 as the example, but theres no reason why 1.4.2 shouldn’t work the same way

Update, 11/4/2007: 1.4.2 seems to install its binaries under prefix/sbin instead of prefix/bin, which is different than 1.4.1 which was used for this document. I’ve also noticed that when copying and pasting from this guide some of the whitespace, apostrophies, and dashes (’ – ) seem to get distorted upon pasting. Its correct in the source, but when its displayed something is munged up. When I figure out what it is, I’ll fix it. In the meantime if you get a syntax error, try retyping what I’ve put on this page instead of copying and pasting. And if you know why its happening, let me know!

Install all the necessary pieces. I started with a very base installation of CentOS 5.

yum -y install ntp php php-bcmath php-gd php-mysql httpd mysql gcc mysql-server mysql-devel net-snmp net-snmp-utils net-snmp-devel net-snmp-libs curl-devel mak

Start up the time server. its important for the time between your devices to be in sync.

/etc/init.d/ntpd start

Download fPing, and install it:

wget http://dag.wieers.com/rpm/packages/fping/fping-2.4-1.b2.2.el5.rf.i386.rpm

rpm -Uvh fping-2.4-1.b2.2.el5.rf.i386.rpm

chmod 7555 /usr/sbin/fping

Create Zabbix user.

useradd zabbix

Download zabbix and untar it.

wget http://superb-east.dl.sourceforge.net/sourceforge/zabbix/zabbix-1.4.1.tar.gz

tar -xzvf zabbix-1.4.1.tar.gz

Start MySQL, and change the root password.

/etc/init.d/mysqld start

/usr/bin/mysqladmin -u root password YourFancyNewRootPassword

Connect to the DB using your newly created root password. Create the zabbix DB, and assign a new user (zabbixmysqluser) with privileges to that DB. You may want to change “zabbixmysqlpassword” to something else. But it should not be the same as any other “critical” password because it will be stored plain text in a config file.

mysql -u root -p

mysql> CREATE DATABASE zabbix;

mysql> GRANT DROP,INDEX,CREATE,SELECT,INSERT,UPDATE,ALTER,DELETE ON zabbix.* TO zabbixmysqluser@localhost IDENTIFIED BY ‘zabbixmysqlpassword’;

mysql> quit;

Create the DB Schema

cd zabbix-1.4.1

cat create/schema/mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/data.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/images_mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

make clean

Compile the agent. I chose to compile it staticly, so it can be copied easily to other hosts.

./configure –enable-agent –prefix=/usr/local/zabbix –enable-static

make install

Add the zabbix server and agent ports to your /etc/services file.

echo ‘zabbix_agent 10050/tcp’ >> /etc/services

echo ‘zabbix_trap 10051/tcp’ >> /etc/services

Copy the sample configs to /etc/zabbix for server and agentd.

mkdir /etc/zabbix

cp misc/conf/zabbix_agentd.conf /etc/zabbix

cp misc/conf/zabbix_server.conf /etc/zabbix

in /etc/zabbix/zabbix_server.conf, modify:

DBUser=zabbixmysqluser

BPassword=zabbixmysqlpassword

DBSocket=/var/lib/mysql/mysql.sock

FpingLocation=/usr/sbin/fping

in /etc/zabbix/zabbix_agentd.conf, modify:

Server=127.0.0.1,Your.Zabbix.Server.IP

Hostname=EnterAUniqueHostNameForEachAgent

cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
cp misc/init.d/redhat/zabbix_server_ctl /etc/init.d/zabbix_server

in /etc/init.d/zabbix_agentd AND /etc/init.d/zabbix_server:

BASEDIR=/usr/local/zabbix

in /etc/init.d/zabbix_agentd (Note the # hash marks, they are necessary), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Agentd

in /etc/init.d/zabbix_server (again, note the # Hash marks, they are required), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Server

Configure automatic starting and stopping of services.

chkconfig –level 345 zabbix_server on

chkconfig –level 345 zabbix_agentd on

chkconfig –level 345 httpd on

chkconfig –level 345 mysqld on

chkconfig –level 0123456 iptables off

/etc/init.d/iptables stop

Note: I turn the iptables firewall OFF because my box is behind a firewall. You should consult with your network folks before turning off the firewall. At the very least you should poke holes for port 80, 10050, and 10051 in the firewall.

cp -r frontends/php /var/www/html/zabbix

in /etc/php.ini, modify:

max_execution_time = 300

date.timezone = America/Los_Angeles

Note: Obviously you should substitute your own time zone. For a list of all valid timezones, click here

/etc/init.d/httpd start

chmod 777 /var/www/html/zabbix/conf

Launch http://your.servers.name/zabbix inyour browser. You should be prompted with a setup screen. Click through the user agreement, and when you get to the Pre requisites screen, make sure you have a green OK next to everything.

When you’ve finished walking through the web interface setup:

chmod 755 /var/www/html/zabbix/conf

mv /var/www/html/zabbix/setup.php /var/www/html/zabbix/setup.php.bak

/etc/init.d/zabbix_agentd start

/etc/init.d/zabbix_server start

You can now login to your zabbix installation by going to http://your.server.name/zabbix, using the username “admin”, with no password. To monitor your zabbix server, you can go to the Configuration Tab, and choose the “hosts” sub Tab. Select the “Zabbix Server” host, by putting a checkmark next to it. and choose the “Activate Selected” button below. Wait a minute or two, then select the “Monitoring” tab, and then the “latest data” sub tab. You should start seeing performance stats appear!

For Reference, your binaries are under /usr/local/zabbix/bin, and your configuration files are in /etc/zabbix.

I’m not a big fan of their default template, I think the naming sucks. Look for a future article talking about renaming zabbix items. But this should be enough to get you started! You can find the answers to most of your questions in the Zabbix manual, available here: http://www.zabbix.com/documentation.php . You can also find lots of answers in the zabbix forums .

Add the following code to your zabbix_agentd.conf file (/etc/zabbix/zabbix_agentd.conf by default), and restart the zabbix agent:

UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

The names are pretty descriptive, but you can find more information on how to decipher /proc/diskstats here: http://www.kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/iostats.txt

After you add the UserParameter’s to your config file, you can check that they work by using zabbix_get from your zabbix server:

zabbix_get -s HostName -p 10050 -k custom.vfs.dev.write.ops[sdb]

It should return a large number, and not something like: ZBX_NOTSUPPORTED. If it comes back not supported, make sure you restarted the agent after adding the userparameters. Then try running the commands outside of zabbix. Note that if you run the commands outsie of zabbix, you’ll need to replace the first $1 with a disk name, and remove one of the $ characters from the awk command.

Once you’ve verfied everything is in working order, you can add it as an item to your zabbix installation:

You can download my exported Template here , which will create a template called Template_app_FileServer. It will give you:

Disk Read/Write: Bytes/sec (or KB, MB per second)

Disk Read/Write: Ops/second

Disk Read/Write: Milliseconds spent reading or writing

Disk:IO currently executing

Disk:IO ms: time spent performing IO

If you’d rather just create your own items, and not bother loading mine, I have attached some screenshots of the config screens, so you can re create it with the right settings:

As always, if you find this page helpful or have any questions, please let me know!

The result of these forming years of my life, is that I like good, fresh food. I am ALSO a firm believer in spending your money like you’re voting with it. So I do my best to buy locally. I’m not a zealot, or obsessive about this practice, but when its possible, I do my best. One way that I do this is by supporting my local farmer through their CSA (Community Support Agriculture) program.

Every week, I get a box full of organic seasonal fruits and vegetables . Strawberries, tomatoes, peppers, cauliflower, peas, carrots, onions, beets, squash, baby salads, big salads, spinach, potatoes, etc. Anything that grows in California between April and November. Its a great way to get fresh fruits and vegetables into my diet, which is obviously good for me. I send my check directly to the farmer, eliminating the middle man, which is good for the farmer. These types of programs are available all over the place. A good place to start your search is LocalHarvest.org

Periodically Mariquita farms (half of two small farms), will have a U pick tomato day. For 50 cents a pound, you can pick all the tomatoes you can carry out of there. Its fabulous. You’re not limited to tomatoes either. Peppers, pumpkins, tomatillos, or anything else you can get your hands on is fair game. 50 cents a pound is great. But I’m a really inquisitive person, so to ME the best part is that you get to pick the farmers brain , and ask him all the dumb questions you want. He’s HAPPY to answer them! Its an excellent experience.

Mariquita is having one of these U Pick days tomorrow, Sunday September, 23rd. If you get a chance, its worth the drive. Just make sure you wear a good pair of boots, its going to be wet out there

Here are some pictures of my bounty from the last U Pick day.

Update: Heres my bounty from the second U Pick day Can you tell I like salsa? I preserved most of it for the winter