Archive for the ‘Zabbix’ Category

Zabbix uses libcurl (libraries, not binaries) to do its Web Scenarios. Web scenarios are very powerful, and allow you to emulate a user experience. Using a Zabbix web scenario, you can emulate logging into your site, accepting the cookie, clicking on something unique (Report showing 10 Last purchases for example), then verify that you get either a particular HTTP code, or that certain text shows up in the response. Way cool stuff. Its got a few kinks to be worked out, however. One very frustrating one is that these web scenarios are not template aware yet… But the zabbix team is working on it, and its going to be a part of a future release. One minor, but significant thing for several of my environments is that the web scenario will error out if the SSL certificate CN (www.yoursite.com) does not match the URL you accessed the web server with.

But Doug, thats bad practice for the CN to not match the url!

I know! However, in most environments its not uncommon for the internal DNS name to NOT match the external DNS name. For example, the CN name for your SSL cert will be www.yoursite.com, but internally you have 10 app servers responding as www. You refer to them as app1.sjc.yoursite.com, app2.phx.yoursite.com.

By default curl (and therefore zabbix) will error out. With the following:

Failed on “HTTPS TEST app1.sjc.yoursite.com” [1 of 1] Error: SSL peer certificate was not ok

I’ve written a patch for the zabbix_server binary, which will instruct libcurl to not error out, and life is peachy! You need to unpack the zabbix source, apply the patch, recompile, and install the new binary. The patch, and steps are below:

I’ve attached the patch to this post, I’d suggest downloading it, instead of copying and pasting, but if you’d like to here it is:

--- src/zabbix_server/httppoller/httptest.c     2007-08-20 12:22:22.000000000 -0700
 +++ src/zabbix_server/httppoller/httptest.c.dp  2007-11-13 17:53:54.000000000 -0800
 @@ -318,6 +318,15 @@ static void        process_httptest(DB_HTTPTEST
 return;
 }
+       /* Process certs whose hostnames do not match the queried hostname. */
 +       if(CURLE_OK != (err = curl_easy_setopt(easyhandle,CURLOPT_SSL_VERIFYHOST , 0)))
 +       {
 +               zabbix_log(LOG_LEVEL_ERR, "Cannot set CURLOPT_SSL_VERIFYHOST [%s]",
 +                       curl_easy_strerror(err));
 +               (void)curl_easy_cleanup(easyhandle);
 +               return;
 +       }
 +
 lastfailedstep=0;
 httptest->time = 0;
 result = DBselect("select httpstepid,httptestid,no,name,url,timeout,posts,required,status_codes from httpstep where httptest
 id=" ZBX_FS_UI64 " order by no",

Link to Patch: libcurl disable ssl verifyhost

Instructions for installing patch:

Shut down zabbix_server process

/etc/init.d/zabbix_server stop

If you already have your zabbix source unpacked, you can skip the first tar step :) I’ve checked the patch with Zabbix 1.4.1 and 1.4.2

tar -zxvf zabbix-1.4.2.tar.gz

cd zabbix-1.4.2

wget http://www.muck.net/wp-content/uploads/2007/11/libcurl_disable_ssl_verifyhost.patch

patch src/zabbix_server/httppoller/httptest.c libcurl_ssl_verifyhost.patch

Then build zabbix_server as normal, for example:

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

Restart zabbix_server

/etc/init.d/zabbix_server start

Your Internal SSL Web Scenarios should now work! That was easy wasn’t it?

As always, I appreciate any feedback, and would love to hear if this helped you, or if you have any questions! :)

If you’re not familiar with linux or open source tools, finding all the dependencies, downloading the source, compiling source, creating the db, etc can be a daunting task. So I’ve created this cut and paste walk through to help you through those steps. Almost everything here is cut and paste, except for hostname, and password information :) You’ll need to provide those on your own. I’ve done my best to make this as accurate as possible. I hate walkthroughs that just aren’t accurate! CentOS was installed choosing zero options, with as base of an installation as it would let me. I used the 2.6.18-8 kernel. If you have any questions, or find any errors, please let me know. And of course as usual, if you find it helpful, also, please let me know :)

I wrote these instructions using 1.4.1 as the example, but theres no reason why 1.4.2 shouldn’t work the same way :)

Update, 11/4/2007: 1.4.2 seems to install its binaries under prefix/sbin instead of prefix/bin, which is different than 1.4.1 which was used for this document. I’ve also noticed that when copying and pasting from this guide some of the whitespace, apostrophies, and dashes (‘ – ) seem to get distorted upon pasting. Its correct in the source, but when its displayed something is munged up. When I figure out what it is, I’ll fix it. In the meantime if you get a syntax error, try retyping what I’ve put on this page instead of copying and pasting. And if you know why its happening, let me know! :)

Install all the necessary pieces. I started with a very base installation of CentOS 5.

yum -y install ntp php php-bcmath php-gd php-mysql httpd mysql gcc mysql-server mysql-devel net-snmp net-snmp-utils net-snmp-devel net-snmp-libs curl-devel mak

Start up the time server. its important for the time between your devices to be in sync.

/etc/init.d/ntpd start

Download fPing, and install it:

wget http://dag.wieers.com/rpm/packages/fping/fping-2.4-1.b2.2.el5.rf.i386.rpm

rpm -Uvh fping-2.4-1.b2.2.el5.rf.i386.rpm

chmod 7555 /usr/sbin/fping

Create Zabbix user.

useradd zabbix

Download zabbix and untar it.

wget http://superb-east.dl.sourceforge.net/sourceforge/zabbix/zabbix-1.4.1.tar.gz

tar -xzvf zabbix-1.4.1.tar.gz

Start MySQL, and change the root password.

/etc/init.d/mysqld start

/usr/bin/mysqladmin -u root password YourFancyNewRootPassword

Connect to the DB using your newly created root password. Create the zabbix DB, and assign a new user (zabbixmysqluser) with privileges to that DB. You may want to change “zabbixmysqlpassword” to something else. But it should not be the same as any other “critical” password because it will be stored plain text in a config file.

mysql -u root -p

mysql> CREATE DATABASE zabbix;

mysql> GRANT DROP,INDEX,CREATE,SELECT,INSERT,UPDATE,ALTER,DELETE ON zabbix.* TO zabbixmysqluser@localhost IDENTIFIED BY ‘zabbixmysqlpassword';

mysql> quit;

Create the DB Schema

cd zabbix-1.4.1

cat create/schema/mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/data.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/images_mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

make clean

Compile the agent. I chose to compile it staticly, so it can be copied easily to other hosts.

./configure –enable-agent –prefix=/usr/local/zabbix –enable-static

make install

Add the zabbix server and agent ports to your /etc/services file.

echo ‘zabbix_agent 10050/tcp’ >> /etc/services

echo ‘zabbix_trap 10051/tcp’ >> /etc/services

Copy the sample configs to /etc/zabbix for server and agentd.

mkdir /etc/zabbix

cp misc/conf/zabbix_agentd.conf /etc/zabbix

cp misc/conf/zabbix_server.conf /etc/zabbix

in /etc/zabbix/zabbix_server.conf, modify:

DBUser=zabbixmysqluser

BPassword=zabbixmysqlpassword

DBSocket=/var/lib/mysql/mysql.sock

FpingLocation=/usr/sbin/fping

in /etc/zabbix/zabbix_agentd.conf, modify:

Server=127.0.0.1,Your.Zabbix.Server.IP

Hostname=EnterAUniqueHostNameForEachAgent

cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
cp misc/init.d/redhat/zabbix_server_ctl /etc/init.d/zabbix_server

in /etc/init.d/zabbix_agentd AND /etc/init.d/zabbix_server:

BASEDIR=/usr/local/zabbix

in /etc/init.d/zabbix_agentd (Note the # hash marks, they are necessary), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Agentd

in /etc/init.d/zabbix_server (again, note the # Hash marks, they are required), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Server

Configure automatic starting and stopping of services.

chkconfig –level 345 zabbix_server on

chkconfig –level 345 zabbix_agentd on

chkconfig –level 345 httpd on

chkconfig –level 345 mysqld on

chkconfig –level 0123456 iptables off

/etc/init.d/iptables stop

Note: I turn the iptables firewall OFF because my box is behind a firewall. You should consult with your network folks before turning off the firewall. At the very least you should poke holes for port 80, 10050, and 10051 in the firewall.

cp -r frontends/php /var/www/html/zabbix

in /etc/php.ini, modify:

max_execution_time = 300

date.timezone = America/Los_Angeles

Note: Obviously you should substitute your own time zone. For a list of all valid timezones, click here

/etc/init.d/httpd start

chmod 777 /var/www/html/zabbix/conf

Launch http://your.servers.name/zabbix inyour browser. You should be prompted with a setup screen. Click through the user agreement, and when you get to the Pre requisites screen, make sure you have a green OK next to everything.

Zabbix pre req’s

Zabbix DB config

When you’ve finished walking through the web interface setup:

chmod 755 /var/www/html/zabbix/conf

mv /var/www/html/zabbix/setup.php /var/www/html/zabbix/setup.php.bak

/etc/init.d/zabbix_agentd start

/etc/init.d/zabbix_server start

You can now login to your zabbix installation by going to http://your.server.name/zabbix, using the username “admin”, with no password. To monitor your zabbix server, you can go to the Configuration Tab, and choose the “hosts” sub Tab. Select the “Zabbix Server” host, by putting a checkmark next to it. and choose the “Activate Selected” button below. Wait a minute or two, then select the “Monitoring” tab, and then the “latest data” sub tab. You should start seeing performance stats appear!

For Reference, your binaries are under /usr/local/zabbix/bin, and your configuration files are in /etc/zabbix.

I’m not a big fan of their default template, I think the naming sucks. Look for a future article talking about renaming zabbix items. But this should be enough to get you started! :) You can find the answers to most of your questions in the Zabbix manual, available here: http://www.zabbix.com/documentation.php . You can also find lots of answers in the zabbix forums .

I like zabbix… but it annoys me that it’s ops per second, and bytes per second data is broken for hard drives in the linux 2.6 kernel. So I created a work around :)

Add the following code to your zabbix_agentd.conf file (/etc/zabbix/zabbix_agentd.conf by default), and restart the zabbix agent:

UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

The names are pretty descriptive, but you can find more information on how to decipher /proc/diskstats here: http://www.kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/iostats.txt

After you add the UserParameter’s to your config file, you can check that they work by using zabbix_get from your zabbix server:

zabbix_get -s HostName -p 10050 -k custom.vfs.dev.write.ops[sdb]

It should return a large number, and not something like: ZBX_NOTSUPPORTED. If it comes back not supported, make sure you restarted the agent after adding the userparameters. Then try running the commands outside of zabbix. Note that if you run the commands outsie of zabbix, you’ll need to replace the first $1 with a disk name, and remove one of the $ characters from the awk command.

Once you’ve verfied everything is in working order, you can add it as an item to your zabbix installation:

You can download my exported Template here , which will create a template called Template_app_FileServer. It will give you:

Disk Read/Write: Bytes/sec (or KB, MB per second)

Disk Read/Write: Ops/second

Disk Read/Write: Milliseconds spent reading or writing

Disk:IO currently executing

Disk:IO ms: time spent performing IO

If you’d rather just create your own items, and not bother loading mine, I have attached some screenshots of the config screens, so you can re create it with the right settings:

Bytes/sec chart

Bytes/Sec config

Operations/sec config

active IO config

As always, if you find this page helpful or have any questions, please let me know! :)