MuckNet

I spent some time googling around this afternoon, and kept reading that iotop requires kernel 2.6.20 and python 2.6, and therefore was a bit of a challenge to get running on RHEL/CENTOS 5.X. However, the requirements are actually pretty basic, and easy to get running on a somewhat recent update of RHEL 5.

Redhat has backported the per process IO accounting feature to the 2.6.18-144 kernel. So if /proc/self/io exists, and you get results from /proc/<PID #>/io then iotop will work.

[root@gateway /]# cat /proc/self/io
rchar: 1900
wchar: 0
syscr: 7
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

[root@gateway /]# cat /proc/3227/io
rchar: 25970197
wchar: 26186855
syscr: 4859727
syscw: 2516047
read_bytes: 3854336
write_bytes: 47595520
cancelled_write_bytes: 274432
[root@gateway self]#

Basic Steps for setting up iotop on RHEL 5.7 (And probably older versions as well, though I haven’t tested).

Update Kernel to 2.6.8-144 or higher.

yum update kernel

Install epel repository – There are plenty of instructions out there, but it will look something like this for x64 RHEL:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Then you just need Python 2.4, python-ctypes, and iotop packages:

yum install python python-ctypes iotop

Thats it! Pretty easy huh?

Update 12/27/2011:

I had a case where iotop wouldn’t run on a linux 3.1 kernel, I believe because of mprotect() (Not sure though). In either case, I discovered that htop gets you most of what iotop will get you , plus a lot more. Its a pretty neat tool. I suggest checking it out.  You still need a kernel that has IO statistics in /proc (as mentioned above).

Starting in 2007, daylight time begins in the United States on the second Sunday in March and ends on the first Sunday in November. On the second Sunday in March, clocks are set ahead one hour at 2:00 a.m. local standard time, which becomes 3:00 a.m. local daylight time. On the first Sunday in November, clocks are set back one hour at 2:00 a.m. local daylight time, which becomes 1:00 a.m. local standard time. These dates were established by Congress in the Energy Policy Act of 2005, Pub. L. no. 109-58, 119 Stat 594 (2005).

Some older cisco IOS routers don’t have the new time zone information. Below is an example of my time related configuration, including NTP and logging options. Configure this and you will no longer be lost when looking at logs!

My timezone is obviously Pacific, but you can insert your own. CDT, EDT, CST, EST, etc

service timestamps debug datetime localtime
service timestamps log datetime localtime
clock timezone PST -8
clock summer-time PDT recurring 2 Sun Mar 2:00 2 Sun Nov 2:00
ntp logging
ntp update-calendar
ntp server 4.us.pool.ntp.org
ntp server 2.us.pool.ntp.org
ntp server 1.us.pool.ntp.org
ntp server 3.us.pool.ntp.org
ntp server 0.us.pool.ntp.org

I needed to reset a Foundry Server Iron XL back to factory defaults, and surprisingly couldn’t find the instructions via my buddy google. Foundry is really stingy with with support documents, and knowledge portal acccess, and unless you have a valid support contract, you can’t find ANYTHING. Luckily I do, so I figured I’d share this knowledge for the sake of future googlers.

First, remove the password:

1) Unplug the Switch

2) Plug the switch back in, and be immediately ready to:

3) Hit b, to enter into the boot monitor

4) Type:

no password

boot system flash primary

5) Foundry will boot up, and you can ‘enable’ without being prompted for a password

To reset your ServerIronXL to factory defaults:

6) After enabling, type:

erase start

NOTE: This is permanent! There is no going back! Make sure this is what you want to do! You are resetting to factory defaults (nothing!)

7) Reboot, and enjoy your clean slate

As always, if you find this helpful, please let me know!

Zabbix uses libcurl (libraries, not binaries) to do its Web Scenarios. Web scenarios are very powerful, and allow you to emulate a user experience. Using a Zabbix web scenario, you can emulate logging into your site, accepting the cookie, clicking on something unique (Report showing 10 Last purchases for example), then verify that you get either a particular HTTP code, or that certain text shows up in the response. Way cool stuff. Its got a few kinks to be worked out, however. One very frustrating one is that these web scenarios are not template aware yet… But the zabbix team is working on it, and its going to be a part of a future release. One minor, but significant thing for several of my environments is that the web scenario will error out if the SSL certificate CN (www.yoursite.com) does not match the URL you accessed the web server with.

But Doug, thats bad practice for the CN to not match the url!

I know! However, in most environments its not uncommon for the internal DNS name to NOT match the external DNS name. For example, the CN name for your SSL cert will be www.yoursite.com, but internally you have 10 app servers responding as www. You refer to them as app1.sjc.yoursite.com, app2.phx.yoursite.com.

By default curl (and therefore zabbix) will error out. With the following:

Failed on “HTTPS TEST app1.sjc.yoursite.com” [1 of 1] Error: SSL peer certificate was not ok

I’ve written a patch for the zabbix_server binary, which will instruct libcurl to not error out, and life is peachy! You need to unpack the zabbix source, apply the patch, recompile, and install the new binary. The patch, and steps are below:

I’ve attached the patch to this post, I’d suggest downloading it, instead of copying and pasting, but if you’d like to here it is:

--- src/zabbix_server/httppoller/httptest.c     2007-08-20 12:22:22.000000000 -0700
 +++ src/zabbix_server/httppoller/httptest.c.dp  2007-11-13 17:53:54.000000000 -0800
 @@ -318,6 +318,15 @@ static void        process_httptest(DB_HTTPTEST
 return;
 }

+       /* Process certs whose hostnames do not match the queried hostname. */
 +       if(CURLE_OK != (err = curl_easy_setopt(easyhandle,CURLOPT_SSL_VERIFYHOST , 0)))
 +       {
 +               zabbix_log(LOG_LEVEL_ERR, "Cannot set CURLOPT_SSL_VERIFYHOST [%s]",
 +                       curl_easy_strerror(err));
 +               (void)curl_easy_cleanup(easyhandle);
 +               return;
 +       }
 +
 lastfailedstep=0;
 httptest->time = 0;
 result = DBselect("select httpstepid,httptestid,no,name,url,timeout,posts,required,status_codes from httpstep where httptest
 id=" ZBX_FS_UI64 " order by no",

Link to Patch: libcurl disable ssl verifyhost

Instructions for installing patch:

Shut down zabbix_server process

/etc/init.d/zabbix_server stop

If you already have your zabbix source unpacked, you can skip the first tar step I’ve checked the patch with Zabbix 1.4.1 and 1.4.2

tar -zxvf zabbix-1.4.2.tar.gz

cd zabbix-1.4.2

wget http://www.muck.net/wp-content/uploads/2007/11/libcurl_disable_ssl_verifyhost.patch

patch src/zabbix_server/httppoller/httptest.c libcurl_ssl_verifyhost.patch

Then build zabbix_server as normal, for example:

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

Restart zabbix_server

/etc/init.d/zabbix_server start

Your Internal SSL Web Scenarios should now work! That was easy wasn’t it?

As always, I appreciate any feedback, and would love to hear if this helped you, or if you have any questions!

I’ve been saying I was going to do it for the last 6 months. I finally did it this weekend. I built a demo version of zabbix that I can give full admin rights to. If you’re not familiar with zabbix you can visit my zabbix page , or the zabbix website .

The problem with the old zabbix demo was that it was the only copy. If someone messed it up, I had to manually go in and restore the DB. It didn’t have a self recovery method. In the end, it meant that I was limited to handing out read only access in most cases. That in turn meant it wasn’t used very often by folks, and because it wasn’t used very often, I neglected it.

However! I have now put zabbix into its own VM universe. I took a snapshot of a “clean” zabbix installation, which is known to be good. Every 24 hours, I wipe out the VM instance, put a new one in place, restore the Zabbix DB, and timeshift all the data so it looks like the data is contiguous, even though in all likelyhood its several weeks old. The process results in about 10 minutes of downtime, and another 20 minutes of data processing in the background. Any data displayed from Midnight PST through Midnight PST the following night, is live, accurate, real data, not timeshifted data.

I’m pretty happy with it so far. I still want to add a lot more items and hosts to the demo, but I think its a great start. As far as I know this is the only Zabbix Demo, so I’d love to hear from you what you’d like to see in the demo. More graphs? Screens? Web Monitoring? oracle monitoring? mysql monitoring? Zabbix Tutorials? Walkthroughs? Clever uses for the Demo? Any and all comments appreciated.

Feel free to add new hosts, or even install a zabbix agent on your side and monitor your own server (until midnight that is, when the DB will be reset). I get a lot of hits on my site every day from people searching google for zabbix demo’s. Hopefully this will raise awareness of zabbix and get more people using this excellent tool.

Of course, I can also help you with your zabbix deployment. I can do a custom install for your environment, or I can ship you a pre configured zabbix box that you just plug in and go. If you’d like talk about my zabbix services, please indicate so on the Zabbix Demo form below. Hopefully the form you have to fill out to get access to the zabbix demo isn’t too painful. If you really dislike me for making you fill out a form to get the login credentials, feel free to call me names in every field except the email field. That one happens to be accurate in order to get the login credentials!

Stay tuned over the next several weeks for updates and new features.

cforms contact form by delicious:days

Installing from source on RHEL5 and CENTOS5 is quite painless. Fuse needs to compile a kernel module for your kernel. I started from a minimal install, and did the following:

Update 12/6/2007: There is a bug with more recent updates of RHEL5 (Similar or the same as this bug: https://bugzilla.redhat.com/show_bug.cgi?id=228430). The bug will cause the original “yum install” command to fail with the following:

Error: No Package Matching glibc.i686

To prevent that, install glibc first, then install the rest of the stuff you want:

yum install glibc

yum install kernel-devel gcc kernel-headers openssl gcc-c++ openssl-devel boost-devel

Update (2/5/2008): I added boost-devel to the install list, because of an error I encountered installing on CentOS 4.

checking boost/shared_ptr.hpp usability… no
checking boost/shared_ptr.hpp presence… no
checking for boost/shared_ptr.hpp… no
configure: error:
Can’t find boost/shared_.h – add the boost include dir to CPPFLAGS and
rerun configure, eg:
export CPPFLAGS=-I/usr/local/include

Download latest fuse (2.7.1 at this time 10/2007)

wget http://internap.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.1.tar.gz

Download latest rlog (1.3.7 at this time)

wget http://freshmeat.net/redir/rlog/47815/url_tgz/rlog-1.3.7.tgz

Download latest encFS (1.3.2-1 at this time)

wget http://freshmeat.net/redir/encfs/45687/url_tgz/encfs-1.3.2-1.tgz

Rlog:

tar -xvzf rlog-1.3.7.tgz

cd rlog-1.3.7

./configure

make

make install

Fuse: lather, rinse

tar -xvzf fuse-2.7.1.tar.gz

cd fuse-2.7.1

./configure

make

make install

encF: and repeat

tar -xzvf encfs-1.3.2-1.tgz

cd encfs-1.3.2

./configure

make

make install

Start fuse:

/etc/init.d/fuse start

Fix init script for CentOS, replace the startup information at the top with:

### BEGIN INIT INFO
# chkconfig: 2345 90 10
# description:       Load the fuse module and mount the fuse control
#       filesystem.
### END INIT INFO

Run:

chkconfig –add fuse

Create an Encrypted Filesystem (Its not really a filesystem… but I digress) as a test:

encfs /usb/disk1/.crypt-raw /usb/disk1/crypt-mount

It really is that easy. Good luck!

If you’re not familiar with linux or open source tools, finding all the dependencies, downloading the source, compiling source, creating the db, etc can be a daunting task. So I’ve created this cut and paste walk through to help you through those steps. Almost everything here is cut and paste, except for hostname, and password information You’ll need to provide those on your own. I’ve done my best to make this as accurate as possible. I hate walkthroughs that just aren’t accurate! CentOS was installed choosing zero options, with as base of an installation as it would let me. I used the 2.6.18-8 kernel. If you have any questions, or find any errors, please let me know. And of course as usual, if you find it helpful, also, please let me know

I wrote these instructions using 1.4.1 as the example, but theres no reason why 1.4.2 shouldn’t work the same way

Update, 11/4/2007: 1.4.2 seems to install its binaries under prefix/sbin instead of prefix/bin, which is different than 1.4.1 which was used for this document. I’ve also noticed that when copying and pasting from this guide some of the whitespace, apostrophies, and dashes (‘ – ) seem to get distorted upon pasting. Its correct in the source, but when its displayed something is munged up. When I figure out what it is, I’ll fix it. In the meantime if you get a syntax error, try retyping what I’ve put on this page instead of copying and pasting. And if you know why its happening, let me know!

Install all the necessary pieces. I started with a very base installation of CentOS 5.

yum -y install ntp php php-bcmath php-gd php-mysql httpd mysql gcc mysql-server mysql-devel net-snmp net-snmp-utils net-snmp-devel net-snmp-libs curl-devel mak

Start up the time server. its important for the time between your devices to be in sync.

/etc/init.d/ntpd start

Download fPing, and install it:

wget http://dag.wieers.com/rpm/packages/fping/fping-2.4-1.b2.2.el5.rf.i386.rpm

rpm -Uvh fping-2.4-1.b2.2.el5.rf.i386.rpm

chmod 7555 /usr/sbin/fping

Create Zabbix user.

useradd zabbix

Download zabbix and untar it.

wget http://superb-east.dl.sourceforge.net/sourceforge/zabbix/zabbix-1.4.1.tar.gz

tar -xzvf zabbix-1.4.1.tar.gz

Start MySQL, and change the root password.

/etc/init.d/mysqld start

/usr/bin/mysqladmin -u root password YourFancyNewRootPassword

Connect to the DB using your newly created root password. Create the zabbix DB, and assign a new user (zabbixmysqluser) with privileges to that DB. You may want to change “zabbixmysqlpassword” to something else. But it should not be the same as any other “critical” password because it will be stored plain text in a config file.

mysql -u root -p

mysql> CREATE DATABASE zabbix;

mysql> GRANT DROP,INDEX,CREATE,SELECT,INSERT,UPDATE,ALTER,DELETE ON zabbix.* TO zabbixmysqluser@localhost IDENTIFIED BY ‘zabbixmysqlpassword’;

mysql> quit;

Create the DB Schema

cd zabbix-1.4.1

cat create/schema/mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/data.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

cat create/data/images_mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix

./configure –enable-server –prefix=/usr/local/zabbix –with-mysql –with-net-snmp –with-libcurl

make install

make clean

Compile the agent. I chose to compile it staticly, so it can be copied easily to other hosts.

./configure –enable-agent –prefix=/usr/local/zabbix –enable-static

make install

Add the zabbix server and agent ports to your /etc/services file.

echo ‘zabbix_agent 10050/tcp’ >> /etc/services

echo ‘zabbix_trap 10051/tcp’ >> /etc/services

Copy the sample configs to /etc/zabbix for server and agentd.

mkdir /etc/zabbix

cp misc/conf/zabbix_agentd.conf /etc/zabbix

cp misc/conf/zabbix_server.conf /etc/zabbix

in /etc/zabbix/zabbix_server.conf, modify:

DBUser=zabbixmysqluser

BPassword=zabbixmysqlpassword

DBSocket=/var/lib/mysql/mysql.sock

FpingLocation=/usr/sbin/fping

in /etc/zabbix/zabbix_agentd.conf, modify:

Server=127.0.0.1,Your.Zabbix.Server.IP

Hostname=EnterAUniqueHostNameForEachAgent

cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
cp misc/init.d/redhat/zabbix_server_ctl /etc/init.d/zabbix_server

in /etc/init.d/zabbix_agentd AND /etc/init.d/zabbix_server:

BASEDIR=/usr/local/zabbix

in /etc/init.d/zabbix_agentd (Note the # hash marks, they are necessary), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Agentd

in /etc/init.d/zabbix_server (again, note the # Hash marks, they are required), add near the top, just below #!/bin/sh:

# chkconfig: 345 95 95
# description: Zabbix Server

Configure automatic starting and stopping of services.

chkconfig –level 345 zabbix_server on

chkconfig –level 345 zabbix_agentd on

chkconfig –level 345 httpd on

chkconfig –level 345 mysqld on

chkconfig –level 0123456 iptables off

/etc/init.d/iptables stop

Note: I turn the iptables firewall OFF because my box is behind a firewall. You should consult with your network folks before turning off the firewall. At the very least you should poke holes for port 80, 10050, and 10051 in the firewall.

cp -r frontends/php /var/www/html/zabbix

in /etc/php.ini, modify:

max_execution_time = 300

date.timezone = America/Los_Angeles

Note: Obviously you should substitute your own time zone. For a list of all valid timezones, click here

/etc/init.d/httpd start

chmod 777 /var/www/html/zabbix/conf

Launch http://your.servers.name/zabbix inyour browser. You should be prompted with a setup screen. Click through the user agreement, and when you get to the Pre requisites screen, make sure you have a green OK next to everything.

When you’ve finished walking through the web interface setup:

chmod 755 /var/www/html/zabbix/conf

mv /var/www/html/zabbix/setup.php /var/www/html/zabbix/setup.php.bak

/etc/init.d/zabbix_agentd start

/etc/init.d/zabbix_server start

You can now login to your zabbix installation by going to http://your.server.name/zabbix, using the username “admin”, with no password. To monitor your zabbix server, you can go to the Configuration Tab, and choose the “hosts” sub Tab. Select the “Zabbix Server” host, by putting a checkmark next to it. and choose the “Activate Selected” button below. Wait a minute or two, then select the “Monitoring” tab, and then the “latest data” sub tab. You should start seeing performance stats appear!

For Reference, your binaries are under /usr/local/zabbix/bin, and your configuration files are in /etc/zabbix.

I’m not a big fan of their default template, I think the naming sucks. Look for a future article talking about renaming zabbix items. But this should be enough to get you started! You can find the answers to most of your questions in the Zabbix manual, available here: http://www.zabbix.com/documentation.php . You can also find lots of answers in the zabbix forums .

I like zabbix… but it annoys me that it’s ops per second, and bytes per second data is broken for hard drives in the linux 2.6 kernel. So I created a work around

Add the following code to your zabbix_agentd.conf file (/etc/zabbix/zabbix_agentd.conf by default), and restart the zabbix agent:

UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$4}’
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$7}’
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$8}’
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$11}’
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$12}’
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$13}’
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$6}’
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | grep $1 | head -1 | awk ‘{print $$10}’

The names are pretty descriptive, but you can find more information on how to decipher /proc/diskstats here: http://www.kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/iostats.txt

After you add the UserParameter’s to your config file, you can check that they work by using zabbix_get from your zabbix server:

zabbix_get -s HostName -p 10050 -k custom.vfs.dev.write.ops[sdb]

It should return a large number, and not something like: ZBX_NOTSUPPORTED. If it comes back not supported, make sure you restarted the agent after adding the userparameters. Then try running the commands outside of zabbix. Note that if you run the commands outsie of zabbix, you’ll need to replace the first $1 with a disk name, and remove one of the $ characters from the awk command.

Once you’ve verfied everything is in working order, you can add it as an item to your zabbix installation:

You can download my exported Template here , which will create a template called Template_app_FileServer. It will give you:

Disk Read/Write: Bytes/sec (or KB, MB per second)

Disk Read/Write: Ops/second

Disk Read/Write: Milliseconds spent reading or writing

Disk:IO currently executing

Disk:IO ms: time spent performing IO

If you’d rather just create your own items, and not bother loading mine, I have attached some screenshots of the config screens, so you can re create it with the right settings:

As always, if you find this page helpful or have any questions, please let me know!

If while installing oracle DB on linux, if you get the following two messages:

ORA-12154: TNS:could not resolve connect identifier specified

ORA-0107: invalid username/password; logon denied

It probably means that you have chosen A) to create a DB while installing oracle, and B) chosen not to read the full set of installation instructions(10gR2 on x64 Linux) available on oracles website. I was guilty of both a couple week ago, and when searching for those error messages together I didn’t find anything. I then went back and read through the entire documentation and found the following password recommendations tidbit, I had previously skipped over thinking “pshah! I know how to create a good password! I don’t need oracle telling ME how to create good passwords”. Oops.

Here is a link to oracles 10gR2 tech reference site, which contains lots of good reading material.

Here is a quote from the relevent section of the installation guide, but please read the whole thing.

Password recommendations

Apply the following guidelines when specifying passwords:

• Passwords must be between 4 and 30 characters long.
• Passwords must be from the database character set and can include the underscore (_), dollar ($), and pound sign (#) characters.
• Passwords must not start with a numeral.
• Passwords must not be the same as the user name.
• Passwords must not be Oracle reserved words.
• The SYS account password must not be change_on_install.
• The SYSTEM account password must not be manager.
• The SYSMAN account password must not be sysman.
• The DBSNMP account password must not be dbsnmp.
• If you choose to use the same password for all the accounts, then that password must not be change_on_install, manager, sysman, or dbsnmp.
• Passwords should have at least 1 alphabetic, 1 numeric, and 1 punctuation mark character
• Passwords should not be simple or obvious words, such as welcome, account, database, and user.

P.S. This problem is not unique to the installation of Oracle, but when I searched google I found several people complaining about this exact problem during installation, with no resolution. I’m hoping that an installation focused article will make its way to the top of the search results for people to find. If you are getting the above error messages when you run dbca, you have the same problem, since the install app, just calls dbca.

If you find this post useful, tell me! Thanks!

While troubleshooting a VOIP problem in one of my clients office environment, I decided I would install mrtg , allowing me to be up and running in 5 minutes, monitoring their network traffic.

Turns out it wasn’t that easy. Sure enough I got MRTG installed and reading data from the cisco ASA 5505 in short order. But shortly thereafter I noticed that the data just didn’t look right. All of the interfaces had almost identical data. The ifInOctets, and ifOutOctets were darn near the same across all three interfaces. Looking at the interface statistics from the CLI confirmed the strange numbers I was getting via SNMP were simply not valid.

Turns out there is a Bug (CSCsi15805) in the 7.2 OS which causes the SNMP iftable to return statistics for the total number of untagged vlan packets, instead of per vlan statistics.

The text of the bug is as follows:

Symptom:
ifTable byte and packet counters are incorrect.

Conditions:
SNMP query on 5505

Workaround:
Use “show interface” CLI command.

Further Problem Description:
SNMP doesn’t properly recognize the EtherSVI and a logical subinterface, so returns aggregate physical inteface stats.

The Fix? Upgrade to ASA OS 7.2.3 or Interim Release 7.2.2.22. This is the first time I’ve ever used the SOHO version of the PIX/ASA, and so far I’m not impressed. I’ve already run into at least two disappointing features/bugs. Given the choice I’d buy a used 515 or 515E.

2 hours later, I’m where I thought I would be an hour and 55 minutes ago.

Cheers! If you find this useful, please let me know!