web analytics

Monitoring 3ware RAID using Nagios

Monitoring 3ware RAID using Nagios

RAID (Redundant Array of Inexpensive Disks) is used for storing the same data in multiple places, which provides a redundant and reliable means of data storage. While you improve the security of your data using RAID, what will happen if something happens to the RAID array itself? Its always better to have a monitoring in place for the RAID you have setup, so that it will alert you even when a minor problem is noticed. Here I am explaining the steps to monitor a 3ware RAID using the monitoring tool, NAGIOS.

We are using a command line tool called tw_cli for monitoring 3ware RAID arrays. The tw_cli needs to be installed in the remote host, the host being monitored. I am calling the remote host with RAID as ‘host’ and the server where nagios is installed as ‘nagios’.

1) First check whether the remote machine is 32 bit or 64 bit.
[root@host ]# arch
x86_64

Here it is a 64 bit system. Now download the appropriate tw_cli client. It can be downloaded from the following links.

For 64 bit system
———————
http://www.3ware.com/download/Escalade9690SA-Series/9.5.1.1/tw_cli-linux-x86_64-9.5.1.1.tgz

For 32 bit system
———————

2) Download and install tw_cli to the remote host.

[root@host ~]#cd /usr/local/sbin/

[root@host sbin]#wget http://203.98.75.4/tw_cli-linux-x86_64-9.5.1.1.tgz

[root@host sbin]#tar -xzvf tw_cli-linux-x86_64-9.5.1.1.tgz

[root@host sbin]#chmod 755 tw_cli

3) You have now downloaded the required version of tw_cli, just confirm if it matches your architecture.
[root@host sbin]# file tw_cli
tw_cli: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, statically linked, for GNU/Linux 2.4.0, stripped

It displays the result as 64-bit LSB executable, hence proceed with the setup.

4) On the remote host, setup a user nagios, for the nagios server to connect to.

[root@host sbin]#useradd nagios

[root@host sbin]#passwd nagios  (Make sure to give a strong password as many hackers utilize this user to hack into the server)

5) Now put the following script under  the home of nagios user. Basically, the script check3ware.sh is a bash script used along with nagios to monitor the raid arrays.

[root@host sbin]#cd /home/nagios/

[root@host nagios]#touch check3ware.sh

[root@host nagios]#chmod 755 check3ware.sh   (Make the script executable)

[root@host nagios]#chown nagios:nagios check3ware.sh

[root@host nagios]#vi check3ware.sh

Now press ‘I’ and insert the following piece of code into it.

#!/bin/bash
check()
{
res=$(sudo /usr/local/sbin/tw_cli /c0 show|grep -v OK|grep ^p|awk ‘{ print $1 ” ” $2 ” “}’|tr -d ‘n’);
}
check
if [ -z “$res” ]
then
echo OK;
exit 0;

elif [ -n “$res” ]
then
echo $res;
exit 2;
fi

Save and close the file [Esc + :wq]

6) Now setup nagios as a sudo user so as to be able to execute the script when called from the nagios server.

First, take a backup of the current sudoers file and then execute visudo.

[root@host nagios]#cp -pv /etc/sudoers  /etc/sudoers.bak.$(date +%F)

[root@host nagios]# visudo

Insert the following line into that file.

nagios remotehost=NOPASSWD: /usr/local/sbin/tw_cli

Make sure to rename ‘remotehost’ with your hostname that is being monitored. Also, find out the following line and comment it out.

Defaults requiretty

7) Nagios server connects to the host server via ssh for checking the health of the RAID array. Each time it connects, it shouldn’t ask for password. Hence we setup SSH keys between the nagios server and the remote host to be monitored. Create a folder .ssh under /home/nagios/

[root@host nagios]#su – nagios

-bash-3.2$mkdir .ssh

-bash-3.2$touch .ssh/authorized_keys

-bash-3.2$chmod og-rxw -R .ssh

8) The key has to be created from the nagios server. Login to the nagios server and execute the following.

[root@nagios ~]#su – nagios

-bash-3.2$mkdir .ssh

-bash-3.2$ssh-keygen -t dsa

Simply press enter for anything asked. The key will be saved in /home/nagios/.ssh/id_dsa.pub

Copy the contents of /home/nagios/.ssh/id_dsa.pub from nagios server to /home/nagios/.ssh/authorized_keys

[root@nagios ~]#scp /home/nagios/.ssh/id_dsa.pub remotehost:/home/nagios/.ssh

9) The content copied above will now be present as /home/nagios/.ssh/id_dsa.pub of remote host. Hence, come back to the host server and execute the following.

[root@host ~]#cd /home/nagios/.ssh

[root@host ~]#mv id_dsa.pub authorized_keys

10) SSH key has now been setup between the nagios server and the remote host. To check if it is working correctly, try logging in to the remote host from the nagios server as user ‘nagios’. If you are able to login without being asked for a password, the key is working and you can proceed with the monitoring setup.

11) Next step is to setup monitoring in the nagios server. Login to nagios server and perform the following.

[root@nagios ~]#cd /usr/local/nagios/etc/

a) Define a command for checking raid.

[root@nagios ~]#vi checkcommands.cfg

Press ‘I’ and Enter the following in it. This command will be executed for checking the RAID health.
We are giving the command name as check_3ware

define command{
command_name check_3ware
command_line $USER1$/check_by_ssh -i /home/nagios/.ssh/id_dsa -l nagios -H $HOSTADDRESS$ -C /home/nagios/check3ware.sh 2>/dev/null
}

Save and exit the file. [Esc + :wq]

b) Define a hostgroup

[root@nagios ~]#vi hostgroups.cfg

define hostgroup{
hostgroup_name check_3ware_raid
alias 3ware_raid
contact_groups contacts
members remotehost
}

Here, hostgroupname is set as check_3ware_raid(anything you like) and make sure to replace remote host in members section with your actual hostname that needs monitoring.

c) Define the service as follows.

[root@nagios ~]#vi services.cfg

define service{
use generic-service
host_name remotehost
hostgroup_name check_3ware_raid
service_description check_3ware
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups contacts
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_3ware
}

Make sure to rename remotehost with the hostname being monitored.

12) With this, the nagios configuration part is over. Proceed to check if there are any errors in the configuration by issuing the following command.

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If there is any error you have to correct that until you get something like this,
————————————————————-
Total Warnings: 0
Total Errors: 0
Things look okay – No serious problems were detected during the pre-flight check
————————————————————

13) Your nagios interface in the web browser should now have your RAID monitoring displayed ‘OK’.
Before checking the interface, you can check if monitoring is working from the backend itself. In the nagios server, execute

[root@nagios ~]#su – nagios

-bash-3.2$/usr/local/nagios/libexec/check_by_ssh -i /home/nagios/.ssh/id_dsa -H hostIP -l nagios -C ~/check3ware.sh

You should replace ‘hostIP’ in the above command with the IP Address of the host that needs to be monitored.

If you see it as OK, everything is working fine as desired. Now, restart nagios.

[root@nagios ~]#/etc/rc.d/init.d/nagios stop
[root@nagios ~]#/etc/rc.d/init.d/nagios start

RAID monitoring has been setup now, and you can check your nagios interface to confirm it.

One comment to Monitoring 3ware RAID using Nagios

  • Peter  says:

    Hi Divz,

    i’m looking for the check_3ware.exe plugin for Windows. The file on monitoringexchange.org is 0 Kb and i can’t get them to post a usable download. If you or any of your readers has it and is willing to share, it would be much appreciated.

    Thanks!

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>