Monitor DRBD devices with nagios

install

on all ganeti nodes

just install check_drbd (http://www.monitoringexchange.org/p_w_upload/download/Check-Plugins/Operating-Systems/Linux/check_drbd/check_drbd) somewhere in the path.  

By default, this script will return CRITICAL if a device is unconfigured, and UNKNOWN if a device does not exists. These situations can happen with ganeti when :

- a vm is down  

- a vm is down and a node has been rebooted (then the drbd device disappears).  

You may want to patch check_drbd like this to avoid "false" alerts :

--- check_drbd  2010-01-2609:10:16.000000000+0100
+++/usr/local/bin/check_drbd   2010-01-2609:06:23.000000000+0100
@@-38,7+38,7@@
'WFConnection'=>{'value'=>'CRITICAL','type'=>'cs'},
'WFReportParams'=>{'value'=>'CRITICAL','type'=>'cs'},
'Connected'=>{'value'=>'OK','type'=>'cs'},
-'Unconfigured'=>{'value'=>'CRITICAL','type'=>'cs'},
+'Unconfigured'=>{'value'=>'OK','type'=>'cs'},
# DRBD 0.6
'SyncingAll'=>{'value'=>'WARNING','type'=>'cs'},
'SyncingQuick'=>{'value'=>'WARNING','type'=>'cs'},
@@-261,7+261,7@@
}
foreachmy $device (@devices){
if(!(defined($cs{$device}))){
-&myexit('UNKNOWN',"Could not find device $device");
+&myexit('OK',"Could not find device $device");
}
                $check{$device}=1;
}

on ganeti master

put this script somewhere in the path (/usr/local/bin/drbd_gen_nagios.sh for ex):  

#!/bin/bash 
# 
# Copyright (C) 2009 Maxence Dunnewind <maxence@dunnewind.net> 
# 
# This program is free software: you can redistribute it and/or modify 
# it under the terms of the GNU General Public License as published by 
# the Free Software Foundation, either version 3 of the License, or 
# (at your option) any later version. 
# 
# This program is distributed in the hope that it will be useful, 
# but WITHOUT ANY WARRANTY; without even the implied warranty of 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the 
# GNU General Public License for more details. 
# 
# You should have received a copy of the GNU General Public License 
# along with this program.  If not, see <http://www.gnu.org/licenses/>. 

gnt-instance list  --no-header --separator=:-o name|while read vm;do
        TMP_FILE=$(mktemp)
        gnt-instance info -s $vm > $TMP_FILE 
        NODE_A=$(grep "nodeA" $TMP_FILE|grep -o "[^ ]\+,"|head -1|sed 
's/,//')
        NODE_B=$(grep "nodeB" $TMP_FILE|grep -o "[^ ]\+,"|head -1|sed 
's/,//')
        DEV_A=$(grep "nodeA" $TMP_FILE|cut -d "="-f 2)
        DEV_B=$(grep "nodeB" $TMP_FILE|cut -d "="-f 2)
        rm -rf $TMP_FILE 
for disk in $DEV_A;do
                echo "define service {"
                echo " host_name                $NODE_A"
                echo " service_description      DRBD instance $vm / 
device n° $disk"
                echo " check_command            check_drbd!$disk"
                echo " use                      generic-service"
                echo "}"
                echo ""
done
for disk in $DEV_B;do
                echo "define service {"
                echo " host_name                $NODE_B"
                echo " service_description      DRBD instance $vm / 
device n° $disk"
                echo " check_command            check_drbd!$disk"
                echo " use                      generic-service"
                echo "}"
                echo ""
done
done

Important : This script only define drbd checks, so you already need  to have the hosts configured in your nagios.  

on nagios

Define check_drbd, which will use check_by_ssh to call check_drbd  script on the nodes.  

define command{
        command_name    check_drbd 
        command_line    /usr/lib/nagios/plugins/check_by_ssh -H $HOSTADDRESS$ -l root -C "/usr/local/bin/check_drbd -d $ARG1$"
}

for example in /etc/nagiosX/conf.d/command.cfg .

Also, be sure nagios user can connect using ssh without password. If you don't have defined that, you need to :

# su - nagios
# ssh-keygen
Generatingpublic/private rsa key pair.
Enter file in which to save the key (/var/run/nagios3/.ssh/id_rsa):/etc/nagios3/id_rsa
Enter passphrase (empty forno passphrase):
Enter same passphrase again:

Don't forget to replace the file. If you keep /var/run, your key will be deleted after next reboot :) Also, keep the passphrase empty. Then, copy the id_rsa.pub under /root/.ssh/authorized_keys on all your hosts. You can also add a "from" field and a "command" one to limit access :  

from="1.1.1.1",command="/usr/local/bin/check_drbd" ssh-rsa AAAAB3NzaC1yc2EAAAABIwA...

Use

To generate the config, after a new instance has been added / removed, or after a replace-disks, run :  

drbd_gen_nagios.sh > drbd.cfg 

on ganeti master, then just copy the generated file into your nagios  conf.d directory, and check the config / restart your nagios.