Local checks - how to write your own checks

1. The principle of local checks

Local checks are an easy way to write your own checks with Check_MK. You do not need to change or extend Check_MK itself, you just need to write simple check scripts in the programming language of your choice that directly run on the host being checked. Since version 1.1.7i3 all agents support local checks.

2. How to write local checks

Writing local checks is done by putting executable programs (usually shell scripts or similar) into a specific directory which is scanned by the agent. The default directory for such local checks is /usr/lib/check_mk_agent/local for Linux/UNIX agents - on your target hosts.

The Windows agents looks in the subdirectory local of where the file check_mk_agent.exe is installed. If you are unsure, you can see the path in the first view lines of the agent output:

root@linux# check_mk -d winhost | head
<<<check_mk>>>
Version: 1.1.7i2
AgentOS: windows
WorkingDirectory: C:\WINDOWS\system32
AgentDirectory: C:\check_mk
PluginsDirectory: C:\check_mk\plugins
LocalDirectory: C:\check_mk\local
<<<df>>>
C:\        NTFS     4184900 4083036 101864  98% C:\
<<<ps>>>
close failed: [Errno 32] Broken pipe

On Windows you have to manually create the local directory.

The agent simply executes every executable file in that directory and prints its output in the section <<<local>>>. Each local check may output one or more lines with four columns of text:

StatusThe check result in Nagios convention: 0 for OK, 1 for WARNING, 2 for CRITICAL and 3 for UNKNOWN
Item nameA name for the check item that is unique within the host. It will be used as Nagios' service description. It must not contain spaces, because spaces separate the four columns.
Performance dataYou may output zero, one or more performance variables here with the standard Nagios convention. That is varname=value;warn;crit;min;max, while the values ;warn;crit;min;max are optional values. If you like to provide only the min and max values, you can provide a definition like varname=value;warn;crit. If you have no performance data simply write a dash (-) instead. From version 1.1.5 on, it is possible to output more then one variable. Separate your variables with a pipe symbol (|). Do not used spaces.
Check outputThe last column is the output of the check. It will be displayed in Nagios. Spaces are allowed here and only here.

This is an example output of a check script counting the number of files in certain directories:

2 Filecount_/var/log count=102 CRITICAL - 102 files in /var/log
0 Filecount_/tmp count=7 OK - 7 files in /tmp

And this is a possible implementation of the upper check:

/usr/lib/check_mk_agent/local/filecount
#!/bin/bash
DIRS="/var/log /tmp"

for dir in $DIRS
do
    count=$(ls $dir | wc --lines)
    if [ $count -lt 50 ] ; then
        status=0
        statustxt=OK
    elif [ $count -lt 100 ] ; then
        status=1
        statustxt=WARNING
    else
        status=2
        statustxt=CRITICAL
    fi
    echo "$status Filecount_$dir count=$count;50;100;0; $statustxt - $count files in $dir"
done

3. Inventory - Integration into Nagios

The integration of the local checks into Nagios is easy: Simply do an Inventory of the host - either with all checks (cmk -I HOSTNAME) or only with the local checks (check_mk -I --checks local HOSTNAME). All (new) local checks will be found, Nagios services will be created for you. The performance data will be collected and processed just like that of the other checks.

4. PNP Templates for local checks

Note for OMD users: The setup will work out of the box as described below. If you want to have custom graph templates, read on.

PNP4Nagios uses the name of the check command for selecting a template for rendering the RRD graph. Since all local checks have the same command check_mk-local, the template check_mk-local.php will be used for all cases.

Since version 1.0.37, Check_mk ships a wrapper template with that name, that tries to find a service specific template file matching a prefix of the service description. Let's assume the service description is LDAP auth/4. The wrapper then first tries to find the file LDAP auth_4.php (slashes are replaced with underscores). If that file is present, it is used as template. Otherwise the file LDAP auth_.php will be tried, then LDAP auth.php, LDAP aut.php and so on until L.php. If none of those files is present, then PNP4Nagios' default.php will be loaded and will generate a graph based on your warn/crit levels.

All those files are expected to be in the same directory as check_mk-local.php.

5. Multiple performance values

As of version 1.1.5 it is possible to send more than one variable in a check. Simply seperated your variable definitions by a pipe symbol (do not use spaces). Here is an example output of a check sending three performance values:

0 Nagios_Status ok=33012|crit=17|warn=120|unkw=16 Services ok/cr/wa/un: 33012/17/120/16

6. Cached local checks (new in 1.2.3i1 in the Linux agent)

Sometimes a script will run for longer than a few seconds. If the run time of all script and plugins of an agent exceed the timeout for active checks of the monitoring core (usually 60 or 120 seconds), then the complete check will be aborted. In order to avoid this you can have local checks be run asynchronously and use cache files. This is done by putting your script into a subdirectory that is named by a number - the number of seconds that the output of the script is valid:

/usr/lib/check_mk_agent/local/300/myscript

In this case the agent will:

  • Run this script in the background and wait not for it to finish.
  • Store the result of the script in a cache file below /var/lib/check_mk_agent/cache.
  • Use that file for 300 seconds before running the script again.

Werk #0016

Linux+Windows agent: allow spooling plugin outputs via files

Werk #0017

local: New state type P for state computation based on perfdata

7. Limitations of local checks

Local checks are easy to setup and have many other advantages. A few limitations are there, however:

  • On Windows the output must only contain ASCII characters. UTF-8 and Unicode is currently not allowed.
  • In versions before 1.1.5, local checks can send at maximum one performance variable.
  • WARNING and CRITICAL levels have to be configured and handled on the host itself.