Creating Perf-O-Meters

What are Perf-O-Meters?

Perf-O-Meters are small elements in the Multisite GUI. The idea is to render a small graphic like area which represents the current status of a service and makes the output comparable to other services of the same type without the need to compare the textual output.

Each service which produces performance data can have a Perf-O-Meter.

How to create a Perf-O-Meter

The Perf-O-Meters are stored in the directory /share/check_mk/web/plugins/perfometer. In OMD the path for custom Perf-O-Meters is local/share/check_mk/web/plugins/perfometer.

All files named *.py are read and used by Multisite.

In the following tutorial the single parts of the Perf-O-Meter will be explained using a static hello world exaple. After the basics are clear we will go deeper.

The Perf-O-Meter function

All starts with the Perf-O-Meter function: It is a simple python function that is being called by Multisite with three parameters:

• row: The data of the current row (service). It contains a python dictionary containing information about the service the Perf-O-Meter is about. That data is not needed in most cases but can be useful if you need context information not contained in the performance data.
• check_command: the real check_command as used in Nagios given as python string.
• perf_data: the performance data as python list. Each perfdata set is given as tuple.

The python function should be named perfometer_<command>. For example perfometer_snmp_uptime for the snmp_uptime check.

The Perf-O-Meter function has to return a pair of two components: first the text to show in the middle of the Perf-O-Meter and second the HTML code of the actual Perf-O-Meter. In the most cases the Perf-O-Meter HTML code is a table element which has a width of 100%. The cells are rendered with a dynamic width and dynamic colors to visualize the state of the Perf-O-Meter.

To start with the Perf-O-Meter create a file called snmp_uptime.py in the Perf-O-Meter directory and add the following code:

def perfometer_check_mk_uptime(row, check_command, perf_data):
    return 'Hello World! :-)', '<table><tr>' \
                               + perfometer_td(20, '#fff') \
                               + perfometer_td(80, '#ff0000') \
                               + '</tr></table>'

The snipet above prints Hello World! :-) in the Perf-O-Meter and adds a Perf-O-Meter with a white area of 20% and a red area of 80%. The perfometer_td function used in the example renders a simple table cell element (<td>) with the needed attributes like color and width. You could also simply write the plain HTML code instead of using that helper function.

Registering a Perf-O-Meter

Each Perf-O-Meter must be registered in multisite to make it useable. In order to register a Perf-O-Meter its function must be added to the perfometers dictionary using the check_command as key.

To register the snmp_uptime Perf-O-Meter the following snippet is needed:

perfometers['check_mk-snmp_uptime'] = perfometer_check_mk_uptime

Now you need to restart the webserver process to ensure that there is no cache which prevents loading the new Perf-O-Meter. After the restart you should see the new Perf-O-Meter in your multisite GUI for each service which uses the snmp_uptime check. Here a screenshot of the example:

Debugging in Perf-O-Meters

At the start of the topic I mentioned the parameters of the Perf-O-Meter functions. Until now we left them blackboxed, but now we will dig into the secrets of debugging Perf-O-Meters. The task: Check out the contents of the single parameters!

So how to do it? Yes, you could write some magic python code which outputs the contents of the vars to a file, but that would not help in all cases. So we try a way to bring debug outputs from the Perf-O-Meter to the multisite GUI.

It's easy! Just replace the hello world Perf-O-Meter function with this snippet:

def perfometer_check_mk_uptime(row, check_command, perf_data):
    return repr(check_command), ''

Now restart the webserver and open a service using that Perf-O-Meter. After that you should see the text check_mk-snmp_uptime right in the middle of the Perf-O-Meter. Yes, easy!

Ok, now let's take a look at the contents of the perf_data variable. Again replace the Perf-O-Meter function, restart the webserver and open a snmp_uptime service in Multisite.

def perfometer_check_mk_uptime(row, check_command, perf_data):
    return repr(perf_data), ''

The Perf-O-Meter should show a text like this: [(u'uptime', u'3962', u'', u'', u'', u'', u'')]. This is a python list with a tuple of seven elements where each tuple is a perfdata set. The elements of the tuple are filled as follows: perfdata label, current value, uom, warn, crit, min, max.

Render/Helper functions

Check_MK comes with a set of default render functions and other helper functions which can be used to generate the HTML code of the Perf-O-Meters.

The single functions will be described in detail below. First a list of all shipped helper functions:

Function	Description
perfometer_td	returns the HTML code of a single colored table cell corresponding to a percentage.
perfometer_linear	returns the HTML code of a complete Perf-O-Meter with two cells, one white and one in a given color and width (which is given as a percentage from from 0 to 100.
perfometer_logarithmic	This is similar to perfometer_linear but uses a logarithmic scale. This can be used if there is no natural upper limit (such as percentage). The scaling is configured using the logarithm of the given half_value and the given base parameter

perfometer_td

We used that function in our hello world example before. Just another simple example:

perfometer_td(20, '#000000')

This renders a black colored <td> element with a width of 20%.

perfometer_linear

This function reanders a HTML table having two cells. The table has a fixed width of 100%. The first cell can have a variable width given as % and a custom color given as HTML color definition.

For example this Perf-O-Meter render function can be used to render percentage usages of e.g. cpu utilisations.

perfometer_linear(65, '#00BB33')

This renders a green area with a width of 65% which might represent a CPU utilization of 65%.

Let's take an example: We use the printer_supply check for this example. How to start now? At first go to the Multsite GUI and take a look at a service using the printer_supply check. Now you should see an empty cell in the Perf-O-Meter column - yes, that's why we create one now!

First create and register the Perf-O-Meter function for sniffing the contents of the perf_data value as explained above. Mine looks as follows:

def perfometer_check_mk_printer_supply(row, check_command, perf_data):
    return repr(perf_data), ''

perfometers["check_mk-printer_supply"] = perfometer_check_mk_printer_supply

Now we get [(u'pages', u'40.0', u'', u'20.0', u'10.0', u'0', u'100.0')] as output for the yellow toner cartridge. This tells me I have 40% left, the service would fire a WARNING state on 20% anda CRITICAL state on 10% left.

My target is to create a Perf-O-Meter which shows a green area when the value is above 20%, a yellow area to 10% and a red area to 0%. So I replace the debug function created above with the following:

def perfometer_check_mk_printer_supply(row, check_command, perf_data):
    left = float(perf_data[0][1])
    warn = float(perf_data[0][3])
    crit = float(perf_data[0][4])
    if left > warn:
        color = "#00ff00"
    elif left > crit:
        color = "#ffff00"
    else:
        color = "#ff0000"

    return "%.0f%%" % left, perfometer_linear(left, color)

perfometers["check_mk-printer_supply"] = perfometer_check_mk_printer_supply

This results in e.g. this Perf-O-Meter when the value drops to 20%:

One note: The world is more complicated as shown in this example, for example some sevices of the printer_supply check have a higher maximum value than 100. This means the Perf-O-Meter needs to calculate the percentage value. Another nice feature of such a Perf-O-Meter would be to have the Perf-O-Meters of the different toner colors shown in their colors. But the code above should be enough for this example.

perfometer_logarithmic

This render function is useful when a check outputs perfdata which has no real maximum but a "common" scale value can be assumed and a wide range of possible values. An example is the load of a linux system. The load can be 0.01, 20.00 or even more. It's not possible to paint such a scale in a width 100% since there is no upper limit.

Lets take a look at the Perf-O-Meter function for the cpu.load shipped with multisite:

def perfometer_check_mk_cpu_loads(row, check_command, perf_data):
    color = { 0: "#68f", 1: "#ff2", 2: "#f22", 3: "#fa2" }[row["service_state"]]
    load = float(perf_data[0][1])
    return "%.1f" % load, perfometer_logarithmic(load, 4, 2, color)

perfometers["check_mk-cpu.loads"] = perfometer_check_mk_cpu_loads

First said: The Perf-O-Meter only handles the 1 minute load (perf_data[0]).

The first line fetches a color using the service_state as reported by Nagios. The second line converts the current value to a float. The third (and most interesting) line executes the perfometer_logarithmic function.

That function is called with the following parameters:

perfometer_logarithmic(value, half_value, base, color)

The first and fourh parameter should be clear: The value parameter takes the current value and the color parameter takes the color to be rendered.

The half_value and base parameters are more advanced. In short the 100% are scaled logarithmic using the half_value and base parameters. Thi

By example: The cpu.load perfometer uses 4 as half_value and 2 as base. This leads to the following scale:

• With a load of 0.20 the colored area has a width of 6%
• With a load of 1.00 the colored area has a width of 30%
• With a load of 2.00 the colored area has a width of 40%
• With a load of 4.00 the colored area has a width of 50%
• With a load of 8.00 the colored area has a width of 60%
• With a load of 16.00 the colored area has a width of 70%
• With a load of 32.00 the colored area has a width of 80%
• With a load of 64.00 the colored area has a width of 90%