Tag Archives: debian

Logging nagios remote commands

Quick trick, I needed it to debug execution of remote nagios commands.

Just drop this file into /var/nagios/.bashrc, assuming your local nagios user is configured to use the /bin/bash shell:

#!/bin/bash
#
# Log every command run by the nagios user
# into /var/log/auth.log (at least on Debian and derivatives)
#
trap 'logger -p auth.info -t nagios "Running $BASH_COMMAND"' DEBUG

The trap function executes a given command or list of commands when the list of signals specified as arguments are raised,
as in:

trap [COMMAND] [SIGNALS]

The DEBUG signal is special: it will fire every time a command is executed. Using logger ensures that whatever command the nagios user is trying to execute will be logged.

Last bit, how do you get the command text? It’s available in the $BASH_COMMAND variable.

Here’s an extract of the resulting log information:

Mar 30 10:48:05 big1 nagios: Running /usr/lib/nagios/plugins/check_cpu -i 5 -w 90 -c 98
Mar 30 10:49:42 big1 nagios: Running /usr/lib/nagios/plugins/check_tcp -p 3306
Mar 30 10:49:42 big1 nagios: Running /var/nagios/libexec/check_load -w40,40,40 -c50,50,50
Mar 30 10:50:26 big1 nagios: Running /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /usr/sbin/cron
Mar 30 10:50:44 big1 nagios: Running /usr/lib/nagios/plugins/check_disk -w 20 -c 10 -r "^/(ssd|store[1-3])?$"
...

To learn more about traps, here’s a web search on “bash traps”.

How to start up varnish with a custom cc_command on Debian

If you need to compile your varnish VCL file with custom options, maybe because of libraries like GeoIP, and you're running Debian, you can not use the init script that's shipped by default.

It will not work because of how shell expansion works in the start-stop-daemon command contained in the init script. I wrote my explanation and a proposed fix in much more detail in this stack overflow question:

http://stackoverflow.com/questions/5906603/varnish-daemon-opts-options-errors/8333333#8333333

TL;DR: (+ quick & dirty fix) patch your init script like this:

 start_varnishd() {
     log_daemon_msg "Starting $DESC" "$NAME"
     output=$(/bin/tempfile -s.varnish)
-    if start-stop-daemon 
-       --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- 
-       -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
+    if bash -c "start-stop-daemon 
+        --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- 
+        -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then
         log_end_msg 0
     else
         log_end_msg 1
         cat $output
         exit 1
     fi
     rm $output
 }

Let me know if it works for you!

EDIT (7/Mar/2012): bug was filed in Debian as #659005. Nothing happened so far. We'll see.

How to detect the Debian version of a server without logging in

As Ops team, we're slowly taking over operations for several other teams here at Opera. One of our first tasks is to:

First idea to check whether a server is Debian Lenny or Squeeze was to login and cat /etc/debian_version. However, if you haven't accessed that machine before, and your ssh keys are not there, you can't do that. In our case, we have to file a request for it, and it can take time. Wondering if there was a quicker way, I came up with this trick:

#!/bin/sh
#
# Tells the Debian version reading the OpenSSH banner
# Requires OpenSSH to be running and ssh port to be open.
#
# Usage: $0 <hostname>
#
# Cosimo, 23/11/2011

HOST=$1

if [ "x$HOST" = "x" ]; then
    echo "Usage: $0 <hostname>"
fi

OPENSSH_BANNER=$(echo "n" | nc ${HOST} 22 | head -1)

#echo "OPENSSH_BANNER=$OPENSSH_BANNER"

IS_SQUEEZE=$(echo $OPENSSH_BANNER | egrep '^SSH-.*OpenSSH_5.*Debian-6')
IS_LENNY=$(echo $OPENSSH_BANNER   | egrep '^SSH-.*OpenSSH_5.*Debian-5')
IS_ETCH=$(echo $OPENSSH_BANNER    | egrep '^SSH-.*OpenSSH_4.*Debian-9')

# SSH-2.0-OpenSSH_5.1p1 Debian-5
# SSH-2.0-OpenSSH_4.3p2 Debian-9etch3
# SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze1

#echo "Squeeze: $IS_SQUEEZE"
#echo "Lenny: $IS_LENNY"
#echo "Etch: $IS_ETCH"

if [ "x$IS_SQUEEZE" != "x" ]; then
    echo "$HOST is Debian 6.x (squeeze)"
    exit 0
fi

if [ "x$IS_LENNY" != "x" ]; then
    echo "$HOST is Debian 5.x (lenny)"
    exit 0
fi

if [ "x$IS_ETCH" != "x" ]; then
    echo "$HOST is Debian 4.x (etch)"
    exit 0
fi

echo "I don't know what $HOST is."
echo "Here's the openssh banner: '$OPENSSH_BANNER'"

exit 1

It reads the OpenSSH server banner to determine the major Debian version (Etch, Lenny, Squeeze). It's really fast, it's very simple and hopefully reliable too. Enjoy. Download from https://gist.github.com/1389206/.

“DebPAN”, a production-grade Debian CPAN repository

The problem

This is a proposal I came up with after talking to Gabor Szabo about his Perl Ecosystem Development proposal.

One of the major "problems" we face while developing and deploying production Perl-based systems with Debian is that the state of the Debian CPAN modules is depressingly outdated. As an example, we're using Catalyst in Lenny, and that dates back to 2008 for the most parts.

This is just not enough.

A solution: maintain your own APT repository

Our current solution is to manually package every bit and maintain our own internal Opera APT repository that our servers and applications depend on. That's not optimal for two reasons:

  • we have to package lots of modules, due to interdependencies, dedicating a fair amount of time to this activity that is not exactly "productive"
  • we can't trust our systems to be Debian anymore, since we're updating bits and pieces with the bold assumption that everything will work fine

Of course, 99.999% will work fine, since it's Perl, but the problem is that one day this could fall down on our heads.

So, given the problem, what are the solutions?

  • Continuing to manually keep an apt repository. Downside: Some(tm) waste of time
  • Use a different packaging/deployment system, like PAR::Repository. I would personally like this, but it doesn't eliminate the need to maintain an own repository. You just don't use dh-make-perl and friends, that are, IMHO, nice to have and useful
  • The "DebPAN"

The DebPAN

I know Jeremiah Foster, and at a couple of Perl events I heard him talking to other CPAN/Perl developers about these issues. His answer would probably be to file a request for packaging for the modules we're interested in.

The reason why that doesn't work is that the lead time for a given RFP to land on Debian stable is unacceptably long, and I realize that is for a good reason. After all, it's supposed to be stable, right?

What about a "DebPAN" repository?

That could be a 3rd party APT repository, something like debpan.perl.org:

  • maintained by a close group of Perl/Debian/CPAN developers
  • guaranteed to have a selection of the most important modules (more on that…) in a reasonably recent version
  • targeted to Debian stable, and maybe other distributions? I think Ubuntu 10.04 suffers from this same problem, but much less than Debian Lenny, just to pick two versions
  • maybe even with patches applied?, but that might be way too much, actually

The can of worms

Of course, lots of problems can arise. However, if we think this is a good idea, then we should try to have something even minimal up and running. Then we'll worry about all the problems…

However, who gets to decide the most useful modules? That should go by popular demand I guess. Even looking at Debian requests for packaging stats, maybe? I can also imagine that bigger companies using Perl would be interested in this to potentially save lots of "infrastructural work".

I'd be really interesting to know other people opinions on this, especially if they use Debian stable, Debian developers, or the Debian-Perl group itself.

Pimp my Debian

Have you ever reinstalled your workstation and found out that your Perl scripts need a shit load of modules that you don't have anymore? Or maybe on a server?

I have. In such cases you have to:

  • run your script,
  • find out which module is missing,
  • figure out if there's a debian package for it,
  • install the debian package,
  • GOTO 10

Today I was so annoyed and lazy, that I decided to put an end to this madness. So I wrote pimp-my-debian. It's an innocent script that you can run as follows:

$ pimp-my-debian --command 'perl ./myscript'

It will keep running your command (perl ./myscript), reading its output, and if it contains something like Can't locate module Foo/Bar.pm in @INC, or Base class package "Foo::Bar" is empty, it will try to figure out a suitable debian package, install it, and retry your command.

It worked so well that I so want to use it again… :-)
Get pimp-my-debian here and have fun!