Tag Archives: github

A command line tool for Debian to purge Varnish objects

I've been using varnish mostly on Debian systems. I found the reload-vcl script included in Debian to be useful.

The reload-vcl script

It's part of the standard varnish debian package. It uses the system defaults in /etc/defaults/varnish, so it knows how to correctly invoke the varnishadm utility to perform administrative commands. As the name implies, it reloads the default VCL file using the vcl.load and vcl.use commands, checking that every step succeeds properly before continuing so it's safe to use. It loads the new VCL file and labels it automatically with a unique id.

Something analogous but regarding the purge functionality could have been useful, so I looked at the source code for reload-vcl. Most of it deals with loading of /etc/defaults/varnish and various sanity checks. I reused that bit to make another script, to control cache purging.

The purge-cache script

Here's the full source code. Below there's a link to download the latest version from github.


#!/bin/sh

# purge-cache: Script to purge varnish cache. Defaults are defined in
# /etc/default/varnish.
#
# Cosimo <cosimo@cpan.org>
# Based on reload-vcl, by Stig Sandbeck Mathisen <ssm at debian dot org>

# Settings
defaults=/etc/default/varnish

# Paths
varnishadm=/usr/bin/varnishadm
date=/bin/date 
tempfile=/bin/tempfile

# Messages
# msg_no_varnishadm: varnishadm
msg_no_varnishadm="Error: Cannot execute %sn"
msg_no_management="Error: $DAEMON_OPTS must contain '-T hostname:port'n"
# msg_defaults_not_readable: defaults
msg_defaults_not_readable="Error: %s is not readablen"
# msg_defaults_not_there: defaults
msg_defaults_not_there="Error: %s does not existn"
msg_usage="Usage: $0 [-h][-q][-u <url>|-r <regex>|-a]nt-htdisplay helpnt-qtbe quietnt-utpurge by exact (relative) url (ex.: /en/products/)nt-rtpurge objects with URL matching a regex (ex.: ^/blogs/)nt-atpurge all objects from cachen"
msg_purge_failed="Error: purge command failedn"
# msg_purge_url: url
msg_purge_url="Purging objects by exact url: %sn"
# msg_purge_regex: regex
msg_purge_regex="Purging objects with URL matching regex: %sn"
msg_purge_all="Purging all cachen"
msg_purge_ok="Purge command successfuln"

# Load defaults file
if [ -f "$defaults" ]
then
    if [ -r "$defaults" ]
    then
        . "$defaults"
    else
        printf >&2 "$msg_defaults_not_readable" $defaults
        exit 1 
    fi
else
    printf >&2 "$msg_defaults_not_there" $defaults
    exit 1
fi

# parse command line arguments
while getopts hqu:r:a flag
do
    case $flag in
        h)
            printf >&2 "$msg_usage"
            exit 0
            ;; 
        u)
            purge_method=url
        url="$OPTARG"
            ;; 
        r)
            purge_method=regex
        regex="$OPTARG"
            ;; 
        a)
            purge_method=all
            ;; 
        q)
            quiet=1
            ;; 
        *)
            printf >&2 "$msg_usagen"
            exit 1
            ;; 
    esac
done

# Parse $DAEMON_OPTS (options must be kept in sync with varnishd).
# Extract the -f and the -T option, and (try to) ensure that the
# management interface is on the form hostname:address
OPTIND=1
while getopts a:b:dFf:g:h:l:n:P:p:s:T:t:u:Vw: flag $DAEMON_OPTS
do
    case $flag in
        f)
            if [ -f "$OPTARG" ]; then
                vcl_file="$OPTARG"
            fi 
            ;; 
        T)
            if [ -n "$OPTARG" -a "$OPTARG" != "${OPTARG%%:*}" ]
                then
                mgmt_interface="$OPTARG"
            fi  
            ;;  
    esac
done

# Sanity checks 
if [ ! -x "$varnishadm" ]
then
    printf >&2 "$msg_no_varnishadm" $varnishadm
    exit 1
fi

if [ -z "$mgmt_interface" ]
then
    printf >&2 "$msg_no_management"
    exit 1
fi

logfile=$($tempfile)
purge_command="vcl.list"

# Now run the purge command against the admin interface
if [[ $purge_method = "url" ]]
then
        purge_command="purge req.url == $url"
        printf >&2 "$msg_purge_url" $url | grep -v "^$" > $logfile
else
    if [[ $purge_method = "regex" ]]
    then
        purge_command="purge.url $regex"
        printf >&2 "$msg_purge_regex" $regex | grep -v "^$" > $logfile
    else
        if [[ $purge_method = "all" ]]
        then
            purge_command="purge.url ."
            printf >&2 "$msg_purge_all" | grep -v "^$" > $logfile
        fi
    fi
fi

# For some reason, using:
#
#   fi | grep -v "^$" > $logfile
#
# results in purge_command assignment being wiped out
# at the end of the block??

if [ -z "$purge_command" ]
then
    printf >&2 "$msg_usagen"
    exit 1
fi

# echo "cmd: $varnishadm -T $mgmt_interface $purge_command"

if $varnishadm -T $mgmt_interface $purge_command
then
    printf >&2 "$msg_purge_ok"
else
    printf >&2 "$msg_purge_failed"
    exitstatus=1
fi | grep -v "^$" > $logfile

# Blather
if [ -z "${quiet}" -o -n "$exitstatus" ]
then
    cat >&2 $logfile
fi

# Cleanup
rm -f $logfile  
exit $exitstatus

You can control how objects are purged from the cache with 3 options:

  • -a: purges all objects
  • -u <url>: purges an exact url
  • -r <regexp>: purges objects matching a regular expression
  • Examples

    
    # Purges all objects
    purge-cache -a
     
    # Purges all objects starting with "/products"
    purge-cache -r '^/products'
    
    # Purges objects with exact URL
    purge-cache -u '/en/homepage'
    

    Goal: no downtime

    Both reload-vcl and purge-cache can be combined together in a single script to be triggered when deploying new VCL code or new backend applications. Instead of restarting varnish, which I really don't like, and it's not very reliable either (on Debian sometimes it won't come back up), I use purge-cache -a to purge all objects and then reload-vcl to load and use the newly deployed VCL code.

    This procedure has no downtime at all. The effect of purging all objects can potentially be hard on the backends, but we're not at that point yet. Usually in the busiest applications we have, it takes around 10-20 seconds to reach 70%-75% of hit rate, so I would say that's not really a problem right now.

    Download!

    You can download the purge-cache script from github. I contacted the maintainer of the reload-vcl script. Maybe he will include purge-cache in the next release of the varnish debian package… or maybe I could package it as a Perl CPAN module.

Matching IPv6 addresses with Regexp::Common

I wish Regexp::Common had a $RE{net}{IPv6} regular expression, but it doesn't (yet).

So I tried to implement this myself, but ripped off the IPv6 matching bit from the existing Regexp::IPv6 which happens to have a working IPv6 regular expression with a reasonable test suite. Now, why Regexp::IPv6 is not part of Regexp::Common?

By the way, I'll copy/paste the full regular expression to match IPv6 addresses, just for fun:

:(?::[0-9a-fA-F]{1,4}){0,5}(?:(?::[0-9a-fA-F]{1,4}){1,2}|:(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.]
(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?
[0-9]{1,2})))|[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}:(?:
[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}:(?:[0-9a-fA-F]{1,4}|:)|(?::(?:[0-9a-fA-F]{1,4})?|(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?
[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4]
[0-9]|[0-1]?[0-9]{1,2}))))|:(?:(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})
[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|[0-9a-fA-F]{1,4}(?::[0-9a-fA-F]
{1,4})?|))|(?::(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|
2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|:[0-9a-fA-F]{1,4}(?::(?:(?:25[0-5]|2[0-4]
[0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.]
(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|(?::[0-9a-fA-F]{1,4}){0,2})|:))|(?:(?::[0-9a-fA-F]{1,4}){0,2}(?::(?:
(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?
[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|(?::[0-9a-fA-F]{1,4}){1,2})|:))|(?:(?::[0-9a-fA-F]{1,4}){0,3}
(?::(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|
[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|(?::[0-9a-fA-F]{1,4}){1,2})|:))|(?:(?::[0-9a-fA-F]{1,4})
{0,4}(?::(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4]
[0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))|(?::[0-9a-fA-F]{1,4}){1,2})|:))

Of course, I will never be able to tell if it's right or wrong, but the fact is that it passes the test suite :)
However, the actual code is not like that: it generates the full regular expression from a few components. Anyway, I've pushed a ipv6 branch on my fork of Regexp::Common. I hope it will be included soon in Regexp::Common or improved it enough to be included in it, so we can finally match IPv6 addresses with:


use Regexp::Common;

my $addr = '2001:0db8:0000:0000:0000:0000:1428:57ab';
if ($addr =~ $RE{net}{IPv6}) {
    print "Yes, it is an IPv6 address";
}
else {
    print "No, it isn't";
}

Geo::IP support for IPv6 geolocation

We're currently looking into IPv6-enabling our services. One of the missing bits is being able to geolocate IPv6 client addresses. We're using the MaxMind GeoIP database. The main Perl library for this is Geo::IP.

The current version of Geo::IP out on CPAN, 1.38, does not support IPv6 lookups. I contacted the maintainer of Geo::IP asking for more information. In the meanwhile, I hacked together just enough of IPv6 support to be able to successfully geolocate a test address. Later on, I discovered that IPv6 is already available in the hopefully soon-to-be-released version of Geo::IP archived at Sourceforge.

Let's hope it lands on CPAN soon. In the meantime, if you really really want, you can try out my changes against CPAN v1.38. It was enough for me to start testing the integration with our other code and projects.

My Geo::IP with IPv6 support

How to convert Opera contacts file to Mutt aliases format

Recently I've been looking more and more into mutt, the email client. I've been a very happy M2 (Opera built-in email client) user for almost 3 years now. But still I felt I was missing something if I didn't try out mutt. I've been a pine user as well, many many years ago :) So, decided to give it a go, I started about a month ago.

I struggled a bit while getting a reasonable .muttrc file together. Fortunately, there's plenty of examples out there. After getting a working config, the problem was to get back my contacts list.

Mutt has a simple address book integration (through abook) and stores the contacts into an alias file, typically ~/.mutt/aliases. Now, Opera can of course export all your mail contacts to an .adr file, a simple "addressbook" text file. Did that, and I needed to convert it to mutt's aliases format.

Ten minutes later, a Perl script to do just that was ready. Here it is:


#!/usr/bin/env perl
#
# Convert Opera contacts file (.adr) into
# mutt aliases file format.
#
# Usage:
#   perl opera-adr-to-mutt-aliases.pl < ~/.opera/contacts.adr >> ~/.mutt/aliases
#
# Cosimo, 31/Jan/2011
#

use strict;
use warnings;
use utf8;

sub harvest ($) {
    my ($contact_info) = @_;

    my ($id)    = $contact_info =~ m{^ s+ ID   = (.*) $}mx;
    my ($name)  = $contact_info =~ m{^ s+ NAME = (.*) $}mx;
    my ($email) = $contact_info =~ m{^ s+ MAIL = (.*) $}mx;

    return if ! $id and ! $email;

    return {
        ID    => $id,
        NAME  => $name,
        MAIL => $email,
    };

}

my $adr_file_contents = q{};
$adr_file_contents .= $_ while <STDIN>;

my @contacts = split m{#CONTACT}, $adr_file_contents;

for (@contacts) {
    my $contact = harvest($_) or next;
    my ($first_word) = $contact->{MAIL} =~ m{ (S+) @ }x;
    printf "alias %s %s <%s>n",
        lc($first_word), $contact->{NAME}, $contact->{MAIL};
}

Download link: https://gist.github.com/803454

Another round of fixes for the varnish Accept-Language VCL extension

This very specific VCL extension to parse and normalize Accept-Language headers is becoming more robust as time goes by. Here's the latest round of fixes:

  • the original req.http.Accept-Language header could be overwritten when calling the vcl_rewrite_accept_language() function. Now this is fixed by copying the original header string into a static buffer and executing the processing on the copy.
  • improved a bit the style of the C code in a few places. Nothing miraculous really. Feels improved for me at least :)
  • fixed the use of a wrong define (string max length instead of languages list max length)
  • use of sizeof instead of using same constants in strncpy, etc…
  • added a small intro on the generated file too. In this way, if you find the code on some server, you can immediately understand what's that supposed to do, and where it came from :)

Enough blah blah, here's the new code. If you were already running this piece of VCL, then I won't tell you this is experimental stuff, because at this point it's no more experimental. But if you were running it already, then by all means upgrade. It is definitely better :)

http://github.com/cosimo/varnish-accept-language

Enjoy!

If Text::Hunspell never worked for you, now it’s time to try it again!

If you don't know, Hunspell is the spell checker engine of OpenOffice.org, and it's also included in the Opera and Mozilla browsers.

We were trying to use it from Perl, using the old Text::Hunspell module, version 1.3, but we had problems with it. Big problems. Like segfaults and tests that wouldn't run.

A bright hacker from Italy :) was then called in to fix the problem, with the promise of a fantastic prize he hasn't seen yet… [ping?] :-)

During the process, I found out I know absolutely nothing about dictionary files and stuff, and my fixes were – I would say – definitely horrible.

But! There's a bright side, of course, and that is that the module works just fine now, at least on Debian/Ubuntu systems. Before using Text::Hunspell, you want to install the following packages:

  • hunspell
  • libhunspell-dev

The example in the POD documentation (and in the examples dir) uses the standard US english dictionary. If you don't have that, you will need to change the script slightly. But the code is tested and should work without a problem. If you try it out and you have feedback, by all means let me know. Thanks!

Source code available on GitHub at:
http://github.com/cosimo/perl5-text-hunspell/

The module, tagged as 2.00 because it's cool :), will be up on CPAN shortly at this address:
http://search.cpan.org/dist/Text-Hunspell/

Wiimote Perl hacks

The wiimote is a geek toy. Here's what I did with it so far.

3D Earth

I wrote this OpenGL program in Perl more than a year ago. It can display Apache access logs in real time, as geolocated spikes on the rotating planet (if your machine is powerful enough :). It was a fun way for me to learn tiny bits of OpenGL, enough to run it. I thought it would be cool to use the wiimote to control the planet, twisting and spinning the Earth with your hands. It worked pretty well, and I'm really satisfied with the result.

Fast forward…

During last couple of weeks, I managed to find some spare time to play with the wiimote again. Here's the result, the wiimote tools.

Wiimote tools

Tools it's probably not the right word. It's small hacks. It 3 different hacks in one box. I know there's already stuff out there that let you do everything like this, but honestly I don't care at all. The interesting part is writing something, preferably in Perl, to do that myself. And if my kids can have fun with it, it's the best a father-hacker can hope for… :)

Copy-paste from the README follows:

wiimote-drumkit

This is something I made for my small kids, so they could just play with sounds. You can write a small config file to assign random mp3s/wav/… to each button of the wii.

Take a look at drumkit.conf or animals.conf as examples.

Then every time you press a button, or wave the wii in the air (pretending you're playing drums), you will hear the corresponding sounds. It requires 'paplay', the Pulse audio command line player. My daughter almost managed to fork bomb my PC pressing buttons everywhere :)

wiimote-joystick

This is a quick hack that I put together to use the wiimote together with any of my favorite games as a digital joystick.

I can use it to play with mame (see pic), of course with legal roms or c64 emulators (with games I of course owned when I was a kid) and so on. Currently it works well with a few games, because of the way the keys are sent to X11, and it works really bad with other games, because of the way the keys are sent to X11. :)

Requires the X11::GUITest module, install with sudo cpan X11::GUITest.

wiimote-conductor

This is the stupidest of the lot. It can control the speed of a movie (or song?) by pretending to use the wiimote like a drunk orchestra conductor would.

It's a bit sucky because it doesn't get the actual tempo from the sound, so you have to pretend the normal (1.0x) speed is represented by wii-beats distanced by 1 second. So if you want to double the speed of the movie/song, you have to wave the wii in the air every 0.5 seconds. Pretty lame, eh? :)

Requires Audio:: Play:: MPlayer with a trivial patch I made to add the speed command.

Everything is nicely packaged here, if you want to experiment with it.

… and this is for mst, have fun!

Another quick update to Ubiquity for Opera

I just pushed a small update to my Ubiquity for Opera user javascript. Two tiny changes:

  • Now the ESC key hides the Ubiquity window
  • Fixed the text selection and focus when you reopen the Ubiquity window and you had input a command before.

Thanks to Martin Šrank for contributing the ubiq_focus() fix.

To download the latest version, go to http://github.com/cosimo/ubiquity-opera/. There's also a minified version there.