Category Archives: Uncategorized

Perl6 hacking, grammars, Digest::MD5 and caffeine levels

I'll be brief. Need some sleep. :)

Perl 6 is here. Now. And there's an immense work waiting to be done: rewriting Perl5's CPAN. Ain't that easy? :)

Anyway, during last couple of weeks, I spent most of my spare time playing with Perl 6:

Grammars

My poor excuse to (try to) learn grammars was to build a Perl6 class that could parse a puppet module and build some documentation for it. Puppet itself uses a grammar to parse its modules, so I thought it wouldn't be impossible to port it to Perl6 and use it to parse puppet code.

Well, turns out it's not so easy, but at least I'm learning how grammars work and having fun.

Perl6 Digest::MD5

This is extremely fascinating, because it's touching the Parrot core. In Parrot, there's already a Digest::MD5 module, so all you have to do (but again, not so easy), is to write a Perl6 "wrapper" around the Parrot code.

And how do you do that? With PIR blocks. This stuff is great. Seriously. It's like going back to Assembly, in some sense(tm). Here's an example of this glue PIR code:

class Digest::MD5 {

    multi method md5_hex (Str $message) {

        pir::load_bytecode('Digest/MD5.pbc');

        my $md5_sum = Q:PIR {
            .local pmc md5sum, md5_sum_get
            md5sum = get_root_global ['parrot'; 'Digest'], '_md5sum'
            $P0 = find_lex '$message'
            $P1 = md5sum($P0)
            md5_sum_get = get_root_global ['parrot'; 'Digest'], '_md5_hex'
            %r = md5_sum_get($P1)
        };

        return $md5_sum;
    }

    multi method md5_hex (@message) {
        my Str $message = @message.join('');
        return Digest::MD5.md5_hex($message);
    }

}

Even if you don't understand Perl 6 or PIR, you can probably recognize a class definition, and polimorphic methods. md5_hex() is in fact defined twice:

  • multi method md5_hex (Str $message)
  • multi method md5_hex (@message)

You don't have to write polimorphic methods, but you can do it if you want. Yes, there's more than one way, and there always will be. I don't like dictators, even if they are benevolent, and Python code looks so flat and dull, seriously. There's no personality in Python code. Yes, sigils are great.

Digest::MD5 is also using alien technology (UFO).

Synopsis documents

Nothing fancy there, just improved the existing CSS. For an example, go read Synopsis 03 about operators.

Good night!

Disassembling a real world Plack PSGI application

After I started playing with Plack, I tried to evaluate whether to continue using it for our mission-critical production stuff or give up, going back to the same techniques we already use (successfully).

I think it's time to develop and deploy a Plack based application. In my grand plan, :-), I'd like to deploy nginx with PSGI support, or even more ambitiously, nginx or apache with Starman as "backend" http server. We'll see…

In the meantime, I'd like to write here a couple of niceties about Plack and Starman, showing some real code I wrote when I started.

A real world PSGI application

Here's a sample PSGI application currently under development:

#!/usr/bin/env perl
#
# Sample PSGI application
#

use strict;
#se warnings;
use constant ENVIRONMENT         => 'development';
use constant APACHE_DEPLOYMENT   => (ENVIRONMENT eq 'production');
use constant ENABLE_ACCESS_LOG   => (ENVIRONMENT eq 'development');
use constant ENABLE_DEBUG_PANELS => (ENVIRONMENT eq 'development');

use Plack::Builder;
use AuthOpera;
use AuthOpera::Account;

my $app = AuthOpera::Account->new(); 

builder {

    enable_if { not APACHE_DEPLOYMENT }
        'Plack::Middleware::Static', 
        path => qr{^/(bitmaps/|images/|js/|css/|downtime/|favicon.ico$|ping.html$)},
        root => '..',
        ;

    mount "/account" => builder {

        enable_if { ENABLE_DEBUG_PANELS } 'StackTrace';
        enable_if { ENABLE_DEBUG_PANELS } 'Debug';   # panels => [ qw(DBITrace Memory) ];
        enable_if { ENABLE_DEBUG_PANELS } 'Lint';
        enable_if { ENABLE_DEBUG_PANELS } 'Runtime';
        enable_if { ENABLE_ACCESS_LOG   } 'AccessLog';

        $app;

    }

}

Of course, the main application code is not here, but in the AuthOpera::Account class. That's not really relevant to what we're discussing here. Let's just say that any class, to be a valid and complete PSGI application, has to:

  • subclass from Plack::Component
  • have a call() method
  • the call() method must return a valid PSGI response. Example:
    package MyPSGIApp;
    
    use strict;
    use Data:: Dumper ();
    use parent 'Plack::Component';
    
    sub call {
        # $env is the full PSGI environment
        my ($self, $env) = @_;
    
        return [
    
            # HTTP Status code
            200,
    
            # HTTP headers as arrayref
            [ 'Content-type' => 'text/html' ],
    
            # Response body as array ref
            [ '<!DOCTYPE html>',
              '<body><h1>Hello world</h1><pre>',
              Data:: Dumper:: Dumper($env),
              '</pre></body></html>',
            ],
        ];
    }
    
    1;
    

That's it, this is a full PSGI application that does dump all its PSGI environment.

Of course in a real example, you probably want a template engine to return the page content, etc… That's what we are building for our applications. Actually just assembling the components we already have developed during these years, so we have template classes, config classes, localization, database access, etc…

So we're basically just gluing these ready made components inside the PSGI application, and then using them. I don't think this is particularly original, but it allows us to quickly "port" our code to PSGI and thus run anywhere we want to.

app.psgi in detail

Now, let's see the PSGI app in more detail.

use constant ENVIRONMENT         => 'development';
use constant APACHE_DEPLOYMENT   => (ENVIRONMENT eq 'production');
use constant ENABLE_ACCESS_LOG   => (ENVIRONMENT eq 'development');
use constant ENABLE_DEBUG_PANELS => (ENVIRONMENT eq 'development');

These constants are used to turn on and off certain features mentioned later in the builder {} block. I just found out the other day that these constants are near to useless. That is because plackup and starman already provide a -E environment switch. If you start your application with:

starman -E development myapp.psgi     # same with plackup, the default server

then Plack will by default enable the debugging panels and the Apache-style access log. I found out about this after having written that file. This means that the following enable_ifs are unnecessary:

mount "/myroot" => builder {
    enable_if { ENABLE_DEBUG_PANELS } 'StackTrace';
    enable_if { ENABLE_DEBUG_PANELS } 'Debug';   # panels => [ qw(DBITrace Memory) ];
    enable_if { ENABLE_DEBUG_PANELS } 'Lint';
    enable_if { ENABLE_DEBUG_PANELS } 'Runtime';
    enable_if { ENABLE_ACCESS_LOG   } 'AccessLog';
    $app;
}

I think Plack enables by default at least StackTrace, Debug, and AccessLog. In my case, however, I'm also enabling RunTime and Lint. But more importantly, I need to differentiate between Apache deployment and Starman deployment. That affects the way static files are served.

When deploying under Apache, I don't need the following:

enable_if { not APACHE_DEPLOYMENT }
    'Plack::Middleware::Static',
    path => qr{^/(bitmaps/|images/|js/|css/|downtime/|favicon.ico$|ping.html$)},
    root => '..';

because my PSGI application is enabled in an Apache <Location> block, as in:

<Location /myroot/>
    SetHandler perl-script
    PerlResponseHandler Plack::Handler::Apache2
    PerlSetVar psgi_app /my/path/to/app.psgi
</Location>

So Apache already takes care of serving the static files for me. However, when running completely under Starman, I need to tell it which folders or paths need to be served as static files, and where they are located. This is the purpose of the Static middleware:

enable_if { not APACHE_DEPLOYMENT } 'Plack::Middleware::Static',
    path => qr{^/(images/|js/|css/|favicon.ico$)},
    root => '/var/www/something';

If you're always deploying through plackup or starman, then, again, you don't need any enable_if, just enable. Maybe it's also a good idea to put everything under /static. For me that wasn't possible, since I already had existing content:

enable 'Plack::Middleware::Static',
    path => qr{^/static/},
    root => '/var/www/something';

Plack::Builder

About the Plack::Builder bit, and the related builder function. That is a function that helps you specify what you want Plack to run and how. Example:

builder {
    enable 'StackTrace';
    enable 'Debug';
    enable 'AccessLog';
    $app
}

where StackTrace, Debug, and AccessLog are all middleware classes, so causes Plack to wrap your final $app application first with the AccessLog middleware, then Debug and then StackTrace. I didn't check the code, but I believe this creates 3 different PSGI applications that are meant to fiddle with the response that your own application generates.

PSGI makes this possible, and it's just great. More middleware means easier and faster development. And ultimately, very good middleware makes for great reuse too.

The mount wrapper

I used mount in my example very basicly, but you can use mount to assemble compounds of applications in a very simple way. The same thing you do, for example, with Django and urls.py, except that, if you have seen a non-trivial urls.py, it looks like spaghetti after a while. Compare with this:

my $app1 = MyApp->new();
my $app2 = MyApp2->new();
#...

builder {

    enable 'Plack::Middleware::Static', 
        path => qr{^/static/},
        root => '/var/www/something';

    mount "/path1" => builder {
        enable 'StackTrace';
        $app;
    }

    mount "/path2" => $app2;

    mount "/path3" => builder {
        enable 'SomeMiddleware';
        $app3;
    }

}

Of course, then you have to add some dispatcher logic to your applications, but in the Plack world, we don't lack good dispatchers.

Plack rocks.

A working bogosort in Perl6

It's not even mine actually. @moritz helped me with that. I couldn't remember the syntax for the reduce operator (and it's [<op>]), so here it is, in its full wikipedia glory.

@list .= pick(*) until [<=] @list

Here's a somewhat more complete example, that you can actually run:

my @list = 1, 2, 7, 4, 3;
@list .= pick(*) until [<=] @list;
say @list.perl;

This will bogosort the array (wikipedia), that is, shuffle it until it's ordered. Look at the Java version for a nice comparison.

Here's the explanation:

my @list = 1, 2, 7, 4, 3;

Declares the list. You don't need parens around the elements. If that was a deck of cards, you could write it as:

my @list = <A 2 3 4 5 6 7 8 9 J Q K>

The angular brackets <> are the equivalent of qw() in Perl 5. Following line is quite dense:

@list .= pick(*) until [<=] @list;

This can be written also as:

while not [<=] @list {
  @list .= pick(*)
}

As you can see, you don't need parens around the while condition. The [<=] @list bit is IINM a reduce operator for a list. For example, to sum all elements of a list, you'd use:

my @num = 1, 2, 3, 4;
say [+] @num;

So the [<=] operator applied to a list compares each element with the next one, then the result with the next one. Assuming @list = 2, 4, 8, 5, that is how I think it runs:

2 <= 4 (true, go on, last evaluated value is 4)
4 <= 8 (true, go on, last value is 8)
8 <= 5 (false, stop here, return value is false)

So, if the array is ordered, the while/until block is skipped. If the array is not ordered, execute the following:

@list .= pick(*)

.= is an operator for in-place method call. Writing that is equivalent to:

@list = @list.pick(*)

That in Perl 5 would be written as:

$list = SomeListObject->new(1, 2, 3, 4, 5);
$list->pick(???);

Actually, there wouldn't be any Perl 5 direct equivalent code, since the magic bit here (the *, aka "whatever") it's not even there. "whatever" is magic for "anything that's applicable here, until there's an end to it", or at least this is my ignorant interpretation of it… :)

So in this context, pick() is a method that returns a random element from a list. Together with *, it means pick a random element from any of the original @list. Mmmh. But then it should be equivalent to:

@list .= pick(@list)

But it's not, so pick(*) knows the right thing to do…
And it's clear that I don't know how pick() works :)
Anyway…

BTW, this is runnable code. If you want to try it, clone the rakudo repository, and run perl Configure.pl --gen-parrot to build it. Have fun!

How to generate an XML document with XML::LibXML

For the "note to self" series, or… if I ever need to do this again, I will find this here, here's how to generate an XML document with XML::LibXML. I usually use XML::Simple or XML::TreePP, but this script was already requiring XML::LibXML to parse some XML fragments, so I thought I would use the same module. It wasn't really immediate…

#!/usr/bin/env perl

#
# Create a simple XML document
#

use strict;
use warnings;
use XML::LibXML;

my $doc = XML:: LibXML:: Document->new('1.0', 'utf-8');

my $root = $doc->createElement("my-root-element");
$root->setAttribute('some-attr'=> 'some-value');

my %tags = (
    color => 'blue',
    metal => 'steel',
);

for my $name (keys %tags) {
    my $tag = $doc->createElement($name);
    my $value = $tags{$name};
    $tag->appendTextNode($value);
    $root->appendChild($tag);
}

$doc->setDocumentElement($root);
print $doc->toString();

and this outputs:

<?xml version="1.0" encoding="utf-8"?>
<my-root-element some-attr="some-value">
    <color>blue</color>
    <metal>steel</metal>
</my-root-element>

And a copy is also archived at stackoverflow :)

Plack, the grand glue

I have been looking at Plack for several months now. I always thought it was a cool project, where "cool" means useful, good code, nice documentation, well structured, strong development "flow", etc…

Lately I've been "rebooting" an internal project, doing a lot of infrastructure work like deployment tools, management of different environments like devel, test, staging and production, bug fixing, etc… Regular stuff that you usually already take for granted, but this project didn't have, for many reasons.

After this rebooting work, time has come to add new features to the web-facing part of this project, so new pages and forms. My frustration came from mainly two factors:

  • the code being tightly integrated with mod_perl
  • the need to change the apache config every time you add a new /something to the application

While the mod_perl integration is not necessarily a bad thing, and mod_perl is a fast and reliable product serving millions of pageviews per day, it's also nice to have code that you can run anywhere, not just on Apache. That might or might not happen, but I'd want to be ready when that's needed. In fact, we're starting to use nginx on some applications, including My Opera

So I decided to invest some time to play with Plack. Plack transposed to Perl concepts from Ruby's Rack and WSGI, a spec born in the Python world I believe.

I'm still at the first experiments with it, but Plack is a great software with a spectacular potential! If you're in doubt, try it for yourself. What made me decide to try Plack was Starman.

Starman is a damn fast PSGI-enabled preforking HTTP web server written in Perl. As soon as I started starman with a stupid simple app.psgi I realized I had to invest some time in it.

A couple of days later I have the slick opera.com design into a bunch of Template Toolkit blocks, with all machinery to make them work for real, and a PSGI application class. From the screenshot you can see this request is loaded in 38 ms, and this my desktop machine, and the Debug middleware gives you useful debugging info through sidebar panels.

I can run this class either as standalone with Starman, or in Apache + mod_perl with Plack::Handler::Apache2 or anything else for that matter, like FastCGI, or even plain CGI if you want that.

And I think that shows:

  • how cool the PSGI/WSGI concept is
  • that you can really code once, run anywhere, and don't care about the web server stack

Wiimote Perl hacks

The wiimote is a geek toy. Here's what I did with it so far.

3D Earth

I wrote this OpenGL program in Perl more than a year ago. It can display Apache access logs in real time, as geolocated spikes on the rotating planet (if your machine is powerful enough :). It was a fun way for me to learn tiny bits of OpenGL, enough to run it. I thought it would be cool to use the wiimote to control the planet, twisting and spinning the Earth with your hands. It worked pretty well, and I'm really satisfied with the result.

Fast forward…

During last couple of weeks, I managed to find some spare time to play with the wiimote again. Here's the result, the wiimote tools.

Wiimote tools

Tools it's probably not the right word. It's small hacks. It 3 different hacks in one box. I know there's already stuff out there that let you do everything like this, but honestly I don't care at all. The interesting part is writing something, preferably in Perl, to do that myself. And if my kids can have fun with it, it's the best a father-hacker can hope for… :)

Copy-paste from the README follows:

wiimote-drumkit

This is something I made for my small kids, so they could just play with sounds. You can write a small config file to assign random mp3s/wav/… to each button of the wii.

Take a look at drumkit.conf or animals.conf as examples.

Then every time you press a button, or wave the wii in the air (pretending you're playing drums), you will hear the corresponding sounds. It requires 'paplay', the Pulse audio command line player. My daughter almost managed to fork bomb my PC pressing buttons everywhere :)

wiimote-joystick

This is a quick hack that I put together to use the wiimote together with any of my favorite games as a digital joystick.

I can use it to play with mame (see pic), of course with legal roms or c64 emulators (with games I of course owned when I was a kid) and so on. Currently it works well with a few games, because of the way the keys are sent to X11, and it works really bad with other games, because of the way the keys are sent to X11. :)

Requires the X11::GUITest module, install with sudo cpan X11::GUITest.

wiimote-conductor

This is the stupidest of the lot. It can control the speed of a movie (or song?) by pretending to use the wiimote like a drunk orchestra conductor would.

It's a bit sucky because it doesn't get the actual tempo from the sound, so you have to pretend the normal (1.0x) speed is represented by wii-beats distanced by 1 second. So if you want to double the speed of the movie/song, you have to wave the wii in the air every 0.5 seconds. Pretty lame, eh? :)

Requires Audio:: Play:: MPlayer with a trivial patch I made to add the speed command.

Everything is nicely packaged here, if you want to experiment with it.

… and this is for mst, have fun!

Puppet, Fabric and a Perl alternative?

Some time later this month I'm going to write more extensively about a project that I've been working on, not continuously, but for the last couple of months. It is about small and medium scale projects configuration management and deployment.

For configuration management I evaluated several products like bcfg2, puppet, cfengine and lcfg, and I finally chose puppet.

For "the last mile", as I call it, the alternatives that I considered were fabric, capistrano, ControlTier and TheNewShinyWheel(tm)

So I settled on puppet + fabric. Puppet is a Ruby system, while Fabric is Python code. None of them is particularly fast, actually Puppet is slow, and Fabric is acceptable. The main problem I'm confronting with, after having learnt how to use these tools, is that Fabric does not support parallel processing of tasks.

This is a severe limitation for us. This was a pilot project. If it works well, it could be applied to many other deployment tasks. That could also mean that a single deployment has to send code or files to tens of servers, and you don't want to do that sequentially waiting for each task to complete.

At the moment, this is impossible to do with Fabric. There is an experimental fork in the works that might support parallel execution, by adding a @parallel task decorator, but it still requires work and a good dose of testing.

During my survey I looked for mature Perl-based deployment tools, but I failed at finding them. While Fabric is nice, I might be tempted to reconsider my choice. Any suggestions?

Undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193

Today I was trying to run some Test::Class-based unit tests for a project I'm working on. This project has lots of functional tests that go through Test::WWW::Mechanize, so they run inside Apache and mod_perl.

What I was trying to do was testing the same code outside the Apache server, just as regular command-line-runnable tests. Being a Perl project, you have all the nice infrastructure already done for you, and so you can just run make test or Build test depending on the exact tool you're using.

However, when I run this, I got a strange error:

PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/unit-tests/basic-sanity.t .. ok
t/unit-tests/handler-xml.t ... Can't load '/usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so' for module APR::Request::Apache2: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193.
 at /usr/lib/perl5/Apache2/Request.pm line 3
Compilation failed in require at /usr/lib/perl5/Apache2/Request.pm line 3.
BEGIN failed--compilation aborted at /usr/lib/perl5/Apache2/Request.pm line 3.

The weird error is:

Can't load '/usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so' for module APR::Request::Apache2: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193.

and specifically, undefined symbol: modperl_xs_sv2request_rec.
Why is this happening?

A search for this error message gives back very few results, and not very useful. However, this more specific search turns up fewer results, but that led me to the solution.

The first hit, in particular, I have no idea what that page says, because it's Japanese, but the code blocks are clearly referring to the PERL_DL_NONLAZY environment variable:

PERL_DL_NONLAZY

    Set to one to have perl resolve all undefined symbols when it
    loads a dynamic library.  The default behaviour is to resolve symbols
    when they are used.

    Setting this variable is useful during testing of extensions as it
    ensures that you get an error on misspelled function names even if
    the test suite doesn't call it.

AFAIK, it all boils down to the modperl_xs_sv2request_rec function not being available if your code is not running inside Apache with mod_perl loaded. So, it makes sense to run the tests with PERL_DL_NONLAZY=0 (that is, be lazy!), like the Japanese guy suggests.

However, changing that in the Makefile incantations seems to be a Royal Pain, so I just gave prove a shot. And once more, I had the proof that prove is just the way to run your test suite, because prove seems to be smart enough to set PERL_DL_NONLAZY automatically to zero.

So now, I'm running prove through a shell script, like this:

...
BASE_DIR=<project_root>
...
export CONFIG_FILE=/etc/blah-blah
export WHATEVER_ELSE=...
...
prove -I $BASE_DIR/lib $* $BASE_DIR/t/unit-tests

And that's it, all tests successful! o/

Communities in Action 2010 in Oslo

Last Monday, 10th of May, the regular Oslo.pm (Oslo Perl Mongers) meeting was a little special. In fact, there wasn't any Oslo.pm meeting, but we went to a free, one evening mini conference sponsored by several norwegian companies: "Communities in Action".

The basic idea was to put together several different communities in the Oslo area, so the conference program was diverse and exciting. There were 7 different tracks. Among them, the most interesting for me, apart from Oslo.pm, were:

  • XP (extreme programming, agile, etc…)
  • scalabin (about the scala language)
  • OWASP
  • Oslo C++ users group

Other tracks were by the NNUG, Norwegian .NET users group and IASA, some norwegian association about something…

There were a couple of colleagues from Opera, and other guys I know from Oslo.pm, but being there alone, I couldn't follow more than one track, so I tried jumping a bit between different rooms. I found the XP talk a bit boring, so I settled on the Oslo.pm track. There was a talk on rakudo * by Karl Rune Nilsen and a talk about meta-object programming in Perl vs Ruby by Matt Trout.

Both talks rocked. I think I had attended a very similar rakudo talk before, but the Ruby one was entirely new. It was funny when Matt, just before starting, tried to attract Ruby folks screaming and shouting
throughout the hotel hall :-)

After the conference, a small group of us gathered and headed over to tilt, a nice pub/pinball place, where I ended up making the day record even if I suck at pinballs…

A couple of bugs fixed on the varnish accept-language VCL extension

Today I received a report of a bug in how the Opera.com startup page was presented to some browsers. In particular, people with Brazilian language.

This was traced down to being a bug in the accept-language.vcl extension that we are using on both My Opera and Opera.com.

In particular, the bug had to do with static buffers being reused and overwritten during the Accept-Language string parsing, essentially due to my non-optimal C skills :) Now this bug is fixed and new test cases were added, so to hopefully avoid regressions in the future.

So now clients with Accept-Language containing strings like pt-BR, es-ca, zh-CN, etc… will be correctly recognized by the VCL code, and they will get the correct page served by varnish.

As always, thanks to Opera, the code is freely available, and I have to say it's not really experimental dangerous stuff anymore. It's been quite proven in the last few months (these few bugs excluded). At least your Varnish server shouldn't explode :-) Have fun!