Monthly Archives: May 2010

A working bogosort in Perl6

It's not even mine actually. @moritz helped me with that. I couldn't remember the syntax for the reduce operator (and it's [<op>]), so here it is, in its full wikipedia glory.

@list .= pick(*) until [<=] @list

Here's a somewhat more complete example, that you can actually run:

my @list = 1, 2, 7, 4, 3;
@list .= pick(*) until [<=] @list;
say @list.perl;

This will bogosort the array (wikipedia), that is, shuffle it until it's ordered. Look at the Java version for a nice comparison.

Here's the explanation:

my @list = 1, 2, 7, 4, 3;

Declares the list. You don't need parens around the elements. If that was a deck of cards, you could write it as:

my @list = <A 2 3 4 5 6 7 8 9 J Q K>

The angular brackets <> are the equivalent of qw() in Perl 5. Following line is quite dense:

@list .= pick(*) until [<=] @list;

This can be written also as:

while not [<=] @list {
  @list .= pick(*)
}

As you can see, you don't need parens around the while condition. The [<=] @list bit is IINM a reduce operator for a list. For example, to sum all elements of a list, you'd use:

my @num = 1, 2, 3, 4;
say [+] @num;

So the [<=] operator applied to a list compares each element with the next one, then the result with the next one. Assuming @list = 2, 4, 8, 5, that is how I think it runs:

2 <= 4 (true, go on, last evaluated value is 4)
4 <= 8 (true, go on, last value is 8)
8 <= 5 (false, stop here, return value is false)

So, if the array is ordered, the while/until block is skipped. If the array is not ordered, execute the following:

@list .= pick(*)

.= is an operator for in-place method call. Writing that is equivalent to:

@list = @list.pick(*)

That in Perl 5 would be written as:

$list = SomeListObject->new(1, 2, 3, 4, 5);
$list->pick(???);

Actually, there wouldn't be any Perl 5 direct equivalent code, since the magic bit here (the *, aka "whatever") it's not even there. "whatever" is magic for "anything that's applicable here, until there's an end to it", or at least this is my ignorant interpretation of it… :)

So in this context, pick() is a method that returns a random element from a list. Together with *, it means pick a random element from any of the original @list. Mmmh. But then it should be equivalent to:

@list .= pick(@list)

But it's not, so pick(*) knows the right thing to do…
And it's clear that I don't know how pick() works :)
Anyway…

BTW, this is runnable code. If you want to try it, clone the rakudo repository, and run perl Configure.pl --gen-parrot to build it. Have fun!

How to generate an XML document with XML::LibXML

For the "note to self" series, or… if I ever need to do this again, I will find this here, here's how to generate an XML document with XML::LibXML. I usually use XML::Simple or XML::TreePP, but this script was already requiring XML::LibXML to parse some XML fragments, so I thought I would use the same module. It wasn't really immediate…

#!/usr/bin/env perl

#
# Create a simple XML document
#

use strict;
use warnings;
use XML::LibXML;

my $doc = XML:: LibXML:: Document->new('1.0', 'utf-8');

my $root = $doc->createElement("my-root-element");
$root->setAttribute('some-attr'=> 'some-value');

my %tags = (
    color => 'blue',
    metal => 'steel',
);

for my $name (keys %tags) {
    my $tag = $doc->createElement($name);
    my $value = $tags{$name};
    $tag->appendTextNode($value);
    $root->appendChild($tag);
}

$doc->setDocumentElement($root);
print $doc->toString();

and this outputs:

<?xml version="1.0" encoding="utf-8"?>
<my-root-element some-attr="some-value">
    <color>blue</color>
    <metal>steel</metal>
</my-root-element>

And a copy is also archived at stackoverflow :)

Plack, the grand glue

I have been looking at Plack for several months now. I always thought it was a cool project, where "cool" means useful, good code, nice documentation, well structured, strong development "flow", etc…

Lately I've been "rebooting" an internal project, doing a lot of infrastructure work like deployment tools, management of different environments like devel, test, staging and production, bug fixing, etc… Regular stuff that you usually already take for granted, but this project didn't have, for many reasons.

After this rebooting work, time has come to add new features to the web-facing part of this project, so new pages and forms. My frustration came from mainly two factors:

  • the code being tightly integrated with mod_perl
  • the need to change the apache config every time you add a new /something to the application

While the mod_perl integration is not necessarily a bad thing, and mod_perl is a fast and reliable product serving millions of pageviews per day, it's also nice to have code that you can run anywhere, not just on Apache. That might or might not happen, but I'd want to be ready when that's needed. In fact, we're starting to use nginx on some applications, including My Opera

So I decided to invest some time to play with Plack. Plack transposed to Perl concepts from Ruby's Rack and WSGI, a spec born in the Python world I believe.

I'm still at the first experiments with it, but Plack is a great software with a spectacular potential! If you're in doubt, try it for yourself. What made me decide to try Plack was Starman.

Starman is a damn fast PSGI-enabled preforking HTTP web server written in Perl. As soon as I started starman with a stupid simple app.psgi I realized I had to invest some time in it.

A couple of days later I have the slick opera.com design into a bunch of Template Toolkit blocks, with all machinery to make them work for real, and a PSGI application class. From the screenshot you can see this request is loaded in 38 ms, and this my desktop machine, and the Debug middleware gives you useful debugging info through sidebar panels.

I can run this class either as standalone with Starman, or in Apache + mod_perl with Plack::Handler::Apache2 or anything else for that matter, like FastCGI, or even plain CGI if you want that.

And I think that shows:

  • how cool the PSGI/WSGI concept is
  • that you can really code once, run anywhere, and don't care about the web server stack

Wiimote Perl hacks

The wiimote is a geek toy. Here's what I did with it so far.

3D Earth

I wrote this OpenGL program in Perl more than a year ago. It can display Apache access logs in real time, as geolocated spikes on the rotating planet (if your machine is powerful enough :). It was a fun way for me to learn tiny bits of OpenGL, enough to run it. I thought it would be cool to use the wiimote to control the planet, twisting and spinning the Earth with your hands. It worked pretty well, and I'm really satisfied with the result.

Fast forward…

During last couple of weeks, I managed to find some spare time to play with the wiimote again. Here's the result, the wiimote tools.

Wiimote tools

Tools it's probably not the right word. It's small hacks. It 3 different hacks in one box. I know there's already stuff out there that let you do everything like this, but honestly I don't care at all. The interesting part is writing something, preferably in Perl, to do that myself. And if my kids can have fun with it, it's the best a father-hacker can hope for… :)

Copy-paste from the README follows:

wiimote-drumkit

This is something I made for my small kids, so they could just play with sounds. You can write a small config file to assign random mp3s/wav/… to each button of the wii.

Take a look at drumkit.conf or animals.conf as examples.

Then every time you press a button, or wave the wii in the air (pretending you're playing drums), you will hear the corresponding sounds. It requires 'paplay', the Pulse audio command line player. My daughter almost managed to fork bomb my PC pressing buttons everywhere :)

wiimote-joystick

This is a quick hack that I put together to use the wiimote together with any of my favorite games as a digital joystick.

I can use it to play with mame (see pic), of course with legal roms or c64 emulators (with games I of course owned when I was a kid) and so on. Currently it works well with a few games, because of the way the keys are sent to X11, and it works really bad with other games, because of the way the keys are sent to X11. :)

Requires the X11::GUITest module, install with sudo cpan X11::GUITest.

wiimote-conductor

This is the stupidest of the lot. It can control the speed of a movie (or song?) by pretending to use the wiimote like a drunk orchestra conductor would.

It's a bit sucky because it doesn't get the actual tempo from the sound, so you have to pretend the normal (1.0x) speed is represented by wii-beats distanced by 1 second. So if you want to double the speed of the movie/song, you have to wave the wii in the air every 0.5 seconds. Pretty lame, eh? :)

Requires Audio:: Play:: MPlayer with a trivial patch I made to add the speed command.

Everything is nicely packaged here, if you want to experiment with it.

… and this is for mst, have fun!

Puppet, Fabric and a Perl alternative?

Some time later this month I'm going to write more extensively about a project that I've been working on, not continuously, but for the last couple of months. It is about small and medium scale projects configuration management and deployment.

For configuration management I evaluated several products like bcfg2, puppet, cfengine and lcfg, and I finally chose puppet.

For "the last mile", as I call it, the alternatives that I considered were fabric, capistrano, ControlTier and TheNewShinyWheel(tm)

So I settled on puppet + fabric. Puppet is a Ruby system, while Fabric is Python code. None of them is particularly fast, actually Puppet is slow, and Fabric is acceptable. The main problem I'm confronting with, after having learnt how to use these tools, is that Fabric does not support parallel processing of tasks.

This is a severe limitation for us. This was a pilot project. If it works well, it could be applied to many other deployment tasks. That could also mean that a single deployment has to send code or files to tens of servers, and you don't want to do that sequentially waiting for each task to complete.

At the moment, this is impossible to do with Fabric. There is an experimental fork in the works that might support parallel execution, by adding a @parallel task decorator, but it still requires work and a good dose of testing.

During my survey I looked for mature Perl-based deployment tools, but I failed at finding them. While Fabric is nice, I might be tempted to reconsider my choice. Any suggestions?

Undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193

Today I was trying to run some Test::Class-based unit tests for a project I'm working on. This project has lots of functional tests that go through Test::WWW::Mechanize, so they run inside Apache and mod_perl.

What I was trying to do was testing the same code outside the Apache server, just as regular command-line-runnable tests. Being a Perl project, you have all the nice infrastructure already done for you, and so you can just run make test or Build test depending on the exact tool you're using.

However, when I run this, I got a strange error:

PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/unit-tests/basic-sanity.t .. ok
t/unit-tests/handler-xml.t ... Can't load '/usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so' for module APR::Request::Apache2: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193.
 at /usr/lib/perl5/Apache2/Request.pm line 3
Compilation failed in require at /usr/lib/perl5/Apache2/Request.pm line 3.
BEGIN failed--compilation aborted at /usr/lib/perl5/Apache2/Request.pm line 3.

The weird error is:

Can't load '/usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so' for module APR::Request::Apache2: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: modperl_xs_sv2request_rec at /usr/lib/perl/5.10/DynaLoader.pm line 193.

and specifically, undefined symbol: modperl_xs_sv2request_rec.
Why is this happening?

A search for this error message gives back very few results, and not very useful. However, this more specific search turns up fewer results, but that led me to the solution.

The first hit, in particular, I have no idea what that page says, because it's Japanese, but the code blocks are clearly referring to the PERL_DL_NONLAZY environment variable:

PERL_DL_NONLAZY

    Set to one to have perl resolve all undefined symbols when it
    loads a dynamic library.  The default behaviour is to resolve symbols
    when they are used.

    Setting this variable is useful during testing of extensions as it
    ensures that you get an error on misspelled function names even if
    the test suite doesn't call it.

AFAIK, it all boils down to the modperl_xs_sv2request_rec function not being available if your code is not running inside Apache with mod_perl loaded. So, it makes sense to run the tests with PERL_DL_NONLAZY=0 (that is, be lazy!), like the Japanese guy suggests.

However, changing that in the Makefile incantations seems to be a Royal Pain, so I just gave prove a shot. And once more, I had the proof that prove is just the way to run your test suite, because prove seems to be smart enough to set PERL_DL_NONLAZY automatically to zero.

So now, I'm running prove through a shell script, like this:

...
BASE_DIR=<project_root>
...
export CONFIG_FILE=/etc/blah-blah
export WHATEVER_ELSE=...
...
prove -I $BASE_DIR/lib $* $BASE_DIR/t/unit-tests

And that's it, all tests successful! o/

Communities in Action 2010 in Oslo

Last Monday, 10th of May, the regular Oslo.pm (Oslo Perl Mongers) meeting was a little special. In fact, there wasn't any Oslo.pm meeting, but we went to a free, one evening mini conference sponsored by several norwegian companies: "Communities in Action".

The basic idea was to put together several different communities in the Oslo area, so the conference program was diverse and exciting. There were 7 different tracks. Among them, the most interesting for me, apart from Oslo.pm, were:

  • XP (extreme programming, agile, etc…)
  • scalabin (about the scala language)
  • OWASP
  • Oslo C++ users group

Other tracks were by the NNUG, Norwegian .NET users group and IASA, some norwegian association about something…

There were a couple of colleagues from Opera, and other guys I know from Oslo.pm, but being there alone, I couldn't follow more than one track, so I tried jumping a bit between different rooms. I found the XP talk a bit boring, so I settled on the Oslo.pm track. There was a talk on rakudo * by Karl Rune Nilsen and a talk about meta-object programming in Perl vs Ruby by Matt Trout.

Both talks rocked. I think I had attended a very similar rakudo talk before, but the Ruby one was entirely new. It was funny when Matt, just before starting, tried to attract Ruby folks screaming and shouting
throughout the hotel hall :-)

After the conference, a small group of us gathered and headed over to tilt, a nice pub/pinball place, where I ended up making the day record even if I suck at pinballs…

A couple of bugs fixed on the varnish accept-language VCL extension

Today I received a report of a bug in how the Opera.com startup page was presented to some browsers. In particular, people with Brazilian language.

This was traced down to being a bug in the accept-language.vcl extension that we are using on both My Opera and Opera.com.

In particular, the bug had to do with static buffers being reused and overwritten during the Accept-Language string parsing, essentially due to my non-optimal C skills :) Now this bug is fixed and new test cases were added, so to hopefully avoid regressions in the future.

So now clients with Accept-Language containing strings like pt-BR, es-ca, zh-CN, etc… will be correctly recognized by the VCL code, and they will get the correct page served by varnish.

As always, thanks to Opera, the code is freely available, and I have to say it's not really experimental dangerous stuff anymore. It's been quite proven in the last few months (these few bugs excluded). At least your Varnish server shouldn't explode :-) Have fun!

embed.ly supports My Opera urls

On My Opera, we started using the OEmbed specification a long time ago. The goal was to provide an easy way to get metadata information, for example from albums or pictures. Example: given a picture URL on My Opera:

http://my.opera.com/365/albums/showpic.dml?album=2721921&picture=42077711

we wanted to give our users a way to extract metadata information about that URL in an easy way. That's what OEmbed is about. So you have the following OEmbed API url:

http://api.my.opera.com/service/oembed?url=http%3A//my.opera.com/365/albums/showpic.dml%3Falbum%3D2721921%26picture%3D42077711

and the result of an HTTP GET to that URL is:

{
   "width" : "3443",
   "author_name" : "giulia-maria",
   "author_url" : "http://my.opera.com/giulia-maria/",
   "provider_url" : "http://my.opera.com/",
   "version" : "1.0",
   "provider_name" : "My Opera Community",
   "height" : "2293",
   "url" : "http://files.myopera.com/giulia-maria/albums/2721921/DSC_5396.JPG",
   "title" : "April 27, 2010. Pink jasmine or pink lilac?",
   "type" : "photo"
}

So that gives you some useful metadata if you need to embed the picture in a page of yours, or even within a widget. Various sites have been supporting OEmbed for many years now. You can see a brief list on oembed.com. However, now there's a new service called embed.ly that works as a oembed "hub", as it transparently supports URLs from many different websites.

And that includes My Opera too! Try it here:

http://api.embed.ly/embed?url=http%3A//my.opera.com/365/albums/showpic.dml?album=2721921%26picture=42077711

or try it with your favourite site.