Monthly Archives: August 2011

Internationalization (i18n) with Mojolicious and Template Toolkit

In a previous post I talked about this new Mojolicious-based application that I've been working on, that btw was rolled out in production today (yay!)

Classic I18N with TT

One of the required features of this app was "i18n", internationalization. To be less vague, the requirement was to present the UI in different languages. We're using Template Toolkit, so our templates need to have strings marked in a special way to allow translation to kick-in at run-time. Usually in TT you do this with:

<html>
 
<head>
<title>[% l('This is the title of the page') %]</title>
</head>
 
<body>
<h1>[% l('Hello, world!') %]</h1>
<p>
[% l('Some text here') %]
</p>
</body>
</html>

so all the strings that have to be translated according to the user language have to be marked up with:

[% l('<your string here>') %]

Enter Mojolicious

Mojolicious includes a built-in I18N plugin that simplifies your life allowing the <% l('somestring)' %> syntax to work. That is, it gives you a l() helper.

Helper?

A helper is a method that it's available both as part of your controller object, and within templates.

Back to Mojolicious…

In the example helper syntax I wrote <% l('somestring)' %> because that's Mojolicious default templating system syntax. However, under Template Toolkit, you can't use that syntax! You have to pass through an extra level, as in:

<!-- This is my TT template -->
[% c.l('<your string here>') %]

I'm not exactly sure why that c. is required, but that's how it is.

I18N workflow: extracting the strings

Everything would be fantastic, except there's one tricky problem. After you worked so hard on your TT templates, now it's time to collect all the marked up strings, presumably to build a .PO file to be shipped to translation agencies or whatever system you're using for that. More on that later.

In the Perl world, there is an equivalent of GNU xgettext, which is xgettext.pl. This tool is part of the Locale::Maketext::Lexicon CPAN distribution, which is kind of "the standard" way to i18n in Perl. Or it is for us here anyway since we started building i18n for my.opera.com in 2008.

The tricky problem is that even though xgettext.pl understands quite a few syntax variants, it didn't understand [% c.l('string') %]. After a few Perl debugger sessions, I managed to teach Locale::Maketext::Extract::Plugin::TT2 how to parse Mojolicious-style syntax. I knew that Clinton Gormley, the maintainer of L::M::Lexicon had a source repository for it on Github, so I forked his repository and pushed my changes on a dedicated branch.

CPAN, Github and the Community

This is where the Github + CPAN model really shines. You're using a CPAN module. You stumble on a problem. Fix the problem. Find its repository on Github. Fork it, push your fix, and if you're lucky, you have your fix merged and out on CPAN the same day.

This is what actually happened. Clinton got in touch the very same day I sent him the pull request and later pushed out the changes on CPAN. If you ask me, that's just awesome. I wish everything worked that way :)

Closing the i18n workflow

Fixed the c.l() problem, everything else was easier. xgettext.pl allows you to collect strings from your code and templates and build a master .PO file with all the strings. Then msgmerge, a standard GNU gettext tool, allows you to take the generated master PO file and merge it with any existing language-specific PO if any. If you don't have any, just copy the master PO file (usually called POT, or reference PO file) to <language>.po and start translating.

Last step is either:

  • compiling the .po files to .mo, a lookup-optimized form of the .po file
  • creating the "lexicon" files. In the Perl world, these are nothing more than Perl modules with a %Lexicon hash that contains all string IDs and their translations

We're long time fans of the latter approach, so our lexicon files look like this:

package AuthOpera::Locale::it;
 
use strict;
use utf8;
use base qw(AuthOpera::Locale);
 
### LEXICON STARTS HERE (don't remove this line)
our %Lexicon = (
 
    # Automatic fallback to string ID when no translation available
    _AUTO => 1,
 
    # String IDs                  # Translations
    "Application name:"        => "Nome dell'applicazione:",
    "Application registration" => "Registrazione dell' applicazione",
    "Data provider:"           => "Provider dei dati:",
 
    # ...
);
### LEXICON ENDS HERE
 
1;

and we use a simple subclass of Locale::PO to read the PO file in memory and write back a lexicon based on a fixed template, hence the ### LEXICON lines above.

Transifex

Currently we also use Transifex, that allows to have external translators contribute to PO files directly from a web page, and if you configure it to do so, commit straight to your source code repository. You can then trigger automated builds of the lexicon files, having completed the full i18n workflow.

I find this system pretty simple but at the same time fully automated and very powerful. I'd love to hear comments or feedback about this stuff, especially from people adopting a different process.

Perl6 LWP::Simple now uses the URI module for added awesome

My little LWP::Simple module for Perl 6 is growing beyond my control. I just merged a a pull request with a patch to remove all existing URL parsing to replace it with the new URI class.

That's fantastic, because:

  • it reduces the amount of code in LWP::Simple
  • the URI module is based on the actual grammar for URIs and IPv6 addresses. Plain and simple. It doesn't adhere to the standard. It is the standard. Isn't that amazing?

So, just clicky clicky and the pull request was merged. Running the test suite confirmed that everything still works fine passing all tests. Can it get much better than this? :)

If you want to have a look at Perl 6 code, modules.perl6.org is the place to go for inspiration. Have fun!

Using Template Toolkit with Mojolicious

For an upcoming project, I decided to try and use Mojolicious in production. That would be the first time, so I'm quite excited to see what's going to happen.

A few days ago I wrote some sample application that just loads a basic Template Toolkit template and renders it, and benchmarked it using both:

  • mod_perl and Plack::Handler::Apache2 and,
  • using starman as self-contained HTTP server running the psgi application

I have to say that I was quite impressed with the performance level of Starman. I got 1,000+ (a thousand plus) requests per second without the server even breaking a sweat. The command line, just in case, was:

starman --workers 32 ./app.psgi

Anyway, back to using TT. I found myself searching for recipes on how to use TT with Mojolicious because there wasn't a clear documented answer on how to do it, or at least I didn't find it. An example of what I came up with follows.

Step 1: the Mojolicious application class

First you have to create your application class. You should probably use the script that generate the basic skeleton for you. There's nice documentation on how to do that. My class looks like this:


package My::PSGI::App;

use strict;
use base 'Mojolicious';
use Opera::Config;

sub startup {
    my $self = shift;
    
    $self->secret('some-secret-random-string');

    # Our internal configuration system
    my $conf = Opera::Config->new();
    my $tmpl_dir = $conf->get('Template:include_dir');
    my $cache_dir = $conf->get('Template:cache_dir');

    # Tell Mojolicious we want to load the TT renderer plugin
    $self->plugin(tt_renderer => {
        template_options => {
            # These options are specific to TT
            INCLUDE_PATH => $tmpl_dir,
            COMPILE_DIR => $cache_dir,
            COMPILE_EXT => '.ttc',
            # ... anything else to be passed on to TT should go here
        },
    });

    $self->renderer->default_handler('tt');

    my $r = $self->routes;

    # Your routes should go here
    $r->route('/login')->to('account#login');
    # ... and so on ...

}

1;

To have your TT templates picked up, you only need a few more things.

Mojolicious::Plugin::TtRenderer

When you declare that you want to load the tt_renderer plugin (see above, $self->plugin(tt_renderer=>...)), then Mojolicious will "camelize" the tt_renderer string, turn it into Mojolicious::Plugin::TtRenderer, and try to load that plugin, if available.

Turns out there was a MojoX::Renderer::TT CPAN module that also contained a class called Mojolicious::Plugin::TtRenderer. I said there was because Sebastian Riedel, the main developer of Mojolicious had in the meantime deprecated the MojoX namespace.

Since we're building the modules we want to use in production as deb packages, we would have run the risk to package MojoX::Renderer::TT to have it changed later because of this namespace conflict. To avoid this, I decided to fork its repository and put together a patch to remove the use of the MojoX:: namespace. With this, I hoped to get the thing done and hopefully picked up quickly by the maintainer of MojoX::Renderer::TT.

Turned out that he was super responsive (thanks Ask!) to merge the change and release it to CPAN, so ladies and gentlemen, I hereby announce we have Mojolicious::Plugin::TtRenderer 1.20+ out!

In fact, the old deprecated MojoX:: module is still there, just don't use it, and install Mojolicious::Plugin::TtRenderer instead.

Templates naming

Another thing you need for TT to work out of the box is that your templates should(*) be named sometemplate.html.tt. (*) probably you can deviate from this convention, I just don't know yet.

Your controller should specify TT as the renderer

UPDATE: this is not needed. If you're using:

$self->renderer->default_handler('tt');

in your main application class, then you won't need to specify format and handler in every controller.

Again, not sure it's really needed (no, it's not, read above), check before you copy/paste. Here's a simple action from one of my controllers (following the previous example):


package My::PSGI::App::Account;
 
use strict;
use base 'Mojolicious::Controller';
 
sub login {
    my $self = shift;
    
    $self->render(
        template => 'path/to/template', # *without* .html.tt
        format   => 'html',
        handler  => 'tt',
    );
    
}

1;

That should be it: have fun!

EDIT: Thanks Robert for the suggestions.