Tag Archives: development

Dependencies suck

We love dependencies. For example, in the CPAN universe. They make our job so damn easier. Thousands of production quality, unit tested modules at your fingertips.

But dependencies also suck really badly, for example when you're using a Linux distribution that has packages that are just too old to be useful. Hey, but they are stable! More stable-as-dead or more stable-as-production-quality? You decide.

It's been many months since I installed a local instance of Transifex, a Django application that allows translators to easily contribute to projects. We're using it for My Opera, but also trying to get other internal projects to use it.

So far, it has worked nicely. I think Transifex is a really good application, its feature set is just right for what we need etc… Last week I decided to upgrade our Transifex instance from v0.8.0-devel to 0.9.0-devel. The improvements were really nice and needed, so I just decided to go for it. I had been upgrading in the 0.8.0 series from their repository (aka HEAD, aka master, aka trunk).

This time though, the list of dependencies was a bit more specific than usual. Also, please note that 0.9.0 is a **BLEEDING EDGE** development version as of June 2010.

Anyway, first dependency listed was "Django = 1.1.2". I think I started going down the wrong path when I upgraded Django with:


$ sudo easy_install 'Django>=1.1.2'

Here you can see that my mind is somewhat hardwired to the Perl culture, where backward compatibility is of paramount importance. I wrote code 10 years ago, using perl 5.005, that it's still in production, unmodified, with perl 5.10, and I'm talking about commercial stuff, not silly home projects. The terrible mistake here is to think that this also applies everywhere else. Forget it. It's not true.

In fact, easy_install picked up Django 1.2.1, which is an entirely different beast that breaks at least a couple of assumptions that Transifex was making. I don't remember exactly now, but one had to do with the automatic export of email.MIMEBase into django.core.mail and another I only remember it broke horribly.

So, a couple of hours later, thanks to the guys on the #transifex channel, I figured out that what I really needed to write was:


$ sudo easy_install 'Django==1.1.2'

This forces to install the given version instead of any later one. So far so good. Then I had another problem, completely unrelated, the required me to strace the ./manage.py Django script, to figure out that it was using a totally screwed up sqlite database coming from a year old test version of transifex I had installed through easy_install and was completely ignoring my local settings that went to a MySQL db. How nice.

So, yes, we always complain about CPAN, dependencies, Module::Install, ExtUtils::MakeMaker and whatnot, but a look at other worlds (easy_install, ruby-gems anyone?) can remind Perl people of the fantastic toolchain and especially culture "we" have built, and that's still kicking everyone else's ass, on any platform.

So, regarding the debate in the Perl community, my vote goes to keeping Sane(tm) backward compability standards, as we always did. It matters, especially for commercial software companies!

Disassembling a real world Plack PSGI application

After I started playing with Plack, I tried to evaluate whether to continue using it for our mission-critical production stuff or give up, going back to the same techniques we already use (successfully).

I think it's time to develop and deploy a Plack based application. In my grand plan, :-), I'd like to deploy nginx with PSGI support, or even more ambitiously, nginx or apache with Starman as "backend" http server. We'll see…

In the meantime, I'd like to write here a couple of niceties about Plack and Starman, showing some real code I wrote when I started.

A real world PSGI application

Here's a sample PSGI application currently under development:

#!/usr/bin/env perl
#
# Sample PSGI application
#

use strict;
#se warnings;
use constant ENVIRONMENT         => 'development';
use constant APACHE_DEPLOYMENT   => (ENVIRONMENT eq 'production');
use constant ENABLE_ACCESS_LOG   => (ENVIRONMENT eq 'development');
use constant ENABLE_DEBUG_PANELS => (ENVIRONMENT eq 'development');

use Plack::Builder;
use AuthOpera;
use AuthOpera::Account;

my $app = AuthOpera::Account->new(); 

builder {

    enable_if { not APACHE_DEPLOYMENT }
        'Plack::Middleware::Static', 
        path => qr{^/(bitmaps/|images/|js/|css/|downtime/|favicon.ico$|ping.html$)},
        root => '..',
        ;

    mount "/account" => builder {

        enable_if { ENABLE_DEBUG_PANELS } 'StackTrace';
        enable_if { ENABLE_DEBUG_PANELS } 'Debug';   # panels => [ qw(DBITrace Memory) ];
        enable_if { ENABLE_DEBUG_PANELS } 'Lint';
        enable_if { ENABLE_DEBUG_PANELS } 'Runtime';
        enable_if { ENABLE_ACCESS_LOG   } 'AccessLog';

        $app;

    }

}

Of course, the main application code is not here, but in the AuthOpera::Account class. That's not really relevant to what we're discussing here. Let's just say that any class, to be a valid and complete PSGI application, has to:

  • subclass from Plack::Component
  • have a call() method
  • the call() method must return a valid PSGI response. Example:
    package MyPSGIApp;
    
    use strict;
    use Data:: Dumper ();
    use parent 'Plack::Component';
    
    sub call {
        # $env is the full PSGI environment
        my ($self, $env) = @_;
    
        return [
    
            # HTTP Status code
            200,
    
            # HTTP headers as arrayref
            [ 'Content-type' => 'text/html' ],
    
            # Response body as array ref
            [ '<!DOCTYPE html>',
              '<body><h1>Hello world</h1><pre>',
              Data:: Dumper:: Dumper($env),
              '</pre></body></html>',
            ],
        ];
    }
    
    1;
    

That's it, this is a full PSGI application that does dump all its PSGI environment.

Of course in a real example, you probably want a template engine to return the page content, etc… That's what we are building for our applications. Actually just assembling the components we already have developed during these years, so we have template classes, config classes, localization, database access, etc…

So we're basically just gluing these ready made components inside the PSGI application, and then using them. I don't think this is particularly original, but it allows us to quickly "port" our code to PSGI and thus run anywhere we want to.

app.psgi in detail

Now, let's see the PSGI app in more detail.

use constant ENVIRONMENT         => 'development';
use constant APACHE_DEPLOYMENT   => (ENVIRONMENT eq 'production');
use constant ENABLE_ACCESS_LOG   => (ENVIRONMENT eq 'development');
use constant ENABLE_DEBUG_PANELS => (ENVIRONMENT eq 'development');

These constants are used to turn on and off certain features mentioned later in the builder {} block. I just found out the other day that these constants are near to useless. That is because plackup and starman already provide a -E environment switch. If you start your application with:

starman -E development myapp.psgi     # same with plackup, the default server

then Plack will by default enable the debugging panels and the Apache-style access log. I found out about this after having written that file. This means that the following enable_ifs are unnecessary:

mount "/myroot" => builder {
    enable_if { ENABLE_DEBUG_PANELS } 'StackTrace';
    enable_if { ENABLE_DEBUG_PANELS } 'Debug';   # panels => [ qw(DBITrace Memory) ];
    enable_if { ENABLE_DEBUG_PANELS } 'Lint';
    enable_if { ENABLE_DEBUG_PANELS } 'Runtime';
    enable_if { ENABLE_ACCESS_LOG   } 'AccessLog';
    $app;
}

I think Plack enables by default at least StackTrace, Debug, and AccessLog. In my case, however, I'm also enabling RunTime and Lint. But more importantly, I need to differentiate between Apache deployment and Starman deployment. That affects the way static files are served.

When deploying under Apache, I don't need the following:

enable_if { not APACHE_DEPLOYMENT }
    'Plack::Middleware::Static',
    path => qr{^/(bitmaps/|images/|js/|css/|downtime/|favicon.ico$|ping.html$)},
    root => '..';

because my PSGI application is enabled in an Apache <Location> block, as in:

<Location /myroot/>
    SetHandler perl-script
    PerlResponseHandler Plack::Handler::Apache2
    PerlSetVar psgi_app /my/path/to/app.psgi
</Location>

So Apache already takes care of serving the static files for me. However, when running completely under Starman, I need to tell it which folders or paths need to be served as static files, and where they are located. This is the purpose of the Static middleware:

enable_if { not APACHE_DEPLOYMENT } 'Plack::Middleware::Static',
    path => qr{^/(images/|js/|css/|favicon.ico$)},
    root => '/var/www/something';

If you're always deploying through plackup or starman, then, again, you don't need any enable_if, just enable. Maybe it's also a good idea to put everything under /static. For me that wasn't possible, since I already had existing content:

enable 'Plack::Middleware::Static',
    path => qr{^/static/},
    root => '/var/www/something';

Plack::Builder

About the Plack::Builder bit, and the related builder function. That is a function that helps you specify what you want Plack to run and how. Example:

builder {
    enable 'StackTrace';
    enable 'Debug';
    enable 'AccessLog';
    $app
}

where StackTrace, Debug, and AccessLog are all middleware classes, so causes Plack to wrap your final $app application first with the AccessLog middleware, then Debug and then StackTrace. I didn't check the code, but I believe this creates 3 different PSGI applications that are meant to fiddle with the response that your own application generates.

PSGI makes this possible, and it's just great. More middleware means easier and faster development. And ultimately, very good middleware makes for great reuse too.

The mount wrapper

I used mount in my example very basicly, but you can use mount to assemble compounds of applications in a very simple way. The same thing you do, for example, with Django and urls.py, except that, if you have seen a non-trivial urls.py, it looks like spaghetti after a while. Compare with this:

my $app1 = MyApp->new();
my $app2 = MyApp2->new();
#...

builder {

    enable 'Plack::Middleware::Static', 
        path => qr{^/static/},
        root => '/var/www/something';

    mount "/path1" => builder {
        enable 'StackTrace';
        $app;
    }

    mount "/path2" => $app2;

    mount "/path3" => builder {
        enable 'SomeMiddleware';
        $app3;
    }

}

Of course, then you have to add some dispatcher logic to your applications, but in the Plack world, we don't lack good dispatchers.

Plack rocks.

Plack, the grand glue

I have been looking at Plack for several months now. I always thought it was a cool project, where "cool" means useful, good code, nice documentation, well structured, strong development "flow", etc…

Lately I've been "rebooting" an internal project, doing a lot of infrastructure work like deployment tools, management of different environments like devel, test, staging and production, bug fixing, etc… Regular stuff that you usually already take for granted, but this project didn't have, for many reasons.

After this rebooting work, time has come to add new features to the web-facing part of this project, so new pages and forms. My frustration came from mainly two factors:

  • the code being tightly integrated with mod_perl
  • the need to change the apache config every time you add a new /something to the application

While the mod_perl integration is not necessarily a bad thing, and mod_perl is a fast and reliable product serving millions of pageviews per day, it's also nice to have code that you can run anywhere, not just on Apache. That might or might not happen, but I'd want to be ready when that's needed. In fact, we're starting to use nginx on some applications, including My Opera

So I decided to invest some time to play with Plack. Plack transposed to Perl concepts from Ruby's Rack and WSGI, a spec born in the Python world I believe.

I'm still at the first experiments with it, but Plack is a great software with a spectacular potential! If you're in doubt, try it for yourself. What made me decide to try Plack was Starman.

Starman is a damn fast PSGI-enabled preforking HTTP web server written in Perl. As soon as I started starman with a stupid simple app.psgi I realized I had to invest some time in it.

A couple of days later I have the slick opera.com design into a bunch of Template Toolkit blocks, with all machinery to make them work for real, and a PSGI application class. From the screenshot you can see this request is loaded in 38 ms, and this my desktop machine, and the Debug middleware gives you useful debugging info through sidebar panels.

I can run this class either as standalone with Starman, or in Apache + mod_perl with Plack::Handler::Apache2 or anything else for that matter, like FastCGI, or even plain CGI if you want that.

And I think that shows:

  • how cool the PSGI/WSGI concept is
  • that you can really code once, run anywhere, and don't care about the web server stack

The “Gran Torino” of keyboards…

My keyboard

It's not the best keyboard for everyone of course, but it's the best for me. Totally awesome personalized keyboard. It's fully supported by my window manager. I don't understand why everyone coming at my desk refuses to write on it…

It's more than 10 years old. It has served me very well, and has been cleaned extensively 3 or 4 times with full disassembly. Here's my personal ritual. Every day when I'm finished working, I cover it from dust with a special cloth. This cloth has been covering my keyboards since I had my first C64, then C128, Amiga and now this one. :-)

Full picture here.