Tag Archives: perl

Self-contained instant functional test suites

Leave a reply

I first learned about functional test suites when I started working on the My Opera code. Two years and some time later, I found myself slightly hating the My Opera functional test suite.

The main reasons:

  • It's too slow. Currently, it takes anywhere between 20 and 30 minutes to complete. Sure, it's thousands of tests, divided into hundreds of test scripts, grouped by functional area, like login, blogs, albums, etc…

    Even considered all of this, I think we should aim to have a single functional test run complete in 5-10 minutes.

    Most of the time is being wasted in the communication with the test server and creation and destruction of the database.

  • It's unreliable, sometimes cumbersome to manage. In our setup, CruiseControl fires a functional test suite run after every commit into the source code repository.

    That initiates an ssh connection to the test server, where a shell script takes care of restarting the running apache, dropping and creating the test database, updating required packages, etc…

    This implies that you need an apache instance running on a specific port, a mysql instance running on another port, etc… In the long run, this proved to be an approach that doesn't scale very well. It's too error prone, and not very reliable. Maybe your test run will fail because mysql has mysteriously crashed, or some other random bad thing.

A functional test suite that sucks less

Given the motivation, I came up with this idea of a functional test suite that is:

  1. instant: check out the source code from the repository, and you're ready to go.
  2. self-contained: it shouldn't have any external servers that need to be managed or even running.
  3. reusable: it doesn't have to be a functional test suite. The same concept can be reused for unit test suites, or anything really.

A few months later the first idea, and a couple of weeks of work on it, and we were ready with the first prototype. Here's how we use it:

  • Check out the source code from the repository in ~/src/myproject
  • cd ~/src/myproject
  • ./bin/run-functional-test-suite
  • Private instances of Apache, MySQL, and the main application are created and started
  • A custom WWW::Mechanizer-based client runs the functional test cases against the Apache instance
  • A TAP stream from the test run is produced and collected
  • All custom instances are destroyed
  • ???
  • Profit!

The temporary test run directory is left there untouched, so you can inspect it in case of problems. This is priceless, because that folder contains all the configuration and log files that the test run generated.

The main ideas

  • The entire functional test suite should run within a unique temporary directory created on the fly. You can run many instances as you want, with your username or a different one. They won't conflict with each other.
  • Use as much as possible the same configuration files as the other environments. If you already have development, staging and production, then functional-test is just another one of your environments. If possible, avoid creating special config files just for the functional test suite.
  • The Apache, MySQL, and Application configuration files are simple templates where you need to fill in the apache hostname, apache port, mysql port, username, password, etc…
  • Apache is started up from the temporary directory using the full command invocation, as:

    /usr/sbin/apache2 -k start -f /tmp/{project}_{user}_testsuite_{pid}/conf/apache.conf

    and all paths in config files are absolute, to avoid any relative references that would lead to files not found.
    The configuration template files have to take this into account, and put a $prefix style variable
    everywhere, like (Apache config, for example):

    Timeout 10
Keepalive On
HostnameLookups Off
...
Listen [% suite.apache.port %]
...

  • The MySQL instances are created and destroyed from scratch for every test suite run, using the amazing MySQL::Sandbox. My colleague Terje had to patch the sandbox creation script to fix a problem with 64-bit environments. We filed a bug on the MySQL::Sandbox RT queue about this. This is the only small problem found in a really excellent Perl tool. If you haven't looked at it, do it now.

All in all, I'm really satisfied of the result. We applied it to the Auth project for now, which is smaller than My Opera, and we fine-tuned the wrapping shell script to compensate for the underlyiung distribution automatically. We've been able to run this functional test suite on Ubuntu from version 8.04 to 10.04, and on Debian Lenny, with no modifications. The required MySQL version is detected according to your system, and MySQL::Sandbox is instructed accordingly.

It also results in a faster execution of the entire functional test suite.

What can we improve?

There's lots of things to improve:

  • Speed. Instead of creating and destroying the database for every test group, it would be nice to use transactions, and rollback everything at the end of the test group (t/{group-name}/*.t). I used to do this many years ago with Postgres, and it was perfectly safe. I'm not sure we will ever make it with MySQL, since for example, ALTER TABLE (and other SQL statements) completely ignores transactions.
  • Reliability. The private instances of Apache, MySQL (and recently we added memcached too), are started up from ports that are calculated from the originating bash script PID. So, if you run the test suite from the bash script running as PID 8029, then MySQL is started up on port 8030, Apache on 8031, memcached on 8032. This has worked very well for now, and we made sure we don't use ports < 1024, but could lead to mysterious failures if some local services are using ports like 5432, or others. The idea is that we could test if a port is available before using it. However, this error is already detected at startup time, so it shouldn't be a huge problem.

Feedback welcome!

More Perl 6 development, String::CRC32

Leave a reply

A couple of days were plenty enough to whip up a new Perl 6 module, String::CRC32.

This is a straight conversion from the Perl 5 CPAN version of String::CRC32. That one uses some C/XS code though. I wrote the equivalent in Perl 6. Was quite simple actually, once I learned a bit more on numeric bitwise operators.

The classic || and && for bitwise "OR" and "AND" are now written as:


Bitwise OR  '+|'
Bitwise AND '+&'
Bitwise XOR '+^'

So the following code:


$x &&= 0xFF;

becomes:


$x +&= 0xFF;

The reason behind this is that now there are not only numeric bitwise operators, but also boolean ones. You can read more in the official specification for Perl 6 operators.

You can download the code for the module on http://github.com/cosimo/perl6-string-crc32. Take a look at various other Perl 6 modules on modules.perl6.org.

Perl 6 LWP::Simple gets chunked transfers support

Leave a reply

With this one I think we're basically done!

Perl6 LWP::Simple gets chunked transfers support. It's probably not excellent or universally working, but for the examples I could try and test, it's totally fine. If you find some URLs where it's broken, please tell me.

I also threw in the getstore() method to save URLs locally.

So, LWP::Simple for Perl 6 is here and it's working. It's not yet "complete" compared to the Perl 5 version, but now that I got the hard bits working, and the internals can perform a full http response parsing, I'll try to reach a 100% API compatibility with the Perl 5 one (where it makes sense).

Try it and let me know. Have fun!

“DebPAN”, a production-grade Debian CPAN repository

Leave a reply

The problem

This is a proposal I came up with after talking to Gabor Szabo about his Perl Ecosystem Development proposal.

One of the major "problems" we face while developing and deploying production Perl-based systems with Debian is that the state of the Debian CPAN modules is depressingly outdated. As an example, we're using Catalyst in Lenny, and that dates back to 2008 for the most parts.

This is just not enough.

A solution: maintain your own APT repository

Our current solution is to manually package every bit and maintain our own internal Opera APT repository that our servers and applications depend on. That's not optimal for two reasons:

  • we have to package lots of modules, due to interdependencies, dedicating a fair amount of time to this activity that is not exactly "productive"
  • we can't trust our systems to be Debian anymore, since we're updating bits and pieces with the bold assumption that everything will work fine

Of course, 99.999% will work fine, since it's Perl, but the problem is that one day this could fall down on our heads.

So, given the problem, what are the solutions?

  • Continuing to manually keep an apt repository. Downside: Some(tm) waste of time
  • Use a different packaging/deployment system, like PAR::Repository. I would personally like this, but it doesn't eliminate the need to maintain an own repository. You just don't use dh-make-perl and friends, that are, IMHO, nice to have and useful
  • The "DebPAN"

The DebPAN

I know Jeremiah Foster, and at a couple of Perl events I heard him talking to other CPAN/Perl developers about these issues. His answer would probably be to file a request for packaging for the modules we're interested in.

The reason why that doesn't work is that the lead time for a given RFP to land on Debian stable is unacceptably long, and I realize that is for a good reason. After all, it's supposed to be stable, right?

What about a "DebPAN" repository?

That could be a 3rd party APT repository, something like debpan.perl.org:

  • maintained by a close group of Perl/Debian/CPAN developers
  • guaranteed to have a selection of the most important modules (more on that…) in a reasonably recent version
  • targeted to Debian stable, and maybe other distributions? I think Ubuntu 10.04 suffers from this same problem, but much less than Debian Lenny, just to pick two versions
  • maybe even with patches applied?, but that might be way too much, actually

The can of worms

Of course, lots of problems can arise. However, if we think this is a good idea, then we should try to have something even minimal up and running. Then we'll worry about all the problems…

However, who gets to decide the most useful modules? That should go by popular demand I guess. Even looking at Debian requests for packaging stats, maybe? I can also imagine that bigger companies using Perl would be interested in this to potentially save lots of "infrastructural work".

I'd be really interesting to know other people opinions on this, especially if they use Debian stable, Debian developers, or the Debian-Perl group itself.

Cache::Memcached::Mock, instant in-process memcached mock

Leave a reply

A week ago, I wrote a Cache::Memcached mock module for some complicated unit tests in this project I'm working on.

A few people asked to upload it to CPAN, so here it is:

Cache::Memcached::Mock v0.01 is on CPAN.

I didn't spend that much time polishing it and making documentation, so it's a bit rough around the edges, but you get the idea.

You can use it as a drop-in replacement for Cache::Memcached when you don't want, or can't afford, to run your own memcached daemon.

I've already got a feature request from a colleague: making sure set() fails if you try to store a value bigger than 1Mb.

Digest::MD5 for Perl 6 finally works!

Leave a reply

It took me an awful long time, and lots of help from the folks on #parrot and #perl6 but in the end, it's done!

It needs a tiny patch to Parrot, but I believe it will be added to the next parrot release. I tried to documented the fixes to the code to help others that might have the same problems.

So now Digest::MD5 for Perl 6 works as good as the Perl 5 one. I'm too tired to say anything else :)

Good night!

LWP::Simple for Perl 6, now with (partial) BasicAuth support and getstore()

Leave a reply

I just pushed out another update for the LWP::Simple module for Perl 6. This time, the main work was:

  • refactoring the code and adding unit tests for the URL parsing (that might even grow into a Perl 6 URI module
  • adding partial basic auth support. To be complete and working, it needs to base64 encode the user/password pair. Not implemented yet. I'll see if I get around to it, or if someone has done it already.
  • adding a getstore() method, that writes on to disk the downloaded content. Unfortunately that needs to strip the HTTP headers and undestand chunked transfers for it to be remotely useful.

It was nice to see the module grow in both functionality, code and unit tests coverage. I had to workaround a couple of problems I couldn't understand. I was extremely lazy and I didn't even look up Synopses, so I assume it's my fault. However.

The first is the use of .match() and ~~ to match against a regular expression. I found that the following code:

my $hostname = 'cosimo:eelst@faveclub.eelst.com';
if $hostname.match('^cosimo') {
    # Doesn't enter here
}

doesn't trigger a match. However, this other here:

my $hostname = 'cosimo:eelst@faveclub.eelst.com';
if $hostname ~~ /^cosimo/ {
    # Does match
}

And, in the same way, something similarly surprising. The following code correctly matches:

my $hostname = 'cosimo:eelst@faveclub.eelst.com';
if $hostname ~~ /^ .+ : .+ @ .+ $/ {
    say '(user:pass@host) matches';
} else {
    say '(user:pass@host) does not match';
}

but adding captures makes the same exact regex fail:

my $hostname = 'cosimo:eelst@faveclub.eelst.com';
if $hostname ~~ /^ (.+) : (.+)  @ (.+) $/ {
    say '(with captures) matches';
} else {
    say '(with captures) does not match';
}

There are also very nice things about programming in Perl 6 that are slowly sucking me in this fantastic language. This is part of a test script for LWP::Simple:

#
# Test the parse_url() method
#
use v6;
use Test;
use LWP:: Simple;

my @test = (
    { User-Agent => 'Opera/9.80 (WinNT; 6.0) Version/10.60' },
    "User-Agent: Opera/9.80 (WinNT; 6.0) Version/10.60rn",

    { Connection => 'close' },
    "Connection: closern",
);

for @test -> %headers, $expected_str {
    my $hdr_str = LWP:: Simple.stringify_headers(%headers);
    is($hdr_str, $expected_str, 'OK - ' ~ $hdr_str);
}

Note how in the for statement we can "extract" the hash and string from the @test array with:

for @test -> %headers, $expected_string {
    # Loop body
}

It's not a big deal, other languages have it, but Perl 6 is filled with this small niceties that make the resulting code still feel like Perl, but also, don't know exactly, more robust perhaps?

So, to conclude:

  • Anyone with a Perl 6 implementation of MIME::Base64 ? Speak up before I create a monster :)
  • Anyone cares enough to take on the chunked transfer encoding support?

Dist::Zilla, cpanhacker, do you use it??!

Leave a reply

Dist::Zilla, cpan hacker, do you use it?

If not, you should. Full stop.

So I've been reading about Dist::Zilla (dzil) for a while now. The first articles didn't really convince me too much. Yes, it's a nice tool, but why would I want to use it? Ok, to save time. Right. But how do I use it the first time? I need a config file. Ok where do I get (a sane) one?

Then I read a clear walk-through example of a Dist::Zilla config by Dave Rolsky. So, the quick start route was clear. Copy/paste the dist.ini file and try it.

Except, the file in that blog post contains so much stuff (read: additional plugins required) that is not practical to use as a first dist.ini config.

Basic premise

It's that you have to have a module/class that you want to upload to the CPAN, and you want to save yourself all the "package-wrapping" activities that a CPAN distribution needs. That is were dzil shines.

Start-using-Dist::Zilla-in-2-minutes guide

Good, you're on your way to spend countless hours less on maintaining your CPAN stuff.

  • First things first: write the thing that you want to package and upload to CPAN. Fatto? Bene.
  • So you probably have a lib folder with your class and a t folder with the test scripts for your class. Good.
  • Then, usually if you want to package everything for CPAN, you need to add a MANIFEST, a README, INSTALL, Changes, possibly a LICENSE, etc… Forget about those!
  • Copy paste the following into a dist.ini file: 
    ; Everything starting with ';' is a comment

name    = Your-Awesome-Module
author  = Joe J. Hacker <jjhacker@example.org>
license = Perl_5
copyright_holder = Joe J. Hacker
copyright_year   = 2010

version = 0.01

[@Basic]
[InstallGuide]
[MetaJSON]

[MetaResources]
bugtracker.web    = http://rt.cpan.org/NoAuth/Bugs.html?Dist=Your-Awesome-Module
bugtracker.mailto = bug-your-awesome-module@rt.cpan.org

; If you have a repository...
;repository.url    = git://github.com/jjhacker/your-awesome-module.git
;repository.web    = http://github.com/jjhacker/your-awesome-module
;repository.type   = git

; You have to have Dist::Zilla::Plugin::<Name> for these to work
;[PodWeaver]
;[NoTabsTests]
;[EOLTests]
;[Signature]
;[CheckChangeLog]

[Prereq]
;DBI = 1.600
;SomeOtherModule = 0

[Prereq / TestRequires]
Test::More = 0

; If you're using git, this is interesting
; You need to install Dist::Zilla::PluginBundle::Git
;[@Git]
  • Add the following line at the top of your main class (.pm) file: 
    # ABSTRACT: Supercool generator of twisty little passages, all alike
  • Run dzil build

That's it! The build command will do everything for you. When I realized how quick this tool makes my CPAN activities, I decided I had to try to convince some more people out there. Until I didn't really try it, I wasn't convinced 100%. Now I definitely am. Life's just too short. Use DZIL.

There's also a release command, that will also send your generated distribution archive to the CPAN directly. I couldn't get that to work. I remember there was some kind of dot file I had to write my credentials to before I could release, but I couldn't find the details today. If anyone knows, please comment here… :-)

Mocking Cache::Memcached

Leave a reply

Today I've been working on some elaborate unit tests that require a database and a memcached object. In My Opera, like probably everywhere else, we use our own classes for both DBI and memcached access. The memcached class in particular is just a subclass of Cache::Memcached, so nothing special there.

I was looking at already existing modules that would mock Cache::Memcached without requiring a running memcached server. The only relevant one I could find is Test::Memcached.

However, Test::Memcached interacts with the memcached binary, trying to start/stop it. This is not really what I want to do. I could do that, but it would be slightly complicated because we have lots of test installations, and we'd have to install or require a memcached daemon running everywhere.
A single shared memcached daemon wouldn't be so smart either since the different installations would interfere with each other. We could probably use key namespaces for that. Mmh.

Anyway, I decided that having a mock object, something that could be named Cache::Memcached::Mock, or Test::Memcached::Mock could be simpler and more easily testable as well. So I wrote a prototype that subclasses Test::MockObject that works fine for now and covers all my needs. It uses just a regular hash as memory storage, and supports get(), set(), flush_all(), and delete().

Not sure if I should upload it to CPAN…