Tag Archives: memcached

Displaying realtime memcached traffic on a backend

Sometimes I like to write down posts like this, to remind myself how to do something, sort of a mental note.
Suppose you have a few application servers that use 1+ memcached servers, and you want some way to display the outbound traffic, providing some insights on what are the most used keys, counters, etc…

Here's a quick way to do that, assuming you're using the memcached text protocol:

tcpflow -ce dst port 11211 
    | cut -b53- 
    | grep ^get 
    | pipestat --clear --runtime 60 --field 2 --time 1 --limit 40

What this does is:

  • Use tcpflow to capture all outbound traffic to destination port 11211, default memcached port.
  • Remove the first 53 bytes from each line, to filter out source and destination ip/ports
  • Only display get requests (alternatively, use set, incr, …)
  • Feed the resulting data to pipestat, a simple but great Perl tool that aggregates the data, displaying the most frequent ones. The specific options I used are good if you want to display quick statistics like other tools as top, mytop, or varnishstat.

It goes without saying that these tools are automatically installed on all servers that our Devops team here at Opera manages. I couldn't work without them :)

Cache::Memcached::Mock, instant in-process memcached mock

A week ago, I wrote a Cache::Memcached mock module for some complicated unit tests in this project I'm working on.

A few people asked to upload it to CPAN, so here it is:

Cache::Memcached::Mock v0.01 is on CPAN.

I didn't spend that much time polishing it and making documentation, so it's a bit rough around the edges, but you get the idea.

You can use it as a drop-in replacement for Cache::Memcached when you don't want, or can't afford, to run your own memcached daemon.

I've already got a feature request from a colleague: making sure set() fails if you try to store a value bigger than 1Mb.

Mocking Cache::Memcached

Today I've been working on some elaborate unit tests that require a database and a memcached object. In My Opera, like probably everywhere else, we use our own classes for both DBI and memcached access. The memcached class in particular is just a subclass of Cache::Memcached, so nothing special there.

I was looking at already existing modules that would mock Cache::Memcached without requiring a running memcached server. The only relevant one I could find is Test::Memcached.

However, Test::Memcached interacts with the memcached binary, trying to start/stop it. This is not really what I want to do. I could do that, but it would be slightly complicated because we have lots of test installations, and we'd have to install or require a memcached daemon running everywhere.
A single shared memcached daemon wouldn't be so smart either since the different installations would interfere with each other. We could probably use key namespaces for that. Mmh.

Anyway, I decided that having a mock object, something that could be named Cache::Memcached::Mock, or Test::Memcached::Mock could be simpler and more easily testable as well. So I wrote a prototype that subclasses Test::MockObject that works fine for now and covers all my needs. It uses just a regular hash as memory storage, and supports get(), set(), flush_all(), and delete().

Not sure if I should upload it to CPAN…