Tag Archives: geoip

How to start up varnish with a custom cc_command on Debian

Leave a reply

If you need to compile your varnish VCL file with custom options, maybe because of libraries like GeoIP, and you're running Debian, you can not use the init script that's shipped by default.

It will not work because of how shell expansion works in the start-stop-daemon command contained in the init script. I wrote my explanation and a proposed fix in much more detail in this stack overflow question:

http://stackoverflow.com/questions/5906603/varnish-daemon-opts-options-errors/8333333#8333333

TL;DR: (+ quick & dirty fix) patch your init script like this:

 start_varnishd() {
     log_daemon_msg "Starting $DESC" "$NAME"
     output=$(/bin/tempfile -s.varnish)
-    if start-stop-daemon 
-       --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- 
-       -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
+    if bash -c "start-stop-daemon 
+        --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- 
+        -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then
         log_end_msg 0
     else
         log_end_msg 1
         cat $output
         exit 1
     fi
     rm $output
 }

Let me know if it works for you!

EDIT (7/Mar/2012): bug was filed in Debian as #659005. Nothing happened so far. We'll see.

Geo::IP support for IPv6 geolocation

Leave a reply

We're currently looking into IPv6-enabling our services. One of the missing bits is being able to geolocate IPv6 client addresses. We're using the MaxMind GeoIP database. The main Perl library for this is Geo::IP.

The current version of Geo::IP out on CPAN, 1.38, does not support IPv6 lookups. I contacted the maintainer of Geo::IP asking for more information. In the meanwhile, I hacked together just enough of IPv6 support to be able to successfully geolocate a test address. Later on, I discovered that IPv6 is already available in the hopefully soon-to-be-released version of Geo::IP archived at Sourceforge.

Let's hope it lands on CPAN soon. In the meantime, if you really really want, you can try out my changes against CPAN v1.38. It was enough for me to start testing the integration with our other code and projects.

My Geo::IP with IPv6 support

Geo-IP lookup without installing Geo-IP libraries (everywhere)

Leave a reply

We've been using GeoDNS to distribute client requests to different data centers around the world, first as a highly experimental project, then more and more as months passed.

Currently we're using it as simple global load balancer for files.myopera.com, help.opera.com, and some get.opera.com stuff.

However, there is another minor feature that we built into it, like a hacker's backdoor :) Since it's using a full DNS server, and it relies on having GeoIP libraries installed and always up-to-date, we thought it was a nice and cool idea to have a quick way to perform geo-ip lookups from the command line.

It works in a similar way as DNS black lists do. Suppose you want to look up the IP address 80.90.100.110. You reverse the IP, and lookup a special geo.opera.com zone:

cosimo@cd01:~$ host -t TXT 110.100.90.80.lookup.geo.opera.com
110.100.90.80.lookup.geo.opera.com descriptive text "ip:80.90.100.110, country:de, continent:europe"

This uses the GeoDNS backends to resolve country and continent of the given IP address, and gets back the information in a very basic string format. A simple shell or Perl script can then process that for you if you need. In fact, I made a ~/bin/geolookup Perl script that I can use like this:

cosimo@cd01:~$ geolookup 80.90.100.110 90.100.110.120 100.110.120.130
80.90.100.110 => de, europe
90.100.110.120 => fr, europe
100.110.120.130 => unknown, unknown

Nothing special, but in this way, no matter what machine I'm on, I can always quickly lookup IPs if I need to, without having to download the Country or City GeoIP databases, and keep them up-to-date. On the geodns backends, this is of course done routinely with a set of simple cronjobs.

A geolocating, distributed DNS service, geodns

Leave a reply

It's been 2 months now that I formally changed my team from My Opera to Core Services. For most of my day, I still work on My Opera, but I get to work on other projects as well.

One of these projects regarded the browser-ballot screen, even if now it's being used for other purposes as well. It is a very interesting project, named Geodns. It is a DNS server.

Its purpose is not unique or new: create geographically-aware DNS zones. Example (just an example): my.geo.opera.com, a geographically-aware my.opera.com, that sends you to our US-based My Opera servers if your browser presents itself with a US or american ip address, norwegian servers if you are in Norway, etc… So, nothing new or particularly clever. Actually someone argues that DNS systems shouldn't be used in this way. But it's really convenient anyway…

So this DNS server is written in Perl, and it uses the omnipresent GeoIP library to find out every client IP address position on Earth, and then uses this information to send the client to the appropriate server based on some simple matching rules:

  • by specific country
  • by specific continent
  • if none match specifically, extract a random server from the pool of those available to serve requests that don't match any other rule

I also made geodns log to a special file that allows to use our own OpenGL engine to display realtime DNS hits on a photo-realistic 3D Earth.

In this picture you can see blue and red dots. The higher the spikes, the more requests there are. Blue is where our datacenters are, red is where clients are sending requests from.

I'm trying to get this published as open source, even if, as I said, it's not really complex or anything. Just a good out-of-the-box solution. It's been running in production for about 3 weeks now, and it's serving around 300 requests per second on average. Stable and fast, but now we're looking at increasing the traffic. My goal is to reach at least 2000 req/s on a single machine. We'll see… :)