{"id":441,"date":"2011-11-13T19:41:13","date_gmt":"2011-11-13T18:41:13","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/"},"modified":"2011-11-13T19:41:13","modified_gmt":"2011-11-13T18:41:13","slug":"migration-of-vcl-configuration-from-varnish-2-0-to-2-1","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/","title":{"rendered":"Migration of VCL configuration from Varnish 2.0 to 2.1"},"content":{"rendered":"

Recently we migrated most of our services from Varnish 2.0 to 2.1<\/strong>.
\nI'd like to explain what we changed with code (VCL) examples side by side,
\nin case anyone still needs to migrate to 2.1 and needs some help as well :-)<\/p>\n

req.<\/code>, bereq.<\/code>, beresp.<\/code>, and obj.<\/code><\/h3>\n

Usually this naming difference in VCL is not really explained. They say "x has been renamed to y"<\/i>
\nand you should change the name. That's kind of annoying. In reality, yes, the names changed, and at first
\nit is annoying, but trying to understand why<\/strong> they changed allows them to stick
\nin your mind very easily.<\/p>\n

In vcl_fetch()<\/code>, obj.<\/code> is now beresp.<\/code>. Why?
\nBecause vcl_fetch()<\/code> is the part of the request stage where Varnish has
\nalready performed a request against a backend and got a response from it. That means that
\nif you refer to obj.<\/code> in vcl_fetch()<\/code>, it really means that your
\ntouching the backend response, hence beresp.<\/code>.<\/p>\n

Similarly in vcl_pipe()<\/code>, that is executed when the result of vcl_recv()<\/code>
\nis to switch to pipe mode<\/strong>. In that case, however, Varnish hasn't made the request
\nto the backend yet, so if you used obj.<\/code> in vcl_pipe()<\/code> you really meant
\nto change the request that was going to be made to the backend<\/strong>, hence bereq.<\/code>.<\/p>\n

Let's see the changes we had to make:<\/p>\n

\r\n sub vcl_fetch {\r\n \r\n-    set obj.ttl = 88s;\r\n-    set obj.grace = 10m;\r\n-    set obj.http.X-My-Opera = "http:\/\/youtube.com\/watch?v=br79xGSpgF4";\r\n+    set beresp.ttl = 88s;\r\n+    set beresp.grace = 60m;\r\n+    set beresp.http.X-dramatic = "http:\/\/www.youtube.com\/watch?v=a1Y73sPHKxw";\r\n \r\n }\r\n<\/pre>\n
And:<\/p>\n
\r\n sub vcl_pipe {\r\n     # Streaming files too (see related vcl_recv() rule).\r\n     # We need to close the request, or varnish remains in pipe\r\n     # mode for the entire session with that client.\r\n-    set req.http.connection = "close";\r\n+    set bereq.http.connection = "close";\r\n }\r\n<\/pre>\n
Backend probes and .initial<\/code><\/h3>\n
Backend probing allows Varnish to detect that backends are either "healthy"<\/i>
\nor "sick"<\/i>. The probe VCL config block allows to tweak how this should work. In
\nparticular, .threshold<\/code> is the number of successful probes that are necessary
\nfor Varnish to consider a backend healthy. .interval<\/code> is the number of seconds
\nbetween one probe and the following one.<\/p>\n
As an example, you can define that a backend should be considered working
\n(healthy) when it answers successfully to at least 3 probes, with an interval of
\n10 seconds between each probe. In Varnish 2.0.4, this means that if restarted,
\nVarnish will wait 3 times 10 = 30 seconds before serving any requests
\nfrom that backend<\/strong>, because all backends were considered dead (sick) at startup.<\/p>\n
In 2.1 this limitation is removed by introducing an .initial<\/code> attribute
\nin the probe block. .initial<\/code> is the number of probes considered successful
\nwhen the service is started, or the backend is added, and there's no information about it.
\nThe default value is assumed to be equal to .threshold<\/code>, so backends are considered
\nhealthy as soon as they are introduced.<\/p>\n
I think you can understand from these tiny details how well Varnish is engineered.
\nThis just makes sense, doesn't it? :-) Here's the diff from 2.0 to 2.1:<\/p>\n
\r\n backend nginx {\r\n     .host = "localhost";\r\n     .port = "8080";\r\n-\r\n-    # Disabled to avoid the 15s startup\r\n-    # 2.0.4-5 doesn't have .initial\r\n-    #\r\n-    #.probe = {\r\n-    #  .url = "\/ping.html";\r\n-    #    .interval = 5s;\r\n-    #    .timeout = 1s;\r\n-    #    .window = 5;\r\n-    #    .threshold = 3;\r\n-    #}\r\n+    .probe = {\r\n+        .url = "\/ping.html";\r\n+        .interval = 10s;\r\n+        .timeout = 2s;\r\n+        .window = 10;\r\n+        .threshold = 3;\r\n+        .initial = 3;\r\n+    }\r\n }\r\n<\/pre>\n
And in vcl_recv()<\/code>:<\/p>\n
\r\n sub vcl_recv {\r\n\r\n [...]\r\n\r\n-    #----------\r\n-    # DISABLED: Only enable when .probe block above is enabled\r\n-    #----------\r\n     # Detect broken backends and keep serving requests if possible\r\n-    #if (! req.backend.healthy) {\r\n-    #    set req.grace = 10m;\r\n-    #} else {\r\n-    #    set req.grace = 5s;\r\n-    #}\r\n+    if (! req.backend.healthy) {\r\n+        set req.grace = 60m;\r\n+    } else {\r\n+        set req.grace = 5s;\r\n+    }\r\n<\/pre>\n
Regular expression matching<\/h3>\n
Another "big" difference is the use in 2.1 of a Perl-compatible regular expression engine,
\n(PCRE<\/strong>) instead of the POSIX-style regex matching that used to be in 2.0.
\nThis is a good change for me, as I'm pretty much used to Perl regex and I know next to nothing
\nabout POSIX.<\/p>\n
This change actually created a subtle problem that I caught only with a thorough testing
\nof our configurations. We use regex matching in a few places in our VCL configuration, 
\nusually to analyze cookies<\/strong> and set special "flags" that are then used to force
\na HTTP Vary<\/code> header, to make Varnish store different cached versions of the same
\nURL.<\/p>\n
One of these cases is the language<\/code> cookie, where we store a sticky
\nuser preference about site language. Here's how the code changed:<\/p>\n
\r\n  # STD: Sticky language cookie\r\n  if (req.http.Cookie ~ "language=") {\r\n      set req.http.X-Language =\r\n-         regsub(req.http.Cookie, "^.*?language=([^;]*?);*.*$", "1");\r\n+         regsub(req.http.Cookie, "^.*?language=([^;]*);*.*$", "1");\r\n  }\r\n\r\n  ...\r\n\r\n  # Mobile view cookie\r\n  if (req.http.Cookie ~ "mobile=") {\r\n-     set req.http.X-Mobile = \r\n-         regsub(req.http.Cookie, "^.*?mobile=([^;]*?);*.*$", "1");\r\n+     set req.http.X-Mobile =\r\n+         regsub(req.http.Cookie, "^.*?mobile=([^;]*);*.*$", "1");\r\n  }\r\n<\/pre>\n
In case you find it difficult to spot the change, it's the removal of the *?<\/code>
\n(non-greedy star) operator. Non-greedy matching was used in 2.0, POSIX matching, to make
\nsure that the * didn't match too many characters, and thus eat part of other cookies. Except
\nPOSIX regex matching does NOT have a non-greedy star operator<\/strong>. I just
\ndidn't know that, and it's of course a bug, but it had worked perfectly so far… WTF???<\/p>\n
For even more weirdness, why did I take the non-greedy star (*?) away now that it should
\nbe supported with PCRE-matching? I removed it because otherwise the result of those
\nregsub()<\/code> expressions are always empty!<\/strong><\/p>\n
Believe it or not, it looks exactly like 2.0 had PCRE and 2.1 has POSIX, which is
\nobviously not what's happening. If you know more about this and you can shed some light,
\nplease contact me or leave a comment below.<\/p>\n
Hope you liked this 2.0 -> 2.1 migration journey. I'm looking forward to 2.1 -> 3.0!
\nIt's a bit more work there, because I will need to migrate my
\nmy accept-language C extension<\/a>
\nto the new vmod<\/strong> system, which I already started working on :-)<\/p>\n
Have fun!<\/p>\n","protected":false},"excerpt":{"rendered":"
Recently we migrated most of our services from Varnish 2.0 to 2.1. I'd like to explain what we changed with code (VCL) examples side by side, in case anyone still needs to migrate to 2.1 and needs some help as well :-) req., bereq., beresp., and obj. Usually this naming difference in VCL is not […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[265,268,242,63,267,264,266,78,60],"yoast_head":"\nMigration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Migration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking\" \/>\n<meta property=\"og:description\" content=\"Recently we migrated most of our services from Varnish 2.0 to 2.1. I'd like to explain what we changed with code (VCL) examples side by side, in case anyone still needs to migrate to 2.1 and needs some help as well :-) req., bereq., beresp., and obj. Usually this naming difference in VCL is not […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2011-11-13T18:41:13+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Migration of VCL configuration from Varnish 2.0 to 2.1\",\"datePublished\":\"2011-11-13T18:41:13+00:00\",\"dateModified\":\"2011-11-13T18:41:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\"},\"wordCount\":754,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"accept-language\",\"cache\",\"devops\",\"github\",\"pcre\",\"posix\",\"regex\",\"upgrade\",\"varnish\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\",\"name\":\"Migration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2011-11-13T18:41:13+00:00\",\"dateModified\":\"2011-11-13T18:41:13+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Migration of VCL configuration from Varnish 2.0 to 2.1\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Migration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/","og_locale":"en_US","og_type":"article","og_title":"Migration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking","og_description":"Recently we migrated most of our services from Varnish 2.0 to 2.1. I'd like to explain what we changed with code (VCL) examples side by side, in case anyone still needs to migrate to 2.1 and needs some help as well :-) req., bereq., beresp., and obj. Usually this naming difference in VCL is not […]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/","og_site_name":"Random hacking","article_published_time":"2011-11-13T18:41:13+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Migration of VCL configuration from Varnish 2.0 to 2.1","datePublished":"2011-11-13T18:41:13+00:00","dateModified":"2011-11-13T18:41:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/"},"wordCount":754,"commentCount":0,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["accept-language","cache","devops","github","pcre","posix","regex","upgrade","varnish"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/","name":"Migration of VCL configuration from Varnish 2.0 to 2.1 - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2011-11-13T18:41:13+00:00","dateModified":"2011-11-13T18:41:13+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/11\/migration-of-vcl-configuration-from-varnish-2-0-to-2-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Migration of VCL configuration from Varnish 2.0 to 2.1"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/441"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=441"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/441\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=441"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=441"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=441"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}