{"id":509,"date":"2010-03-07T14:09:39","date_gmt":"2010-03-07T13:09:39","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/"},"modified":"2010-03-07T14:09:39","modified_gmt":"2010-03-07T13:09:39","slug":"varnish-sess_workspace-and-why-it-is-important","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/","title":{"rendered":"Varnish &#8220;sess_workspace&#8221; and why it is important"},"content":{"rendered":"<p>When using Varnish on a high traffic site like <a href=\"http:\/\/www.opera.com\" rel=\"nofollow\">opera.com<\/a> or <a href=\"http:\/\/my.opera.com\/community\/\" rel=\"nofollow\">my.opera.com<\/a>, it is important to reach a stable and sane configuration (both <a href=\"http:\/\/varnish-cache.org\/wiki\/VCL\" rel=\"nofollow\">VCL<\/a> and general service tuning).<\/p>\n<p>If you&#39;re just starting using Varnish now, it&#39;s easy to overlook things (like I did, for example :) and later experience some crashes or unexpected problems.<\/p>\n<p>Of course, you should read the <a href=\"http:\/\/varnish.projects.linpro.no\" rel=\"nofollow\">Varnish wiki<\/a>, but I&#39;d suggest you also read at least the following links. I found them to be very useful for me:<\/p>\n<ul>\n<li><a href=\"http:\/\/kristian.blog.linpro.no\/tag\/varnish\/\" rel=\"nofollow\">Kristian Lyngst\u00c3\u00b8l&#39;s blog<\/a>, the varnish-related posts, but other stuff as well. I had the opportunity to attend a 2-day Varnish training at <a href=\"http:\/\/www.redpill-linpro.no\" rel=\"nofollow\">Linpro<\/a>, and he was holding the course. I can&#39;t say good enough of the advice Kristian gives in his blog. Really, go read it <b>now!<\/b>.\n<\/li>\n<li>Other users mails on the <code>varnish-misc<\/code> mailing list. In particular, two messages that carry so much helpful information that one could study for a month probably. <a href=\"http:\/\/projects.linpro.no\/pipermail\/varnish-dev\/2009-February\/000968.html\" rel=\"nofollow\">This one by Twitter&#39;s John Adams<\/a>, and <a href=\"http:\/\/www.mail-archive.com\/varnish-misc@projects.linpro.no\/msg01571.html\" rel=\"nofollow\">This one by Audun Ytterdal<\/a>, now working at <a href=\"http:\/\/vg.no\" rel=\"nofollow\">VG.no<\/a>, one of the biggest norwegian newspapers.\n<\/li>\n<li><a href=\"http:\/\/www.oscon.com\/oscon2009\/public\/schedule\/detail\/10433\" rel=\"nofollow\">Artur Bergman OSCON 2009 talk on Varnish<\/a> (<a href=\"http:\/\/assets.en.oreilly.com\/1\/event\/27\/Varnish%20-%20A%20State%20of%20the%20Art%20High-Performance%20Reverse%20Proxy%20Presentation.pdf\" rel=\"nofollow\">PDF<\/a>, or on <a href=\"http:\/\/www.slideshare.net\/crucially\/varnish-oscon-2009\" rel=\"nofollow\">slideshare<\/a>). Dense with useful tips, Bergman runs a high traffic site chaining multiple distributed varnish servers.<\/li>\n<\/ul>\n<p>A couple of weeks ago, we experienced some random Varnish crashes, 1 per day on average. That happened during a weekend. As usual, we didn&#39;t really notice that Varnish was crashing until we looked at our Munin graphs. Once you know that Varnish is crashing, everything is easier :)<\/p>\n<p>Just look at your syslog file. We did, and we found the following error message:<code><\/p>\n<pre>\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child (27707) died signal=6\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child (27707) Panic message: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 188:#012  Condition((p) != 0) not true.  thread = (cache-worker)sp = 0x7f8007c7f008 {#012  fd = 239, id = 239, xid = 1109462166,#012  client = 213.236.208.102:39798,#012  step = STP_LOOKUP,#012  handling = hash,#012  ws = 0x7f8007c7f078 { overflow#012    id = &quot;sess&quot;,#012    {s,f,r,e} = {0x7f8007c7f808,,+16369,(nil),+16384},#012  },#012    worker = 0x7f82c94e9be0 {#012    },#012    vcl = {#012      srcname = {#012        &quot;input&quot;,#012        &quot;Default&quot;,#012        &quot;\/etc\/varnish\/accept-language.vcl&quot;,#012      },#012    },#012},#012\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child cleanup complete\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: child (3710) Started\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child (3710) said Closed fds: 3 4 5 10 11 13 14\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child (3710) said Child starts\r\nFeb 26 06:58:26 p26-01 varnishd[19110]: Child (3710) said Ready\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child (7327) died signal=6\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child (7327) Panic message: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 188:#012  Condition((p) != 0) not true.  thread = (cache-worker)sp = 0x7f8008e84008 {#012  fd = 248, id = 248, xid = 447481155,#012  client = 213.236.208.101:39963,#012  step = STP_LOOKUP,#012  handling = hash,#012  ws = 0x7f8008e84078 { overflow#012    id = &quot;sess&quot;,#012    {s,f,r,e} = {0x7f8008e84808,,+16378,(nil),+16384},#012  },#012    worker = 0x7f81a4f5fbe0 {#012    },#012    vcl = {#012      srcname = {#012        &quot;input&quot;,#012        &quot;Default&quot;,#012        &quot;\/etc\/varnish\/accept-language.vcl&quot;,#012      },#012    },#012},#012\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child cleanup complete\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: child (30662) Started\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child (30662) said Closed fds: 3 4 5 10 11 13 14\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child (30662) said Child starts\r\nFeb 26 18:13:37 p26-01 varnishd[19110]: Child (30662) said Ready\r\n<\/pre>\n<p><\/code><\/p>\n<p>A quick research brought me to <a href=\"http:\/\/www.google.no\/search?client=opera&amp;rls=en&amp;q=varnish+sess_workspace&amp;sourceid=opera&amp;ie=utf-8&amp;oe=utf-8\" rel=\"nofollow\"><code>sess_workspace<\/code><\/a>.<\/p>\n<p>We found out we had to increase the default (16kb), especially since we&#39;re doing quite a bit of HTTP header copying and rewriting around. In fact, if you do that, each varnish thread uses a memory space at most <code>sess_workspace<\/code> bytes.<\/p>\n<p>If you happen to need more space, maybe because clients are sending long HTTP header values, or because you are (like we do) writing lots of additional varnish-specific headers, then Varnish won&#39;t be able to allocate enough memory, and will just write the assert condition on syslog and drop the request.<\/p>\n<p>So, we bumped <code>sess_workspace<\/code> to <b>256kb<\/b> by setting the following in the startup file:<\/p>\n<pre><code>\r\n-p sess_workspace=262144\r\n<\/code><\/pre>\n<\/p>\n<p>And since then we haven&#39;t been having crashes anymore.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When using Varnish on a high traffic site like opera.com or my.opera.com, it is important to reach a stable and sane configuration (both VCL and general service tuning). If you&#39;re just starting using Varnish now, it&#39;s easy to overlook things (like I did, for example :) and later experience some crashes or unexpected problems. Of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[75,76,77,60],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Varnish &quot;sess_workspace&quot; and why it is important - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Varnish &quot;sess_workspace&quot; and why it is important - Random hacking\" \/>\n<meta property=\"og:description\" content=\"When using Varnish on a high traffic site like opera.com or my.opera.com, it is important to reach a stable and sane configuration (both VCL and general service tuning). If you&#039;re just starting using Varnish now, it&#039;s easy to overlook things (like I did, for example :) and later experience some crashes or unexpected problems. Of [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2010-03-07T13:09:39+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Varnish &#8220;sess_workspace&#8221; and why it is important\",\"datePublished\":\"2010-03-07T13:09:39+00:00\",\"dateModified\":\"2010-03-07T13:09:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\"},\"wordCount\":404,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"performance\",\"servers\",\"sess_workspace\",\"varnish\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\",\"name\":\"Varnish \\\"sess_workspace\\\" and why it is important - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2010-03-07T13:09:39+00:00\",\"dateModified\":\"2010-03-07T13:09:39+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Varnish &#8220;sess_workspace&#8221; and why it is important\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Varnish \"sess_workspace\" and why it is important - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/","og_locale":"en_US","og_type":"article","og_title":"Varnish \"sess_workspace\" and why it is important - Random hacking","og_description":"When using Varnish on a high traffic site like opera.com or my.opera.com, it is important to reach a stable and sane configuration (both VCL and general service tuning). If you&#39;re just starting using Varnish now, it&#39;s easy to overlook things (like I did, for example :) and later experience some crashes or unexpected problems. Of [&hellip;]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/","og_site_name":"Random hacking","article_published_time":"2010-03-07T13:09:39+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Varnish &#8220;sess_workspace&#8221; and why it is important","datePublished":"2010-03-07T13:09:39+00:00","dateModified":"2010-03-07T13:09:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/"},"wordCount":404,"commentCount":1,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["performance","servers","sess_workspace","varnish"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/","name":"Varnish \"sess_workspace\" and why it is important - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2010-03-07T13:09:39+00:00","dateModified":"2010-03-07T13:09:39+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/03\/varnish-sess_workspace-and-why-it-is-important\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Varnish &#8220;sess_workspace&#8221; and why it is important"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/509"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=509"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/509\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=509"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=509"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}