{"id":503,"date":"2010-04-25T21:03:06","date_gmt":"2010-04-25T20:03:06","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/"},"modified":"2010-04-25T21:03:06","modified_gmt":"2010-04-25T20:03:06","slug":"handling-a-file-server-workload-with-varnish-part-2","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/","title":{"rendered":"Handling a file-server workload with varnish &#8211; part 2"},"content":{"rendered":"<p>I <a href=\"\/cstrep\/blog\/show.dml\/9815511\" rel=\"nofollow\">wrote about tuning varnish for a file server<\/a>. I&#39;d like to continue, detailing what we had to change compared to the varnish defaults to achieve a good and stable performance.<\/p>\n<p>These were the main topics I had mentioned:<\/p>\n<ul>\n<li>threads related parameters\n<\/li>\n<li>hash bucket size\n<\/li>\n<li>Session related parameters\n<\/li>\n<li>Grace config and health checking<\/li>\n<\/ul>\n<h3>Threads-related parameters<\/h3>\n<p>One thing is certain. You&#39;d better bump up the minimum threads number. When varnish starts up, it creates &quot;n&quot; threads. If more threads are needed, it can create as much as &quot;m&quot; threads but no more.<\/p>\n<p>&quot;n&quot; is given by <code>thread_pool_min * thread_pools<\/code>. The defaults are 2 thread pools, and <code>thread_pool_min<\/code> is 200, so varnish will create 400 threads when starting up. We found that we need at least 6,000 threads, sometimes peaking at 8,000. In this case, it&#39;s better to start up directly with 7-8,000 threads. We set:<\/p>\n<ul>\n<li><code>thread_pools = 8<\/code> since we have a 8 cores machine\n<\/li>\n<li><code>thread_pool_min = 800<\/code><\/li>\n<\/ul>\n<p> With these settings, on start Varnish will create 6,400 threads and keep them running all the time.<\/p>\n<p>We also set a related param, <code>thread_pool_add_delay<\/code> to <code>2 ms<\/code>, instead of the default I believe 20 ms. This allows Varnish to create a lot of threads more quickly when it starts up, or when more threads are needed. Using the default 20 ms value could slow down the threads creation process, and prevent Varnish to serve requests quickly enough.<\/p>\n<h3>Hash bucket size<\/h3>\n<p>Don&#39;t know much about the hashing internals, but I know we have tens of millions of files, even more, so we have to make sure the hash tables used to store cached objects are big enough, to prevent too many hashing collisions.<\/p>\n<p>This is controlled by the <code>-h<\/code> option to <code>varnishd<\/code>. The default bucket size is 50023. We set it to 500009 (<code>-h classic,500009<\/code>). In this way, even if we could keep 10 million files in memory, we would only have 20 entries in each bucket on average. That&#39;s not ideal, but it&#39;s better than the default.<\/p>\n<p>We didn&#39;t experiment with the new hashing algorithms like <code>critbit<\/code>.<\/p>\n<h3>Session-related parameters<\/h3>\n<p>Not so much on this particular server, but in general, we had to bump up the <code>sess_workspace<\/code> parameter. The default is <b>16kbytes (16384)<\/b>. <code>sess_workspace<\/code> controls the amount of memory dedicated by varnish to each connection (session in varnish speak), that is used as a working memory for the HTTP header manipulations. We set it to 32k here. On other servers, <b>where we use a more elaborate VCL config<\/b>, we use <b>128k<\/b> as the default value.<\/p>\n<h3>Grace and health checking<\/h3>\n<p>Varnish can check that your defined backends are &quot;healthy&quot;. That means that they respond to queries in the defined time, and they don&#39;t miss heartbeats. You enable health checks just by defining a <code>.probe<\/code> block in your <code>backend<\/code> definition (search the Varnish wiki for details).<\/p>\n<p>Having health checks is very convenient. You can instruct varnish to extend the <a href=\"http:\/\/overstimulate.com\/articles\/varnish-getting-started\" rel=\"nofollow\">grace period<\/a> when\/if your backend is dead. This means: if Varnish detects that your backends are dead or overloaded and they miss some heartbeats, it will keep serving stale objects from its cache, even if they expired (their TTL is already over). You enable this behaviour by saying:<\/p>\n<pre>\r\nsub vcl_recv {\r\n   set req.backend = mybackend;\r\n\r\n   # Default grace period is 10s\r\n   set req.grace = 10s;\r\n\r\n   # OMG. Backend dead. Keep serving stuff until we recover them.\r\n   if (! req.backend.healthy) {\r\n      set req.grace = 4h;\r\n   }\r\n    \r\n   ...\r\n}\r\n\r\nsub vcl_fetch {\r\n\r\n   # Renew cached objects every minute ...\r\n   set obj.ttl = 60s;\r\n\r\n   # ... but keep all objects way past their expire date\r\n   # in case we need them because backends died\r\n   set obj.grace = 4h;\r\n\r\n   ...\r\n\r\n}\r\n<\/pre>\n<p>That&#39;s it. We&#39;re continuing to refine our configs and best practices for Varnish servers. If you have feedback, leave a comment or drop me an email.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I wrote about tuning varnish for a file server. I&#39;d like to continue, detailing what we had to change compared to the varnish defaults to achieve a good and stable performance. These were the main topics I had mentioned: threads related parameters hash bucket size Session related parameters Grace config and health checking Threads-related parameters [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[465,75,326,60],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Handling a file-server workload with varnish - part 2 - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Handling a file-server workload with varnish - part 2 - Random hacking\" \/>\n<meta property=\"og:description\" content=\"I wrote about tuning varnish for a file server. I&#039;d like to continue, detailing what we had to change compared to the varnish defaults to achieve a good and stable performance. These were the main topics I had mentioned: threads related parameters hash bucket size Session related parameters Grace config and health checking Threads-related parameters [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2010-04-25T20:03:06+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Handling a file-server workload with varnish &#8211; part 2\",\"datePublished\":\"2010-04-25T20:03:06+00:00\",\"dateModified\":\"2010-04-25T20:03:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\"},\"wordCount\":548,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"files\",\"performance\",\"server\",\"varnish\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\",\"name\":\"Handling a file-server workload with varnish - part 2 - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2010-04-25T20:03:06+00:00\",\"dateModified\":\"2010-04-25T20:03:06+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Handling a file-server workload with varnish &#8211; part 2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Handling a file-server workload with varnish - part 2 - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/","og_locale":"en_US","og_type":"article","og_title":"Handling a file-server workload with varnish - part 2 - Random hacking","og_description":"I wrote about tuning varnish for a file server. I&#39;d like to continue, detailing what we had to change compared to the varnish defaults to achieve a good and stable performance. These were the main topics I had mentioned: threads related parameters hash bucket size Session related parameters Grace config and health checking Threads-related parameters [&hellip;]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/","og_site_name":"Random hacking","article_published_time":"2010-04-25T20:03:06+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Handling a file-server workload with varnish &#8211; part 2","datePublished":"2010-04-25T20:03:06+00:00","dateModified":"2010-04-25T20:03:06+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/"},"wordCount":548,"commentCount":0,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["files","performance","server","varnish"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/","name":"Handling a file-server workload with varnish - part 2 - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2010-04-25T20:03:06+00:00","dateModified":"2010-04-25T20:03:06+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/04\/handling-a-file-server-workload-with-varnish-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Handling a file-server workload with varnish &#8211; part 2"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/503"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=503"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/503\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=503"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=503"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=503"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}