{"id":477,"date":"2010-07-13T17:24:32","date_gmt":"2010-07-13T16:24:32","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/"},"modified":"2010-07-13T17:24:32","modified_gmt":"2010-07-13T16:24:32","slug":"bandwidth-limiting-a-varnish-server","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/","title":{"rendered":"Bandwidth limiting a varnish server"},"content":{"rendered":"<p>I know, this might sound like blasphemy. After all, <b>why on earth should one want to limit bandwidth, and serve your clients slower, when using Varnish to accelerate HTTP serving?<\/b><\/p>\n<p>Sounds silly. And it is, but if your <b>Varnish servers are saturating your internet wires<\/b>, and you have other services running, you might want to investigate on bandwidth limiting possibilities. On an installation of Varnish here at Opera, I&#39;m constantly seeing nodes hitting <b>950+ Mbit\/s<\/b> peak, and 600-700 Mbit\/s average. That&#39;s a damn fast bit pushing, near to the physical theoretical capacity of 1 Gbit\/s for one Gb ethernet card.<\/p>\n<p>Now, back to the problem. How to bw limit Varnish?<\/p>\n<p>I think I know the Varnish project well enough to be sure that something like bandwidth limiting will <b>never be implemented<\/b> inside Varnish. Asking on the mailing list, I got the tip to start looking into <code>tc<\/code> and its <a href=\"http:\/\/lartc.org\/\" rel=\"nofollow\">companion howto<\/a>.<\/p>\n<p>I&#39;d never heard of <code>tc<\/code> before, so I started searching around. Turns out that half of the internet is heavily outdated :) I found lots of old stale material, and very few working examples. After a bit of research, I came up with a semi-working solution.<\/p>\n<p><code>tc<\/code> is a linux command installed by default almost anywhere I think. Its purpose is to control network traffic. It&#39;s not very simple to use, and I don&#39;t want to pretend I know it. I just wished there were more examples about it, easier to understand, and <b>recently updated<\/b>.<\/p>\n<p>The best example I found is archived at <a href=\"http:\/\/blog.edseek.com\/~jasonb\/articles\/traffic_shaping\/scenarios.html\" rel=\"nofollow\">http:\/\/blog.edseek.com\/~jasonb\/articles\/traffic_shaping\/scenarios.html<\/a> (&quot;8.2 Guaranteeing rate&quot;):<\/p>\n<pre>\r\n#!\/bin\/bash\r\n\r\nRATE=8000\r\n\r\nif [ x$1 = &#39;xstop&#39; ]\r\nthen\r\n        tc qdisc del dev eth0 root &gt;\/dev\/null 2&gt;&amp;1\r\nfi\r\n\r\ntc qdisc add dev eth0 root handle 1: htb default 90\r\ntc class add dev eth0 parent 1: classid 1:1 htb rate ${RATE}kbit ceil ${RATE}kbit\r\n\r\ntc class add dev eth0 parent 1:1 classid 1:10 htb rate 6000kbit ceil ${RATE}kbit\r\ntc class add dev eth0 parent 1:1 classid 1:20 htb rate 1000kbit ceil ${RATE}kbit\r\ntc class add dev eth0 parent 1:1 classid 1:50 htb rate 500kbit ceil ${RATE}kbit\r\ntc class add dev eth0 parent 1:1 classid 1:90 htb rate 500kbit ceil 500kbit\r\n\r\ntc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10\r\ntc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10\r\ntc qdisc add dev eth0 parent 1:50 handle 50: sfq perturb 10\r\ntc qdisc add dev eth0 parent 1:90 handle 90: sfq perturb 10\r\n\r\ntc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 80 0xffff classid 1:10\r\ntc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 22 0xffff classid 1:20\r\ntc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 25 0xffff classid 1:50\r\ntc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 110 0xffff classid 1:5\r\n<\/pre>\n<p>These commands create a <b>queue discipline<\/b> class tree (&quot;qdisc&quot;) that is a control structure that is used by <code>tc<\/code> to know how to shape or rate limit the network traffic. You can do pretty much anything, but the previous code creates one &quot;htb&quot; bucket for all the traffic. &quot;htb&quot; means <b>&quot;Hierarchy Token Bucket&quot;<\/b>. That is supposed to contain other &quot;buckets&quot; that can define different arbitrary bandwidth limits.<\/p>\n<p>Simplifying: the main &quot;htb&quot; bucket (named &quot;1:1&quot;) corresponds to the full pipe bandwidth, say 8Mbit\/s. Then under this one, we create other 4 buckets, named &quot;1:10&quot;, &quot;1:20&quot;, &quot;1:50&quot;, &quot;1:90&quot; of respectively, 6Mbit\/s, 1Mbit\/s, 500kbit\/s, 500kbit\/s. These ones are managed through the &quot;sfq&quot;, <b>&quot;Stochastic Fairness Queueing&quot;<\/b>. Read: everyone gets their fair piece of the pie :)<\/p>\n<p>So we have <b>these 4 different pipes, 6Mbit\/s, 1Mbit\/s, 500kbit\/s, 500kbit\/s.<\/b> After that, last block, we can decide which pipe should the traffic go through.<\/p>\n<p><code>sport 80<\/code> means that source (outgoing source, so it&#39;s your clients destination) port 80, where your HTTP is supposedly listening to, will get the big 6Mbit\/s slice, sport 22 (ssh) will get 1Mbit\/s, and so on&#8230;<\/p>\n<p>Now this <b>did work on my test machine<\/b>, and I could set the bandwidth limit, download a file with <code>wget<\/code> and see that the speed was exactly matching the desired one, while other connections were unlimited. However, when I tried to put this in production on the actual Varnish machines, the same script and settings didn&#39;t work.<\/p>\n<p>I figured I had to bandwidth limit the whole &quot;htb&quot; bucket, instead of limiting just the HTTP traffic. Which sucks, I guess. But nevertheless, it works. So, I&#39;ll copy\/paste the entire magic here for whoever might be interested. And maybe explain me why this doesn&#39;t work exactly like in my tests. Traffic measured with <code>iptraf<\/code> and <code>iftop<\/code> show consistent results.<\/p>\n<pre>\r\n!\/bin\/sh\r\n#\r\n# Set up bandwidth limiting for an interface \/ service. Based on &#39;tc&#39;.\r\n# Defaults can be overridden by \/etc\/default\/traffic-shaper\r\n# Cosimo, 2010\/07\/13\r\n#\r\nTC=\/sbin\/tc\r\n\r\ntest -f \/etc\/default\/traffic-shaper &amp;&amp; . \/etc\/default\/traffic-shaper\r\n\r\nIF=${IF:-eth0}\r\nRATE=${RATE:-100Mbit}\r\nHTTP_RATE=${HTTP_RATE:-50Mbit}\r\nHTTP_PORT=${HTTP_PORT:-80}\r\nSSH_RATE=${SSH_RATE:-500kbit}\r\n\r\necho &quot;[$IF] HTTP (:$HTTP_PORT) rate=$HTTP_RATE\/$RATE&quot;\r\necho &quot;[$IF] SSH  (:22) rate=$SSH_RATE&quot;\r\n\r\n#exit\r\n\r\nif [ &quot;x$1&quot; = &quot;xstop&quot; ]; then\r\n        echo &#39;Stopping traffic shaper...&#39;\r\n        $TC qdisc del dev $IF root &gt;\/dev\/null 2&gt;&amp;1 &amp;&amp; echo &#39;Done&#39;\r\n        exit\r\nelif [ &quot;x$1&quot; = &quot;xshow&quot; ]; then\r\n        $TC qdisc show dev $IF\r\n        exit\r\nelif [ &quot;x$1&quot; = &quot;xstats&quot; ]; then\r\n        $TC -d -s qdisc show dev $IF\r\n        exit\r\nfi\r\n\r\necho &quot;Traffic shaping setup ($HTTP_RATE\/$RATE) on port $HTTP_PORT.&quot;\r\necho &quot;Reserving $SSH_RATE for interactive sessions.&quot;\r\n\r\n$TC qdisc add dev $IF root handle 1: htb default 10\r\n\r\n# I should be using this line, but I had to replace it with the following\r\n### $TC class add dev $IF parent 1: classid 1:1 htb rate ${RATE} ceil ${RATE}\r\n$TC class add dev $IF parent 1: classid 1:1 htb rate ${HTTP_RATE} ceil ${RATE}\r\n\r\n# Doesn&#39;t seem to have any effect (?)\r\n$TC class add dev $IF parent 1:1 classid 1:10 htb rate ${HTTP_RATE} ceil ${RATE}\r\n$TC class add dev $IF parent 1:1 classid 1:90 htb rate ${SSH_RATE} ceil ${RATE}\r\n\r\n$TC qdisc add dev $IF parent 1:10 handle 10: sfq perturb 10\r\n$TC qdisc add dev $IF parent 1:90 handle 90: sfq perturb 10\r\n\r\n$TC filter add dev $IF parent 1:0 protocol ip u32 match ip sport $HTTP_PORT 0xffff classid 1:10\r\n$TC filter add dev $IF parent 1:0 protocol ip u32 match ip sport 22 0xffff match ip dport 22 0xffff classid 1:90\r\n<\/pre>\n<p>Have fun, but don&#39;t try this at home :)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I know, this might sound like blasphemy. After all, why on earth should one want to limit bandwidth, and serve your clients slower, when using Varnish to accelerate HTTP serving? Sounds silly. And it is, but if your Varnish servers are saturating your internet wires, and you have other services running, you might want to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[399,398,401,400,402,397,60],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Bandwidth limiting a varnish server - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bandwidth limiting a varnish server - Random hacking\" \/>\n<meta property=\"og:description\" content=\"I know, this might sound like blasphemy. After all, why on earth should one want to limit bandwidth, and serve your clients slower, when using Varnish to accelerate HTTP serving? Sounds silly. And it is, but if your Varnish servers are saturating your internet wires, and you have other services running, you might want to [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2010-07-13T16:24:32+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Bandwidth limiting a varnish server\",\"datePublished\":\"2010-07-13T16:24:32+00:00\",\"dateModified\":\"2010-07-13T16:24:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\"},\"wordCount\":612,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"bandwidth limiting\",\"htb\",\"qdisc\",\"sfq\",\"tc\",\"traffic shaping\",\"varnish\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\",\"name\":\"Bandwidth limiting a varnish server - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2010-07-13T16:24:32+00:00\",\"dateModified\":\"2010-07-13T16:24:32+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Bandwidth limiting a varnish server\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bandwidth limiting a varnish server - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/","og_locale":"en_US","og_type":"article","og_title":"Bandwidth limiting a varnish server - Random hacking","og_description":"I know, this might sound like blasphemy. After all, why on earth should one want to limit bandwidth, and serve your clients slower, when using Varnish to accelerate HTTP serving? Sounds silly. And it is, but if your Varnish servers are saturating your internet wires, and you have other services running, you might want to [&hellip;]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/","og_site_name":"Random hacking","article_published_time":"2010-07-13T16:24:32+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Bandwidth limiting a varnish server","datePublished":"2010-07-13T16:24:32+00:00","dateModified":"2010-07-13T16:24:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/"},"wordCount":612,"commentCount":0,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["bandwidth limiting","htb","qdisc","sfq","tc","traffic shaping","varnish"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/","name":"Bandwidth limiting a varnish server - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2010-07-13T16:24:32+00:00","dateModified":"2010-07-13T16:24:32+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/07\/bandwidth-limiting-a-varnish-server\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Bandwidth limiting a varnish server"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/477"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=477"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/477\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=477"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=477"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=477"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}