{"id":447,"date":"2011-07-23T14:55:37","date_gmt":"2011-07-23T13:55:37","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/"},"modified":"2011-07-23T14:55:37","modified_gmt":"2011-07-23T13:55:37","slug":"how-to-detect-tcp-retransmit-timeouts-in-your-network","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/","title":{"rendered":"How to detect TCP retransmit timeouts in your network"},"content":{"rendered":"<p>Some months ago, while investigating on a problem in our infrastructure, I put together a small tool to help detecting TCP retransmits happening during HTTP requests.<\/p>\n<p>TCP retransmissions can happen, for example, when a client sends a <b>SYN<\/b> packet to the server, the server responds with a SYN-ACK but, for any reason, the client never receives the SYN-ACK. In this case, the client correctly waits for a given time, called the <a href=\"http:\/\/www.ietf.org\/rfc\/rfc2988.txt\" rel=\"nofollow\">TCP Retransmission Timeout<\/a>. In the usual case, this time is set to <b>3 seconds<\/b>.<\/p>\n<p>There&#39;s probably a million reasons why the client may never receive a SYN-ACK. The one I&#39;ve seen more often is packet loss, which in turn can have lots of reasons, for example a malfunctioning or misconfigured network switch.<\/p>\n<p>However, you can immediately spot if your timeout\/hang problems are caused by TCP retransmission because they happen to cause response times that are unusually frequently distributed around 3, 9 and 21 seconds (and on, of course).<\/p>\n<p>In fact, the TCP retransmission timeout starts at 3 seconds, but if the client tries to resend after a timeout and still receives no answer, it doubles the wait to 6 s, so <b>the total response time will be 9 seconds<\/b>, assuming that the client now finally receives the SYN-ACK. Otherwise, 3 + 6 + 12 = 21, then 3 + 6 + 12 + 24 = 45 s and so on and so forth.<\/p>\n<p>So, this little tool fires a quick batch of HTTP requests to a given server and measures the response times, highlighting slow responses (&gt; 0.5s). If you see that the reported response times are 3.002s, 9.005s or similar, then you are probably in presence of TCP retransmission and\/or packet loss.<\/p>\n<p>Finally, <a href=\"https:\/\/gist.github.com\/1101500\" rel=\"nofollow\">here it is<\/a>:<\/p>\n<pre><code>\r\n#!\/usr\/bin\/env perl\r\n#\r\n# <a href=\"https:\/\/gist.github.com\/1101500\" rel=\"nofollow\" target=\"_blank\">https:\/\/gist.github.com\/1101500<\/a>\r\n#\r\n# Fires HTTP request batches at the specified hostname\r\n# and analyzes the response times.\r\n#\r\n# If you have suspicious frequency of 3.00x, 9.00x, 21.00x\r\n# seconds, then most probably you have a problem of packet loss\r\n# in your network.\r\n#\r\n# cosimo@opera.com, sometime in 2011\r\n#\r\n\r\nuse strict;\r\nuse LWP::UserAgent ();\r\nuse Time::HiRes ();\r\n\r\n$| = 1;\r\n\r\nmy $ua = LWP::UserAgent-&gt;new();\r\n$ua-&gt;agent(&quot;$0\/0.01&quot;);\r\n\r\n# Tests this hostname\r\nmy $server = $ARGV[0] || die &quot;Usage: $0 &lt;hostname&gt;n&quot;;\r\n\r\n# Picks the URLs to test in this list, one after the other\r\nmy @url_pool = qw(\r\n\t\/ping.html\r\n);\r\n\r\nmy $total_reqs = 0;\r\nmy $total_elapsed = 0.0;\r\nmy $n_pick = 0;\r\nmy $url_to_fire;\r\n\r\nmy $max_elapsed = 0.0;\r\nmy $max_elapsed_when = &#39;&#39;;\r\nmy $failed_reqs = 0;\r\nmy $slow_responses = 0;\r\nmy $terminate_now = 0;\r\n\r\nsub output_report {\r\n\tprint &quot;Report for:            $server at &quot; . localtime() . &quot;n&quot;;\r\n\tprintf &quot;Total requests:        %d in %.3f sn&quot;, $total_reqs, $total_elapsed;\r\n\tprint &quot;Failed requests:       $failed_reqsn&quot;;\r\n\tprint &quot;Slow responses (&gt;1s):  $slow_responses (slowest $max_elapsed s at $max_elapsed_when)n&quot;;\r\n\tprintf &quot;Average response time: %.3f s (%.3f req\/s)n&quot;, $total_elapsed \/ $total_reqs, $total_reqs \/ $total_elapsed;\r\n\tprint &quot;--------------------------------------------------------------------n&quot;;\r\n\tsleep 1;\r\n\treturn;\r\n}\r\n\r\n$SIG{INT} = sub { $terminate_now = 1 };\r\n\r\nwhile (not $terminate_now) {\r\n\r\n\t$url_to_fire = &quot;<a href=\"http:\/\/\" rel=\"nofollow\" target=\"_blank\">http:\/\/<\/a>&quot; . $server . $url_pool[$n_pick];\r\n\r\n\tmy $t0 = [ Time::HiRes::gettimeofday() ];\r\n\tmy $resp = $ua-&gt;get($url_to_fire);\r\n\tmy $elapsed = Time::HiRes::tv_interval($t0);\r\n\r\n\t$failed_reqs++ if ! $resp-&gt;is_success;\r\n\r\n\t$total_reqs++;\r\n\t$total_elapsed += $elapsed;\r\n\r\n\tif ($elapsed &gt; $max_elapsed) {\r\n\t\t$max_elapsed = $elapsed;\r\n\t\t$max_elapsed_when = scalar localtime;\r\n\t\tprintf &quot;[SLOW] %s, %s served in %.3f sn&quot;, $max_elapsed_when, $url_to_fire, $max_elapsed;\r\n\t}\r\n\r\n\t$slow_responses++ if $elapsed &gt;= 0.5;\r\n\t$n_pick = 0       if ++$n_pick &gt; $#url_pool;\r\n\toutput_report()   if $total_reqs &gt; 0 and ($total_reqs % 1000 == 0);\r\n\r\n}\r\ncontinue {\r\n    Time::HiRes::usleep(100000);\r\n}\r\n\r\noutput_report();\r\n\r\n# End\r\n<\/code><\/pre>\n<p>It&#39;s also published here on Github, <a href=\"https:\/\/gist.github.com\/1101500\" rel=\"nofollow\">https:\/\/gist.github.com\/1101500<\/a>. Have fun!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Some months ago, while investigating on a problem in our infrastructure, I put together a small tool to help detecting TCP retransmits happening during HTTP requests. TCP retransmissions can happen, for example, when a client sends a SYN packet to the server, the server responds with a SYN-ACK but, for any reason, the client never [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[299,75,50,296,298,76,263,297,48],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to detect TCP retransmit timeouts in your network - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to detect TCP retransmit timeouts in your network - Random hacking\" \/>\n<meta property=\"og:description\" content=\"Some months ago, while investigating on a problem in our infrastructure, I put together a small tool to help detecting TCP retransmits happening during HTTP requests. TCP retransmissions can happen, for example, when a client sends a SYN packet to the server, the server responds with a SYN-ACK but, for any reason, the client never [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2011-07-23T13:55:37+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"How to detect TCP retransmit timeouts in your network\",\"datePublished\":\"2011-07-23T13:55:37+00:00\",\"dateModified\":\"2011-07-23T13:55:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\"},\"wordCount\":291,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"network\",\"performance\",\"perl\",\"retransmission\",\"retransmit\",\"servers\",\"tcp\",\"timeout\",\"web\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\",\"name\":\"How to detect TCP retransmit timeouts in your network - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2011-07-23T13:55:37+00:00\",\"dateModified\":\"2011-07-23T13:55:37+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to detect TCP retransmit timeouts in your network\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to detect TCP retransmit timeouts in your network - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/","og_locale":"en_US","og_type":"article","og_title":"How to detect TCP retransmit timeouts in your network - Random hacking","og_description":"Some months ago, while investigating on a problem in our infrastructure, I put together a small tool to help detecting TCP retransmits happening during HTTP requests. TCP retransmissions can happen, for example, when a client sends a SYN packet to the server, the server responds with a SYN-ACK but, for any reason, the client never [&hellip;]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/","og_site_name":"Random hacking","article_published_time":"2011-07-23T13:55:37+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"How to detect TCP retransmit timeouts in your network","datePublished":"2011-07-23T13:55:37+00:00","dateModified":"2011-07-23T13:55:37+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/"},"wordCount":291,"commentCount":0,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["network","performance","perl","retransmission","retransmit","servers","tcp","timeout","web"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/","name":"How to detect TCP retransmit timeouts in your network - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2011-07-23T13:55:37+00:00","dateModified":"2011-07-23T13:55:37+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2011\/07\/how-to-detect-tcp-retransmit-timeouts-in-your-network\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"How to detect TCP retransmit timeouts in your network"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/447"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=447"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/447\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=447"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=447"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=447"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}