{"id":869,"date":"2019-06-06T18:20:21","date_gmt":"2019-06-06T16:20:21","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/?p=869"},"modified":"2019-06-06T18:20:21","modified_gmt":"2019-06-06T16:20:21","slug":"jvm-memory-usage-monitoring-tools","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/","title":{"rendered":"JVM memory usage monitoring tools"},"content":{"rendered":"

During the last few days I looked into JVM memory monitoring tools.<\/p>\n

That is one area of expertise that I am definitely not familiar with. The general approach in my career so far has been to avoid Java for long running processes :-) The only other (empirical) pearl of wisdom I remember from past experiences with Java\u00e2\u201e\u00a2 is that the max heap memory should be less or equal to the max system memory divided by 2 (maxHeap <= totalMem \/ 2<\/code>). I\u00e2\u20ac\u2122ve seen this relation work nicely over the years for a fairly busy Solr search cluster with anywhere from 24 to 96Gb memory servers, where maxHeap was never higher than 8Gb though, as those servers had a lot<\/em> of other processes running at the same time. Obviously if other processes are running concurrently on the system, the max heap size should be decreased accordingly.<\/p>\n

When observing servers, it’s typical to monitor the \u00e2\u20ac\u0153usable\u00e2\u20ac\u009d memory (in Datadog, that would be system.mem.usable<\/code>) which is a sum of the system current free and cached or buffered memory. That is, the total memory that the system can grab and use at any time if needed.<\/p>\n

This measure doesn\u00e2\u20ac\u2122t necessarily tell us which processes are using memory and why. We\u00e2\u20ac\u2122ve also observed spikes of memory usage when log aggregation tools like filebeat are reading, parsing and shipping logs to the logging servers. It would be useful to start tracking how much memory is used by the specific merlin java process rather than looking at an aggregated memory metric.<\/p>\n

Searching around, I found a few useful articles (among them https:\/\/www.pushtechnology.com\/support\/kb\/understanding-the-java-virtual-machine-heap-for-high-performance-applications\/<\/a>) and several tools that helped dig deeper and extract more information about what is happening on the gameservers with regards to memory usage. I\u00e2\u20ac\u2122d like to mention them here for future reference and to collect eventual feedback from others.<\/p>\n

jvmtop<\/h2>\n

\u00e2\u20ac\u2039<\/span><\/span>URL: github.com\/patric-r\/jvmtop<\/a><\/p>\n

Simple console application that provides high level stats about heap usage and garbage collection CPU usage. Here\u00e2\u20ac\u2122s an example from the documentation page:<\/p>\n

\n
\n
JvmTop 0.8.0 alpha   amd64  8 cpus, Linux 2.6.32-27, load avg 0.12\r\n https:\/\/github.com\/patric-r\/jvmtop\r\n\r\n  PID MAIN-CLASS      HPCUR HPMAX NHCUR NHMAX    CPU     GC    VM USERNAME   #T DL\r\n 3370 rapperSimpleApp  165m  455m  109m  176m  0.12%  0.00% S6U37 web        21\r\n11272 ver.resin.Resin [ERROR: Could not attach to VM]\r\n27338 WatchdogManager   11m   28m   23m  130m  0.00%  0.00% S6U37 web        31\r\n19187 m.jvmtop.JvmTop   20m 3544m   13m  130m  0.93%  0.47% S6U37 web        20\r\n16733 artup.Bootstrap  159m  455m  166m  304m  0.12%  0.00% S6U37 web        46<\/code><\/pre>\n<\/div>\n<\/div>\n
where the various columns are:<\/p>\n
\n
\n
PID = process id\r\nMAIN-CLASS = the \"jvm name\" but often the entry point class (with used main() method)\r\nHPCUR = currently used heap memory\r\nHPMAX = maximum heap memory the jvm can allocate\r\nNHCUR = currently used non-heap memory (e.g. PermGen)\r\nNHMAX = maximum non-heap memory the jvm can allocate\r\nCPU = CPU utilization\r\nGC = percentage of time spent in garbage collection (~100% means that the process does garbage collection only)\r\nVM = Shows JVM vendor, java version and release number (S6U37 = Sun JVM 6, Update 37)\r\nUSERNAME = Username which owns this jvm process\r\n#T = Number of jvm threads\r\nDL = If !D is shown if the jvm detected a thread deadlock<\/code><\/pre>\n<\/div>\n<\/div>\n
Useful to get a quick glance at a few critical parameters. I have tested the most recent version (0.9.0) and the compilation from the source code was quick and easy.<\/p>\n
jvm-mon<\/h2>\n
\u00e2\u20ac\u2039<\/span><\/span>URL: github.com\/ajermakovics\/jvm-mon<\/a><\/p>\n
Another console application, but a bit more sophisticated than jvmtop<\/span>. It also displays trends as it\u00e2\u20ac\u2122s meant to be run for a longer period of time. I think this is the appropriate level of fancy I like :-)<\/p>\n
\"\"<\/a>I quite like jvm-mon, it’s clear and the data is easy to understand. The charts resize dynamically based on how long you keep it running.<\/p>\n
jps_stat<\/h2>\n
URL: github.com\/amarjeetanandsingh\/jps_stat<\/a><\/p>\n
A simpler shell script that displays more or less the same information shown by jvmtop<\/span> and also keeps running and updating the stats. AFAIK, it\u00e2\u20ac\u2122s not possible to run it \u00e2\u20ac\u0153one-shot\u00e2\u20ac\u009d. That would be useful to build our own metric monitoring.<\/p>\n
This is what jps_stat<\/span> looks like when executed:\"jps_stat<\/a><\/p>\n
jstat<\/h2>\n
https:\/\/docs.oracle.com\/javase\/7\/docs\/technotes\/tools\/share\/jstat.html<\/a><\/p>\n
Last of the lot is jstat<\/code> which is part of the JVM distribution. Usage is very simple and it can easily be embedded in a one-liner or a shell script, as in:<\/p>\n
watch -d -n1 jstat -gc $(pidof java)<\/code><\/pre>\n
`jstat -gc`<\/span> outputs garbage collection statistics, some of which I still haven\u00e2\u20ac\u2122t understood the purpose. A sample output is the following:<\/p>\n
Every 1.0s: jstat -gc 12743                                                                                                                                                                Thu Jun  6 09:33:23 2019\r\n\r\n S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT\r\n34048.0 34048.0 10229.3  0.0   272640.0 37062.4  2731264.0  1519963.6  32192.0 30728.8 3804.0 3510.3  34926  873.839  26      2.781  876.620<\/code><\/pre>\n
These metrics are a bit more detailed than just looking at heap usage as a whole. I’m not really sure I need to look into all of this specifically.<\/p>\n
Conclusion<\/h2>\n
Not much to say, but for now it was useful to get a bit more details of how the server is running, how much heap memory is consumed, how much non-heap memory is used, and how the total memory used by the java process itself compares to the total memory used on the server.<\/p>\n
Plus, it’s handy to write down these things so I can refer back to them when needed :-)<\/p>\n","protected":false},"excerpt":{"rendered":"
During the last few days I looked into JVM memory monitoring tools. That is one area of expertise that I am definitely not familiar with. The general approach in my career so far has been to avoid Java for long running processes :-) The only other (empirical) pearl of wisdom I remember from past experiences […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"yoast_head":"\nJVM memory usage monitoring tools - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"JVM memory usage monitoring tools - Random hacking\" \/>\n<meta property=\"og:description\" content=\"During the last few days I looked into JVM memory monitoring tools. That is one area of expertise that I am definitely not familiar with. The general approach in my career so far has been to avoid Java for long running processes :-) The only other (empirical) pearl of wisdom I remember from past experiences […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2019-06-06T16:20:21+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"JVM memory usage monitoring tools\",\"datePublished\":\"2019-06-06T16:20:21+00:00\",\"dateModified\":\"2019-06-06T16:20:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\"},\"wordCount\":675,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"image\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\",\"name\":\"JVM memory usage monitoring tools - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png\",\"datePublished\":\"2019-06-06T16:20:21+00:00\",\"dateModified\":\"2019-06-06T16:20:21+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage\",\"url\":\"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png\",\"contentUrl\":\"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"JVM memory usage monitoring tools\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"JVM memory usage monitoring tools - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/","og_locale":"en_US","og_type":"article","og_title":"JVM memory usage monitoring tools - Random hacking","og_description":"During the last few days I looked into JVM memory monitoring tools. That is one area of expertise that I am definitely not familiar with. The general approach in my career so far has been to avoid Java for long running processes :-) The only other (empirical) pearl of wisdom I remember from past experiences […]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/","og_site_name":"Random hacking","article_published_time":"2019-06-06T16:20:21+00:00","og_image":[{"url":"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png"}],"author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"JVM memory usage monitoring tools","datePublished":"2019-06-06T16:20:21+00:00","dateModified":"2019-06-06T16:20:21+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/"},"wordCount":675,"commentCount":1,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"image":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage"},"thumbnailUrl":"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/","name":"JVM memory usage monitoring tools - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage"},"image":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage"},"thumbnailUrl":"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png","datePublished":"2019-06-06T16:20:21+00:00","dateModified":"2019-06-06T16:20:21+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#primaryimage","url":"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png","contentUrl":"http:\/\/www.streppone.it\/cosimo\/blog\/wp-content\/uploads\/2019\/06\/jvm-mon-1024x691.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2019\/06\/jvm-memory-usage-monitoring-tools\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"JVM memory usage monitoring tools"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/869"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=869"}],"version-history":[{"count":1,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/869\/revisions"}],"predecessor-version":[{"id":872,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/869\/revisions\/872"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=869"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=869"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=869"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}