(This question has also been published on StackOveflow <https://stackoverflow.com/q/44484940/416300>)
I am looking for a way to monitor memory usage of YARN containers over time. Specifically - given a YARN application-id, how can you get a graph, showing the memory usage of each of its containers over time? The main goal is to better fit memory allocation requirements for our YARN applications (Spark / Map-Reduce), to avoid over allocation and cluster resource waste. A side goal would be the ability to debug memory issues when developing our jobs and attempting to pick reasonable resource allocations. We've tried using the Data-Dog integration, But it doesn't break down the metrics by container. Another approach was to parse the hadoop-yarn logs. These logs have messages like: Memory usage of ProcessTree 57251 for container-id container_e116_1495951495692_35134_01_000001: 1.9 GB of 11 GB physical memory used; 14.4 GB of 23.1 GB virtual memory used Parsing the logs correctly can yield data that can be used to plot a graph of memory usage over time. That's exactly what we want, but there are two downsides: It involves reading human-readable log lines and parsing them into numeric data. We'd love to avoid that. If this data can be consumed otherwise, we're hoping it'll have more information that we might be interest in in the future. We wouldn't want to put the time into parsing the logs just to realize we need something else. Is there any other way to extract these metrics, either by plugging in to an existing producer or by writing a simple listener? Perhaps a whole other approach? -- [image: Logo] <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Shmuel Blitz *Big Data Developer* www.similarweb.com <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Like Us <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Follow Us <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Watch Us <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Read Us <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
