All,
We've developed a patch as proposed below, and have been running it in our
production environment (currently Tomcat 6.0.37) for several months. The
patch applies cleanly to the Tomcat 6.0.x, 7.0.x, and 8.0.x series, with
very minor variation in the exact line numbers.
The patch introduces the following System properties:
* org.apache.jasper.runtime.BodyContentImpl.TAG_BUFFER_SIZE
Allows overriding the initial tag buffer size; useful for applications
that make heavy use of tags, and very often exceed the default 512 bytes
defined in org.apache.jasper.Constants.DEFAULT_TAG_BUFFER_SIZE, eliminating
the need for reallocation of the buffer. Defaults
to Constants.DEFAULT_TAG_BUFFER_SIZE for backwards compatibility.
* org.apache.jasper.runtime.BodyContentImpl.MAX_TAG_BUFFER_SIZE
Introduces the maximum buffer size that should be reused by future tags;
if a buffer exceeds this limit, *and*
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER is set to true, the
buffer will be discarded, and a new buffer of
size org.apache.jasper.runtime.BodyContentImpl.TAG_BUFFER_SIZE will be
created. Defaults to current value
of org.apache.jasper.runtime.BodyContentImpl.TAG_BUFFER_SIZE, again for
backwards compatibility.
No changes will occur for users with the default configuration, or for
users with org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER set to
true. This patch only adds new, optional functionality.
In our production environment, we've been running with the following
settings:
* LIMIT_BUFFER=true
* TAG_BUFFER_SIZE=16384
* MAX_TAG_BUFFER_SIZE=65536
Our metrics show that for our application, we're experiencing significantly
less garbage collection of tag buffers (versus old behavior with
LIMIT_BUFFERS=true), while maintaining a more consistent memory profile
(versus old behavior with LIMIT_BUFFERS=false). With LIMIT_BUFFERS=false,
we previously found that requests for very large pages rendered within a
tag could cause memory usage to balloon, eventually leading to an OOME.
Please let me know if there's anything I can/should do to ensure this patch
is applied to the 6, 7, and 8 releases.
Thanks!
Scott Severtson
Lead Platform Architect
Digital Measures
On Wed, Nov 13, 2013 at 4:23 PM, Scott Severtson <
ssevert...@digitalmeasures.com> wrote:
> All,
>
> We've noticed that our Tomcat 6.0.37 instances have increasing overhead
> the longer they stay up. We've captured some heap dumps, expecting to
> find memory leaks within our code, but found something different instead.
>
> Jasper's BodyContentImpl is actually holding most of the non-collectible
> garbage, in the internal char[] buffer. I've checked trunk of Tomcat 6,
> 7 and 8, and see that the same implementation is still used, so this
> issue applies to all currently supported versions.
>
> This issue appears to be well known:
> *
> http://stackoverflow.com/questions/10421908/is-there-a-
> fix-for-bodycontentimpl-jsp-tag-memory-leak-using-tomcat
> * https://issues.apache.org/bugzilla/show_bug.cgi?id=43925
>
> Specifically what we see happening is due to large pages being rendered
> in JSPs - sometimes in the multi-megabyte range, especially for internal
> tools. Apparently, some of the tags we use require buffering the output,
> versus sending it directly to the Writer.
>
> Now, we certainly can simply set LIMIT_BUFFER and forget about it (throw
> away any buffers larger than 8k), but this isn't optimal for our
> application. For about 75% of our requests, ~8k is the baseline page
> size, with many larger than that. So, depending how the data size, we
> could be reallocating multiple times, only to throw away the buffer at
> the end.
>
> We'd like to propose a different behavior, and are wondering what others
> think about it. If it sounds like a good idea, we're certainly willing
> to submit patches to implement it.
>
> --Proposal--
> Instead of a hard-coded DEFAULT_TAG_BUFFER_SIZE of 8k, make it
> configurable. In addition, if LIMIT_BUFFER is set, provide another
> configurable setting called something like MAX_TAG_BUFFER_SIZE (if not
> set, defaults to DEFAULT_TAG_BUFFER_SIZE). This would give applications
> a more predictable *range* of memory allocated for tag buffers, while
> decreasing the frequency of reallocation.
> --/Proposal--
>
> We'd probably set DEFAULT_TAG_BUFFER_SIZE to 16k, and set
> MAX_TAG_BUFFER_SIZE to either 64k or 128k, based on real-world
> performance testing.
>
>
> One other, slightly less defined question - is there a reason we're
> managing char arrays ourselves, instead of using something like Java's
> StringBuilder or Javalution's TextBuilder? StringBuilder would reduce
> code maintenance, while something like TextBuilder would also reduce
> array allocation.
>
> We'd love to hear your thoughts.
>
> Thanks,
> Scott Severtson
> Lead Platform Architect
> Digital Measures
>
Index: java/org/apache/jasper/runtime/BodyContentImpl.java
===================================================================
--- java/org/apache/jasper/runtime/BodyContentImpl.java (revision 1554528)
+++ java/org/apache/jasper/runtime/BodyContentImpl.java (working copy)
@@ -43,7 +43,13 @@
System.getProperty("line.separator");
private static final boolean LIMIT_BUFFER =
Boolean.valueOf(System.getProperty("org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER", "false")).booleanValue();
-
+ private static final int TAG_BUFFER_SIZE =
+ Integer.valueOf(System.getProperty("org.apache.jasper.runtime.BodyContentImpl.TAG_BUFFER_SIZE",
+ String.valueOf(Constants.DEFAULT_TAG_BUFFER_SIZE))).intValue();
+ private static final int MAX_TAG_BUFFER_SIZE =
+ Integer.valueOf(System.getProperty("org.apache.jasper.runtime.BodyContentImpl.MAX_TAG_BUFFER_SIZE",
+ String.valueOf(TAG_BUFFER_SIZE))).intValue();
+
private char[] cb;
private int nextChar;
private boolean closed;
@@ -56,7 +62,7 @@
*/
public BodyContentImpl(JspWriter enclosingWriter) {
super(enclosingWriter);
- cb = new char[Constants.DEFAULT_TAG_BUFFER_SIZE];
+ cb = new char[TAG_BUFFER_SIZE];
bufferSize = cb.length;
nextChar = 0;
closed = false;
@@ -468,8 +474,8 @@
throw new IOException();
} else {
nextChar = 0;
- if (LIMIT_BUFFER && (cb.length > Constants.DEFAULT_TAG_BUFFER_SIZE)) {
- cb = new char[Constants.DEFAULT_TAG_BUFFER_SIZE];
+ if (LIMIT_BUFFER && (cb.length > MAX_TAG_BUFFER_SIZE)) {
+ cb = new char[TAG_BUFFER_SIZE];
bufferSize = cb.length;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
