[Bug 56713] Limit time that incoming request waits while webapp is reloading
https://issues.apache.org/bugzilla/show_bug.cgi?id=56713 --- Comment #2 from Mark Thomas --- Remy highlighted the potential problems of pausing requests during reload (large numbers of threads blocking waiting for the reload) and I agree that this could be a problem. However, the only user complaints we have had about this feature is that it wasn't working or that it could work better. No-one has complained about large numbers of threads blocking. Givent that no-one has complained about large numbers of blocked threads on reloads, I am wondering if there is really a problem here that needs fixing. Regarding the questions on proposed solution: How is this going to work if it is the ROOT context that is being reloaded? (1) a) and d) have obvious issues. I'd suggest configuration on the host with the ability to override it on the context if desired (2) There might not be a ROOT application (there should be but there might not). How is the Context going to be re-registered once it has started? I don't think any user is not going to want the context to be available eventually. (3) Presumably the Context eventually starts and then is shutdown. I was about to say that there are lots of complexities here - just like automatic deployment that led to producing this page: http://tomcat.apache.org/tomcat-8.0-doc/config/automatic-deployment.html Reading that it occurred to me that really what we are talking about here is the difference between reloading and redeploying. Would it not be simpler to leave this in the hands of the system administrator? They can choose whether to redploy or reload (if reload is an option for the change they want to make). One thing we could do is add some logging to notify the system admin that threads have been blocked and that they can use redeploy to avoid this. I was thinking along the lines of one log message the first time a thread was blocked and then further messages every TBD (15?, background thread?) seconds giving the current number of waiting threads until the context is reloaded. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
ContainerBackgroundProcessor and compounding OOMEs
All,
It seems that ContainerBackgroundProcessor can die and end up silently
destabilizing things.
I got this in a development environment today:
Exception in thread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:916)
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(File.java:937)
at
org.apache.naming.resources.FileDirContext$FileResourceAttributes.get
LastModified(FileDirContext.java:1008)
at
org.apache.naming.resources.FileDirContext$FileResourceAttributes.get
Creation(FileDirContext.java:980)
at
org.apache.naming.resources.FileDirContext$FileResourceAttributes.(FileDirContext.java:925)
at
org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirCon
text.java:403)
at
org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirConte
xt.java:1137)
at
org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirConte
xt.java:1090)
at
org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirCon
text.java:882)
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Okay, this kind of thing sometimes happens in our development instance.
But what I noticed is that the ContainerBackgroundProcessor thread has
not died.
That means that even if the OOME was transient and the JVM could recover
from the failure, the background processor thread is dead and things
like sessions will pile up until memory is truly exhausted.
The problem is in the code for the runnable method. Somewhat simplified,
it's just a loop to do stuff:
while(!done) {
try {
sleep();
processChildren();
} catch (Throwable t) {
ExceptionUtils.handleException(t);
log("error", t);
}
}
Although the stack trace doesn't show it, the above error clearly
occurred in processChildren().
ExceptionUtils.handleException checks for two things that I think we
might want to change:
1. If the exception is StackOverflowError, it silently ignores the error
and continues. I think we should at least log something, probably at
WARN level.
2. If the exception is VirtualMachineError, it gets re-thrown with no
log. This skips the "log" call in the above code and so the only log
will come from the VM's "unhandled exception" logger which may not go
where you expect it to go. The exception propagates, and the thread's
run() method finishes (escapes due to uncaught exception). After that,
regardless of the recoverability of the situation (OOME), the background
processor will not run and therefore no auto-reload applications will
auto-reload, no sessions will ever die, etc.
If we think that StackOverflowError is recoverable, why not OutOfMemory?
What about other VirtualMachineErrors?
-chris
signature.asc
Description: OpenPGP digital signature
Re: ContainerBackgroundProcessor and compounding OOMEs
On 14/07/2014 17:01, Christopher Schultz wrote:
> All,
>
> It seems that ContainerBackgroundProcessor can die and end up
> silently destabilizing things.
It is only silent if the system admin has done something stupid like
redirecting stderr to /dev/null. Frankly, if they do that they deserve
everything they get.
> That means that even if the OOME was transient and the JVM could
> recover from the failure,
I'd argue that an OOME is never recoverable. Once it occurs you have
no idea what allocations may have failed. Even if the thread where the
error is reported can recover from the OOME, you have no way of
determining if any other threads were affected or how they were affected.
> the background processor thread is dead and things like sessions
> will pile up until memory is truly exhausted.
>
> The problem is in the code for the runnable method. Somewhat
> simplified, it's just a loop to do stuff:
>
> while(!done) { try { sleep(); processChildren(); } catch (Throwable
> t) { ExceptionUtils.handleException(t); log("error", t); } }
>
> Although the stack trace doesn't show it, the above error clearly
> occurred in processChildren().
>
> ExceptionUtils.handleException checks for two things that I think
> we might want to change:
>
> 1. If the exception is StackOverflowError, it silently ignores the
> error and continues. I think we should at least log something,
> probably at WARN level.
No it doesn't. Go and read the code again. The exception is logged at
ERROR level.
> 2. If the exception is VirtualMachineError, it gets re-thrown with
> no log. This skips the "log" call in the above code and so the only
> log will come from the VM's "unhandled exception" logger which may
> not go where you expect it to go.
It goes to stderr unless they system admin has redirected it and if
they have, they should know where to look for it. Further, they should
also be monitoring it.
> The exception propagates, and the thread's run() method finishes
> (escapes due to uncaught exception). After that, regardless of the
> recoverability of the situation (OOME), the background processor
> will not run and therefore no auto-reload applications will
> auto-reload, no sessions will ever die, etc.
>
> If we think that StackOverflowError is recoverable, why not
> OutOfMemory?
There is no assumption that StackOverflowError is recoverable. That is
why it is logged at ERROR level. It is guaranteed that only that
thread was affected so you know (from the stack trace) exactly what
failed where and you can also determine how bad things are and opt to
restart Tomcat if necessary.
With respect to OOME how, exactly, do you propose to differentiate
between a "recoverable" OOME and a non-recoverable one?
> What about other VirtualMachineErrors?
The current position is that they are all non-recoverable. If it can
be demonstrated that more of them should be treated like StackOverflow
then we can do so.
Mark
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Tomcat 8 Documentation JNDI Datasource Howto
Hi, On this page http://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.html there is an example of a global JDBC resource being defined. There is also a comment saying "This author has not had success here, although others have reported so. Clarification would be appreciated here." I defined the global postgres jdbc definition, and then I add a resource link in the two web application contexts that I wanted to use the datasource with, and this resolved the issue. If I did not define the resource links in the context elements I would receive an error that the driver was not present. Is this the correct approach to solving this problem? If so, should I document it this way? Basically: 1. Define the global JDBC resource 2. Define the resource references in the contexts you would like to share the datasource with. -- With Regards, Andrew Carr e. andrewlanec...@gmail.com w. andrew.c...@openlogic.com h. 4235255668 c. 4239489852 a. 101 Francis Drive, Greeneville, TN, 37743
Re: Tomcat 8 Documentation JNDI Datasource Howto
On Mon, Jul 14, 2014 at 2:10 PM, Andrew Carr wrote: > Hi, > > On this page > http://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.html > there is an example of a global JDBC resource being defined. There is also > a comment saying "This author has not had success here, although others > have reported so. Clarification would be appreciated here." > Not sure about the why behind this comment, maybe someone else can comment on the comment :) You should be able to define global resources though. If global resources don't work for you then it probably due to a configuration error. Most often, people forget to add resource links. > > I defined the global postgres jdbc definition, and then I add a resource > link in the two web application contexts that I wanted to use the > datasource with, and this resolved the issue. > Sounds right. > > If I did not define the resource links in the context elements I would > receive an error that the driver was not present. > That's correct. A global resource is not exposed to any applications by default. You need to define resource links to indicate which applications can see a given global resource. > > Is this the correct approach to solving this problem? It's one correct way. You can also just define a Resource tag directly in your context. See here. http://tomcat.apache.org/tomcat-8.0-doc/config/context.html#Resource_Definitions The difference being a global resource can be shared across multiple applications, while a resource defined directly in the context (even in conf/context.xml) is specific to that application. > If so, should I > document it this way? Basically: > > 1. Define the global JDBC resource > 2. Define the resource references in the contexts you would like to share > the datasource with. > Does this documentation link help to clarify? http://tomcat.apache.org/tomcat-8.0-doc/jndi-resources-howto.html#Global_configuration If not and you feel there's room for improvement, I think the general suggestion is that patches are welcome. Dan
Re: ContainerBackgroundProcessor and compounding OOMEs
Mark,
On 7/14/14, 12:18 PM, Mark Thomas wrote:
>> ExceptionUtils.handleException checks for two things that I think
>> we might want to change:
>>
>> 1. If the exception is StackOverflowError, it silently ignores the
>> error and continues. I think we should at least log something,
>> probably at WARN level.
>
> No it doesn't. Go and read the code again. The exception is logged at
> ERROR level.
Sorry, yes: the StackOverflow is logged.
>> 2. If the exception is VirtualMachineError, it gets re-thrown with
>> no log. This skips the "log" call in the above code and so the only
>> log will come from the VM's "unhandled exception" logger which may
>> not go where you expect it to go.
>
> It goes to stderr unless they system admin has redirected it and if
> they have, they should know where to look for it. Further, they should
> also be monitoring it.
True, but the background thread does die.
>> The exception propagates, and the thread's run() method finishes
>> (escapes due to uncaught exception). After that, regardless of the
>> recoverability of the situation (OOME), the background processor
>> will not run and therefore no auto-reload applications will
>> auto-reload, no sessions will ever die, etc.
>>
>> If we think that StackOverflowError is recoverable, why not
>> OutOfMemory?
>
> There is no assumption that StackOverflowError is recoverable.
public static void handleThrowable(Throwable t) {
if (t instanceof ThreadDeath) {
throw (ThreadDeath) t;
}
if (t instanceof StackOverflowError) {
// Swallow silently - it should be recoverable
return;
}
if (t instanceof VirtualMachineError) {
throw (VirtualMachineError) t;
}
// All other instances of Throwable will be silently swallowed
}
As of April 17, 2014, markt disagrees with you (r1588269):
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/jasper/util/ExceptionUtils.java?r1=1588269&r2=1588268&pathrev=1588269
> That is why it is logged at ERROR level.
You are right, StackOverflow gets logged, but things like OOME are not
-- they are left to the unhandled exception handler. But they do
take-down the background thread when they do.
> It is guaranteed that only that
> thread was affected so you know (from the stack trace) exactly what
> failed where and you can also determine how bad things are and opt to
> restart Tomcat if necessary.
For OOME, if you are using sessions you will need to schedule a restart
pretty much immediately, since they will never die.
> With respect to OOME how, exactly, do you propose to differentiate
> between a "recoverable" OOME and a non-recoverable one?
If the admin is monitoring (which we are), then they can make the
determination. The current situation is that Tomcat needs to be bounced
regardless of the severity of the OOME, because the background thread stops.
>> What about other VirtualMachineErrors?
>
> The current position is that they are all non-recoverable. If it can
> be demonstrated that more of them should be treated like StackOverflow
> then we can do so.
Okay. I think OOME should be treated similarly to StackOverflowError.
-chris
signature.asc
Description: OpenPGP digital signature
svn commit: r1610562 - in /tomcat/trunk: java/org/apache/catalina/mapper/MapperListener.java test/org/apache/catalina/mapper/TestMapperListener.java webapps/docs/changelog.xml
Author: kkolinko
Date: Mon Jul 14 23:57:35 2014
New Revision: 1610562
URL: http://svn.apache.org/r1610562
Log:
Fix https://issues.apache.org/bugzilla/show_bug.cgi?id=56717
Fix duplicate registration of MapperListener during repeated starts of embedded
Tomcat.
Properly unregister it when Tomcat stops.
Added:
tomcat/trunk/test/org/apache/catalina/mapper/TestMapperListener.java
(with props)
Modified:
tomcat/trunk/java/org/apache/catalina/mapper/MapperListener.java
tomcat/trunk/webapps/docs/changelog.xml
Modified: tomcat/trunk/java/org/apache/catalina/mapper/MapperListener.java
URL:
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/mapper/MapperListener.java?rev=1610562&r1=1610561&r2=1610562&view=diff
==
--- tomcat/trunk/java/org/apache/catalina/mapper/MapperListener.java (original)
+++ tomcat/trunk/java/org/apache/catalina/mapper/MapperListener.java Mon Jul 14
23:57:35 2014
@@ -116,6 +116,9 @@ public class MapperListener extends Life
@Override
public void stopInternal() throws LifecycleException {
setState(LifecycleState.STOPPING);
+
+Engine engine = (Engine) service.getContainer();
+removeListeners(engine);
}
Added: tomcat/trunk/test/org/apache/catalina/mapper/TestMapperListener.java
URL:
http://svn.apache.org/viewvc/tomcat/trunk/test/org/apache/catalina/mapper/TestMapperListener.java?rev=1610562&view=auto
==
--- tomcat/trunk/test/org/apache/catalina/mapper/TestMapperListener.java (added)
+++ tomcat/trunk/test/org/apache/catalina/mapper/TestMapperListener.java Mon
Jul 14 23:57:35 2014
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.catalina.mapper;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.catalina.Container;
+import org.apache.catalina.ContainerListener;
+import org.apache.catalina.Context;
+import org.apache.catalina.LifecycleException;
+import org.apache.catalina.LifecycleListener;
+import org.apache.catalina.startup.Tomcat;
+import org.apache.catalina.startup.TomcatBaseTest;
+import org.apache.tomcat.util.buf.ByteChunk;
+import org.apache.tomcat.websocket.server.WsContextListener;
+
+public class TestMapperListener extends TomcatBaseTest {
+
+@Test
+public void testTomcatRestartListenerCount_Bug56717() throws IOException,
+LifecycleException {
+// The test runs Tomcat twice, tests that it has started successfully,
+// and compares the counts of listeners registered on containers
+// after the first and the second starts.
+// Sample request is from TestTomcat#testSingleWebapp()
+
+Tomcat tomcat = getTomcatInstance();
+
+File appDir = new File(getBuildDirectory(), "webapps/examples");
+// app dir is relative to server home
+Context ctxt = tomcat.addWebapp(null, "/examples",
+appDir.getAbsolutePath());
+ctxt.addApplicationListener(WsContextListener.class.getName());
+tomcat.start();
+
+ByteChunk res;
+String text;
+res = getUrl("http://localhost:"; + getPort()
++ "/examples/servlets/servlet/HelloWorldExample");
+text = res.toString();
+Assert.assertTrue(text, text.contains("Hello World!"));
+
+List listenersFirst = new ArrayList<>();
+populateListenersInfo(listenersFirst, tomcat.getEngine());
+
+tomcat.stop();
+tomcat.start();
+
+res = getUrl("http://localhost:"; + getPort()
++ "/examples/servlets/servlet/HelloWorldExample");
+text = res.toString();
+Assert.assertTrue(text, text.contains("Hello World!"));
+
+List listenersSecond = new ArrayList<>();
+populateListenersInfo(listenersSecond, tomcat.getEngine());
+
+Assert.assertEquals(listenersFirst.size(), listenersSecond.size());
+for (int i = 0, len = listenersFirst.size(); i < len; i++) {
+
svn commit: r1610564 - in /tomcat/trunk: java/org/apache/catalina/mapper/LocalStrings.properties java/org/apache/catalina/mapper/Mapper.java test/org/apache/catalina/mapper/TestMapper.java
Author: kkolinko
Date: Tue Jul 15 00:45:55 2014
New Revision: 1610564
URL: http://svn.apache.org/r1610564
Log:
Fix https://issues.apache.org/bugzilla/show_bug.cgi?id=56717
Allow repeated Mapper.addHost() calls for the same host. E.g. if the host has
been implicitly added by addContextVersion() call.
Modified:
tomcat/trunk/java/org/apache/catalina/mapper/LocalStrings.properties
tomcat/trunk/java/org/apache/catalina/mapper/Mapper.java
tomcat/trunk/test/org/apache/catalina/mapper/TestMapper.java
Modified: tomcat/trunk/java/org/apache/catalina/mapper/LocalStrings.properties
URL:
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/mapper/LocalStrings.properties?rev=1610564&r1=1610563&r2=1610564&view=diff
==
--- tomcat/trunk/java/org/apache/catalina/mapper/LocalStrings.properties
(original)
+++ tomcat/trunk/java/org/apache/catalina/mapper/LocalStrings.properties Tue
Jul 15 00:45:55 2014
@@ -13,6 +13,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+mapper.addHost.success=Registered host [{0}]
+mapper.addHost.sameHost=Duplicate registration of the same host [{0}]. Ignored.
+mapper.addHostAlias.success=Registered alias [{0}] for host [{1}]
+mapper.addHostAlias.sameHost=Duplicate registration of alias [{0}] for the
same host [{1}]. Ignored.
mapper.removeWrapper=Removing wrapper from Context [{0}] with path [{1}]
mapper.duplicateHost=Duplicate Host [{0}]. The name is already used by Host
[{1}]. This Host will be ignored.
mapper.duplicateHostAlias=Duplicate host Alias [{0}] in Host [{1}]. The name
is already used by Host [{2}]. This Alias will be ignored.
Modified: tomcat/trunk/java/org/apache/catalina/mapper/Mapper.java
URL:
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/mapper/Mapper.java?rev=1610564&r1=1610563&r2=1610564&view=diff
==
--- tomcat/trunk/java/org/apache/catalina/mapper/Mapper.java (original)
+++ tomcat/trunk/java/org/apache/catalina/mapper/Mapper.java Tue Jul 15
00:45:55 2014
@@ -96,12 +96,25 @@ public final class Mapper {
MappedHost newHost = new MappedHost(name, host);
if (insertMap(hosts, newHosts, newHost)) {
hosts = newHosts;
+if (log.isDebugEnabled()) {
+log.debug(sm.getString("mapper.addHost.success", name));
+}
} else {
MappedHost duplicate = hosts[find(hosts, name)];
-log.error(sm.getString("mapper.duplicateHost", name,
-duplicate.getRealHostName()));
-// Do not add aliases, as removeHost(hostName) won't be able to
remove them
-return;
+if (duplicate.object == host) {
+// The host is already registered in the mapper.
+// E.g. it might have been added by addContextVersion()
+if (log.isDebugEnabled()) {
+log.debug(sm.getString("mapper.addHost.sameHost", name));
+}
+newHost = duplicate;
+} else {
+log.error(sm.getString("mapper.duplicateHost", name,
+duplicate.getRealHostName()));
+// Do not add aliases, as removeHost(hostName) won't be able to
+// remove them
+return;
+}
}
List newAliases = new ArrayList<>(aliases.length);
for (String alias : aliases) {
@@ -158,6 +171,10 @@ public final class Mapper {
MappedHost[] newHosts = new MappedHost[hosts.length + 1];
if (insertMap(hosts, newHosts, newAlias)) {
hosts = newHosts;
+if (log.isDebugEnabled()) {
+log.debug(sm.getString("mapper.addHostAlias.success",
+newAlias.name, newAlias.getRealHostName()));
+}
return true;
} else {
MappedHost duplicate = hosts[find(hosts, newAlias.name)];
@@ -165,6 +182,10 @@ public final class Mapper {
// A duplicate Alias for the same Host.
// A harmless redundancy. E.g.
// localhost
+if (log.isDebugEnabled()) {
+log.debug(sm.getString("mapper.addHostAlias.sameHost",
+newAlias.name, newAlias.getRealHostName()));
+}
return false;
}
log.error(sm.getString("mapper.duplicateHostAlias", newAlias.name,
Modified: tomcat/trunk/test/org/apache/catalina/mapper/TestMapper.java
URL:
http://svn.apache.org/viewvc/tomcat/trunk/test/org/apache/catalina/mapper/TestMapper.java?rev=1610564&r1=1610563&r2=1610564&view=diff
==
--- tomcat/trunk/test/org/apache/catalina/mapper/TestMapper.java (original
[Bug 56717] MapperListener does not remove its listeners. It matters for repeated starts of embedded Tomcat,
https://issues.apache.org/bugzilla/show_bug.cgi?id=56717 Konstantin Kolinko changed: What|Removed |Added Component|Catalina|Catalina Version|8.0.9 |7.0.54 Product|Tomcat 8|Tomcat 7 Target Milestone||--- OS||All --- Comment #1 from Konstantin Kolinko --- Fixed in trunk by r1610562 and r1610564. It will be in 8.0.11 onwards. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 53281] Tomcat returns garbage data with HTTP/0.9 200 OK header when SSL port is accessed using http
https://issues.apache.org/bugzilla/show_bug.cgi?id=53281 --- Comment #18 from xudong.liu --- I meet the same issue in my test. I want to know if this issue can be fixed? Why it's status is marked invalid? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
