[Bug 65154] New: java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154
Bug ID: 65154
Summary: java.io.IOException: Unexpected error [104] writing
data to the APR/native socket
Product: Tomcat 9
Version: 9.0.43
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: sl...@aceslash.net
Target Milestone: -
Created attachment 37742
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37742&action=edit
java.io.IOException stack example
Hello,
Investigating a download issue on several copies of our application (the
application is deployed on hundreds of tomcats, each application is
independent, no clustering, just different client), we found out that we have
thousands of errors such as the one attached.
We are using tomcat 9.0.41 and 9.0.43 but I think this issue is much older than
that (we only keep 3 months of logs). The JDK version used is either OpenJDK 8
or 11 from AdoptOpenJDK (the JDK version doesn't change anything).
The installation of tomcat is from the official website and we are compiling
APR automatically following the documentation on each tomcat release.
The connector configuration is:
The service is behind an haproxy which handle TLS.
Java parameters varies for memory but other than that, this is a typical
setenv.sh:
#!/bin/sh
# Ansible managed
export JAVA_HOME="/srv/resources/jvm/jdk-11.0.10+9"
export CATALINA_HOME="/applications/tomcat/engine"
export compilation="-XX:-TieredCompilation"
export
logs="-Xlog:gc*:file=${CATALINA_HOME}/logs/gc.log::filecount=10,filesize=20m:time,tid,level,tags"
export timeouts="-Dsun.net.client.defaultConnectTimeout=30
-Dsun.net.client.defaultReadTimeout=720"
export
FlightRecording="-XX:StartFlightRecording=disk=true,dumponexit=true,filename=${CATALINA_HOME}/logs"
export JAVA_OPTS="-Djava.awt.headless=true
-Dcom.sun.security.enableAIAcaIssuers=true -XX:+UseG1GC
-XX:MaxGCPauseMillis=800 -XX:+UseStringDeduplication
-Djava.net.preferIPv4Stack=true -Xmx64g -XX:ReservedCodeCacheSize=2g
-XX:MetaspaceSize=10g -XX:G1HeapRegionSize=32M ${compilation} ${logs}
${timeouts} ${FlightRecording}"
# JMX monitoring
export CATALINA_OPTS="-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=12345
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/applications/tomcat/settings/jmxremote/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/applications/tomcat/settings/jmxremote/jmxremote.access
-Djava.rmi.server.hostname=REDACTED"
# Activate Apache Portable Runtime (APR) based Native library for Tomcat
export LD_LIBRARY_PATH=/usr/local/apr/lib
Changing the connector from "org.apache.coyote.http11.Http11AprProtocol" to
"org.apache.coyote.http11.Http11Nio2Protocol" completely eliminate the issue,
we see 0 error. Reproducing the issue is complicated and we have no automatic
way to trigger this yet.
The issue https://bz.apache.org/bugzilla/show_bug.cgi?id=62626 looks similar
and we are indeed using imageio (when we use JDK 8, we are using
jai-imageio-core-1.4.0, when we use JDK 11, we use the included imageio).
I'm sorry but I may not be able to help you further since were are switching
everthing to nio2 now, but I still wanted to raise the issue so that you know
that such problem still exist.
--
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65131] OpenSSLEngine errors on a connection affect other connections
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131 --- Comment #4 from Mark Thomas --- I can't recreate this using the provided test case, Tomcat Native 1.2.26 and the latest OpenSSL 1.1.1 source code. I'll note that Tomcat 1.2.26 won't compile with OpenSSL 1.0.2 so I do wonder exactly what versions were in used in the tests described here. I'll also note that OpenSSL 1.0.2 is no longer supported. Please retest with the latest OpenSSL 1.1.1 release. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65131] OpenSSLEngine errors on a connection affect other connections
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #5 from Remy Maucherat ---
(In reply to Mark Thomas from comment #4)
> I can't recreate this using the provided test case, Tomcat Native 1.2.26 and
> the latest OpenSSL 1.1.1 source code.
>
> I'll note that Tomcat 1.2.26 won't compile with OpenSSL 1.0.2 so I do wonder
> exactly what versions were in used in the tests described here.
>
> I'll also note that OpenSSL 1.0.2 is no longer supported.
>
> Please retest with the latest OpenSSL 1.1.1 release.
Should I still replace clearLastError with this ?
private static void clearLastError() {
while (SSL.getLastErrorNumber() != SSL.SSL_ERROR_NONE);
}
This does not seem unreasonable given the docs ...
--
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65154] java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154 Mark Thomas changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #1 from Mark Thomas --- Error code 104 is "Connection reset by peer". Therefore, this is not a Tomcat issue. I recommend NIO rather than NIO2 unless performance testing has indicated a clear benefit of using NIO2. Further assistance, if required, can be obtained from the users mailing list. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65118] Spurious null pointer exception in Http2UpgradeHandler when reloading web page
https://bz.apache.org/bugzilla/show_bug.cgi?id=65118 Mark Thomas changed: What|Removed |Added Status|REOPENED|NEEDINFO --- Comment #6 from Mark Thomas --- Moving to NEEDINFO as we need a test case for this. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65154] java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154 sl...@aceslash.net changed: What|Removed |Added Resolution|INVALID |FIXED --- Comment #2 from sl...@aceslash.net --- We also have "Unexpected error [32]". And how do you explain that the switch from APR does eliminate the issue? Currently I have 2 tomcats with the exact same configuration except one is using APR and the other one is using NIO2 and I see the errors only on the first one. Also the tomcat that did have issues last week with APR doesn't have any issue today with NIO2. Anyway, thank you for pointing us to NIO, we'll check it out. Of course, you can disregard this report, we'll keep future issues to our self if they don't interest you. I thought that it would be interesting to know that such problem existed, but it seems you prefer to consider that it's simply a "Connection reset by peer", OK. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65154] java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154
--- Comment #3 from sl...@aceslash.net ---
This is the result on one application:
# for n in {01..22}; do echo -n "log.2021-02-${n}_09.log "; grep "Unexpected
error" log.2021-02-${n}_09.log | wc -l; done
log.2021-02-01_09.log 6144
log.2021-02-02_09.log 5469
log.2021-02-03_09.log 5476
log.2021-02-04_09.log 4653
log.2021-02-05_09.log 5471
log.2021-02-06_09.log 4348
log.2021-02-07_09.log 0 (Sunday, almost 0 activity)
log.2021-02-08_09.log 2997
log.2021-02-09_09.log 4523
log.2021-02-10_09.log 4709
log.2021-02-11_09.log 4291
log.2021-02-12_09.log 5725
log.2021-02-13_09.log 3398
log.2021-02-14_09.log 7 (Sunday, almost 0 activity)
log.2021-02-15_09.log 4216
log.2021-02-16_09.log 4078
log.2021-02-17_09.log 4064
log.2021-02-18_09.log 4112
(switch from APR to NIO2)
log.2021-02-19_09.log 0
log.2021-02-20_09.log 0
log.2021-02-21_09.log 0
log.2021-02-22_09.log 0
Of course, you can say that switching to NIO2 did nothing, but I think that it
did.
--
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65154] java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154 Mark Thomas changed: What|Removed |Added Resolution|FIXED |INVALID --- Comment #4 from Mark Thomas --- Error code 104 is a client disconnect. That is never a Tomcat bug. Error code 32 is a broken pipe. That error code was never mentioned in the original bug report. The Tomcat Native/APR code does not use pipes as far as I can tell although it is possible I am missing something under the covers in either the APR library or OpenSSL. If you have a test case that reproduces the 32 error on a clean install of the latest Tomcat 9 and Native Connector then please open a new bug report and provide the steps to reproduce the error and any additional information in that new bug report. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65118] Spurious null pointer exception in Http2UpgradeHandler when reloading web page
https://bz.apache.org/bugzilla/show_bug.cgi?id=65118 Mark Thomas changed: What|Removed |Added Status|NEEDINFO|NEW --- Comment #7 from Mark Thomas --- Woot! I managed to trigger this locally just with Tomcat's default home page. Investigating... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65154] java.io.IOException: Unexpected error [104] writing data to the APR/native socket
https://bz.apache.org/bugzilla/show_bug.cgi?id=65154 --- Comment #5 from Remy Maucherat --- If we're comparing the connectors as of right now, I would say I prefer NIO2 over NIO on Java 8, since Oracle was not fixing or improving NIO then. Now with the next LTS coming up, I would say NIO is now better since it has fixes/refactorings and new features [NIO2 does not have the new features]. On clients I would say NIO2 is always better as long as the new features are not needed. So I would have, if things don't change: Java 8: NIO2 > NIO Java 11: tie Java 17: NIO > NIO 2 This is not making things better overall ... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] branch master updated (b9b3fa7 -> ffe30d9)
This is an automated email from the ASF dual-hosted git repository. markt pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/tomcat.git. from b9b3fa7 Fix bz 65137 Don't corrupt response on early termination new 23f8eb5 Add missing parameter in log message. Fix comment typo. new ffe30d9 Additional fix for BZ 65118 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +- java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 02/02: Additional fix for BZ 65118
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit ffe30d91cd90c5c02bc15ec8b15cff43f18557f5
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:32:55 2021 +
Additional fix for BZ 65118
When replacing a closed Stream with a RecycledStream instance, the
priority hierarchy was not maintained. This led to NPEs during the
pruning process.
---
java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
index bbdfa33..38761ad 100644
--- a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
+++ b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
@@ -122,7 +122,7 @@ abstract class AbstractNonZeroStream extends AbstractStream
{
* level. This is the caller's responsibility.
*/
void replaceStream(AbstractNonZeroStream replacement) {
-replacement.setParentStream(getParentStream());
+getParentStream().addChild(replacement);
detachFromParent();
for (AbstractNonZeroStream child : getChildStreams()) {
replacement.addChild(child);
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 01/02: Add missing parameter in log message. Fix comment typo.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 23f8eb5c42c6e4bd3c877a95083020912ee67eb1
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:30:29 2021 +
Add missing parameter in log message. Fix comment typo.
---
java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
index 0903f04..eaf7e88 100644
--- a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
+++ b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
@@ -747,7 +747,7 @@ class Http2UpgradeHandler extends AbstractStream implements
InternalHttpUpgradeH
void writeBody(Stream stream, ByteBuffer data, int len, boolean finished)
throws IOException {
if (log.isDebugEnabled()) {
log.debug(sm.getString("upgradeHandler.writeBody", connectionId,
stream.getIdAsString(),
-Integer.toString(len)));
+Integer.toString(len), Boolean.valueOf(finished)));
}
reduceOverheadCount();
@@ -1251,7 +1251,7 @@ class Http2UpgradeHandler extends AbstractStream
implements InternalHttpUpgradeH
// work through the ordered list of streams.
while (toClose > 0 && parent.getIdAsInt() > 0 &&
parent.getIdAsInt() < stream.getIdAsInt() &&
parent.getChildStreams().isEmpty()) {
-// This case is safe since we know parent ID > 0
therefore
+// This cast is safe since we know parent ID > 0
therefore
// this isn't the connection
stream = (AbstractNonZeroStream) parent;
parent = stream.getParentStream();
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] branch 9.0.x updated (e0da301 -> 5c9c7eb)
This is an automated email from the ASF dual-hosted git repository. markt pushed a change to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/tomcat.git. from e0da301 Align code with 8.5.x new 8636627 Add missing parameter in log message. Fix comment typo. new 5c9c7eb Additional fix for BZ 65118 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +- java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 01/02: Add missing parameter in log message. Fix comment typo.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 863662778f61598ba9185e6b74006e3a7c9a3a89
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:30:29 2021 +
Add missing parameter in log message. Fix comment typo.
---
java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
index 6647624..ca8a2bf 100644
--- a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
+++ b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
@@ -747,7 +747,7 @@ class Http2UpgradeHandler extends AbstractStream implements
InternalHttpUpgradeH
void writeBody(Stream stream, ByteBuffer data, int len, boolean finished)
throws IOException {
if (log.isDebugEnabled()) {
log.debug(sm.getString("upgradeHandler.writeBody", connectionId,
stream.getIdAsString(),
-Integer.toString(len)));
+Integer.toString(len), Boolean.valueOf(finished)));
}
reduceOverheadCount();
@@ -1251,7 +1251,7 @@ class Http2UpgradeHandler extends AbstractStream
implements InternalHttpUpgradeH
// work through the ordered list of streams.
while (toClose > 0 && parent.getIdAsInt() > 0 &&
parent.getIdAsInt() < stream.getIdAsInt() &&
parent.getChildStreams().isEmpty()) {
-// This case is safe since we know parent ID > 0
therefore
+// This cast is safe since we know parent ID > 0
therefore
// this isn't the connection
stream = (AbstractNonZeroStream) parent;
parent = stream.getParentStream();
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 02/02: Additional fix for BZ 65118
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 5c9c7eb1d86e0c6d98aa28ecab98304c6cad2bde
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:32:55 2021 +
Additional fix for BZ 65118
When replacing a closed Stream with a RecycledStream instance, the
priority hierarchy was not maintained. This led to NPEs during the
pruning process.
---
java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
index bbdfa33..38761ad 100644
--- a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
+++ b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
@@ -122,7 +122,7 @@ abstract class AbstractNonZeroStream extends AbstractStream
{
* level. This is the caller's responsibility.
*/
void replaceStream(AbstractNonZeroStream replacement) {
-replacement.setParentStream(getParentStream());
+getParentStream().addChild(replacement);
detachFromParent();
for (AbstractNonZeroStream child : getChildStreams()) {
replacement.addChild(child);
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] branch 8.5.x updated (af7b667 -> 0a3fd6b)
This is an automated email from the ASF dual-hosted git repository. markt pushed a change to branch 8.5.x in repository https://gitbox.apache.org/repos/asf/tomcat.git. from af7b667 Fix bz 65137 Don't corrupt response on early termination new 11f1377 Add missing parameter in log message. Fix comment typo. new 0a3fd6b Additional fix for BZ 65118 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +- java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 01/02: Add missing parameter in log message. Fix comment typo.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch 8.5.x
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 11f137718fe8df61fdadc379d5edd6637de44843
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:30:29 2021 +
Add missing parameter in log message. Fix comment typo.
---
java/org/apache/coyote/http2/Http2UpgradeHandler.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
index 9f1cd8a..4ee1104 100644
--- a/java/org/apache/coyote/http2/Http2UpgradeHandler.java
+++ b/java/org/apache/coyote/http2/Http2UpgradeHandler.java
@@ -750,7 +750,7 @@ public class Http2UpgradeHandler extends AbstractStream
implements InternalHttpU
void writeBody(Stream stream, ByteBuffer data, int len, boolean finished)
throws IOException {
if (log.isDebugEnabled()) {
log.debug(sm.getString("upgradeHandler.writeBody", connectionId,
stream.getIdAsString(),
-Integer.toString(len)));
+Integer.toString(len), Boolean.valueOf(finished)));
}
reduceOverheadCount();
@@ -1250,7 +1250,7 @@ public class Http2UpgradeHandler extends AbstractStream
implements InternalHttpU
// work through the ordered list of streams.
while (toClose > 0 && parent.getIdAsInt() > 0 &&
parent.getIdAsInt() < stream.getIdAsInt() &&
parent.getChildStreams().isEmpty()) {
-// This case is safe since we know parent ID > 0
therefore
+// This cast is safe since we know parent ID > 0
therefore
// this isn't the connection
stream = (AbstractNonZeroStream) parent;
parent = stream.getParentStream();
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[tomcat] 02/02: Additional fix for BZ 65118
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch 8.5.x
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 0a3fd6b162e550477b2e44a4aabbb99c0e677e08
Author: Mark Thomas
AuthorDate: Mon Feb 22 16:32:55 2021 +
Additional fix for BZ 65118
When replacing a closed Stream with a RecycledStream instance, the
priority hierarchy was not maintained. This led to NPEs during the
pruning process.
---
java/org/apache/coyote/http2/AbstractNonZeroStream.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
index d85840d..930f131 100644
--- a/java/org/apache/coyote/http2/AbstractNonZeroStream.java
+++ b/java/org/apache/coyote/http2/AbstractNonZeroStream.java
@@ -122,7 +122,7 @@ abstract class AbstractNonZeroStream extends AbstractStream
{
* level. This is the caller's responsibility.
*/
void replaceStream(AbstractNonZeroStream replacement) {
-replacement.setParentStream(getParentStream());
+getParentStream().addChild(replacement);
detachFromParent();
for (AbstractNonZeroStream child : getChildStreams()) {
replacement.addChild(child);
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65118] Spurious null pointer exception in Http2UpgradeHandler when reloading web page
https://bz.apache.org/bugzilla/show_bug.cgi?id=65118 Mark Thomas changed: What|Removed |Added Resolution|--- |FIXED Status|NEW |RESOLVED --- Comment #8 from Mark Thomas --- Found it. The priority hierarchy was being corrupted when a closed stream was replaced with a RecycledStream instance. Fixed in: - 10.0.x for 10.0.3 onwards - 9.0.x for 9.0.44 onwards - 8.5.x for 8.5.63 onwards 7.0.x is not affected. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65131] OpenSSLEngine errors on a connection affect other connections
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131 --- Comment #6 from Mark Thomas --- Yes, the OP is correct that the clearLastError call in unwrap is there is part of this fix: https://github.com/apache/tomcat/commit/ebab25 I agree that using a loop looks to be the right approach here. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
tomcat with epollWait always return keyCount=0, but the read buffer still have data and cause many close_wait
1. Environment OS: CentOS Linux release 7.8.2003 (Core) JDK: java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) Tomcat: tomcat-embed-core-8.5.32 with spring-boot-2.0.4-RELEASE 2. Reproduce Steps 1. request large data download api, then request 2000+ request to another lightweight api 2. the system now is Full GC, and the 2000+ request will blocked, then close all these request 3. after system throw OufOfMemoryError, the memory will be released, the cpu and memory occupation is normal, the system is ok to visit mysql database and execute the schedule job 4. request any api, the response is always slow, and too may close_wait [image: image.png] 3. Problem Shooting a) I check the thread with jstack, ClientPoller in NioEndpoint, BlockPoller in NioBlockingSelector are both in "http-nio-8043-ClientPoller-0" #83 daemon prio=5 os_prio=0 tid=0x7faf58b5 nid=0x4a82 runnable [0x7faefc8d7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xe3c8b478> (a sun.nio.ch.Util$3) - locked <0xe3c8b468> (a java.util.Collections$UnmodifiableSet) - locked <0xe3c8b330> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:798) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None "NioBlockingSelector.BlockPoller-1" #72 daemon prio=5 os_prio=0 tid=0x7faf591fa800 nid=0x4a77 runnable [0x7faefd3e2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xe3c8bb78> (a sun.nio.ch.Util$3) - locked <0xe3c8bb68> (a java.util.Collections$UnmodifiableSet) - locked <0xe3c8ba40> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298) Locked ownable synchronizers: b) arthas check https://github.com/alibaba/arthas i hook the process to see what are the two poller select(selectorTimeout) return, the keyCount = 0, but the read buffer have 191 byte to read, why epollWait always return keyCount = 0? the expected behavior is, tomcat can read the data from buffer and the close the socket successfully c) test with another method I change the protocol to "Http11Nio2Protocol", and the close_wait will be recycled I set java.nio.channels.spi.SelectorProvider with PollSelectorProvider, the close_wait will be recycled too
Re: tomcat with epollWait always return keyCount=0, but the read buffer still have data and cause many close_wait
On Tue, Feb 23, 2021 at 4:07 AM Dan Zheng wrote: > 1. Environment > OS: CentOS Linux release 7.8.2003 (Core) > > JDK: java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > > Tomcat: tomcat-embed-core-8.5.32 with spring-boot-2.0.4-RELEASE > > 2. Reproduce Steps > > 1. request large data download api, then request 2000+ request to > another lightweight api > 2. the system now is Full GC, and the 2000+ request will blocked, then > close all these request > 3. after system throw OufOfMemoryError, the memory will be released, the > cpu and memory occupation is normal, the system is ok to visit mysql > database and execute the schedule job > 4. request any api, the response is always slow, and too may close_wait > [image: image.png] > > 3. Problem Shooting > a) I check the thread with jstack, ClientPoller in NioEndpoint, > BlockPoller in NioBlockingSelector are both in > > "http-nio-8043-ClientPoller-0" #83 daemon prio=5 os_prio=0 > tid=0x7faf58b5 nid=0x4a82 runnable [0x7faefc8d7000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0xe3c8b478> (a sun.nio.ch.Util$3) > - locked <0xe3c8b468> (a > java.util.Collections$UnmodifiableSet) > - locked <0xe3c8b330> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:798) > at java.lang.Thread.run(Thread.java:748) > >Locked ownable synchronizers: > - None > > > "NioBlockingSelector.BlockPoller-1" #72 daemon prio=5 os_prio=0 > tid=0x7faf591fa800 nid=0x4a77 runnable [0x7faefd3e2000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0xe3c8bb78> (a sun.nio.ch.Util$3) > - locked <0xe3c8bb68> (a > java.util.Collections$UnmodifiableSet) > - locked <0xe3c8ba40> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298) > >Locked ownable synchronizers: > > > > b) arthas check https://github.com/alibaba/arthas > i hook the process to see what are the two poller select(selectorTimeout) > return, > > the keyCount = 0, but the read buffer have 191 byte to read, why epollWait > always return keyCount = 0? > > the expected behavior is, tomcat can read the data from buffer and the > close the socket successfully > > c) test with another method > I change the protocol to "Http11Nio2Protocol", and the close_wait will be > recycled > > I set java.nio.channels.spi.SelectorProvider > with PollSelectorProvider, the close_wait will be recycled too > As you found out, this is a JVM bug and there are workarounds if you experience it: https://bz.apache.org/bugzilla/show_bug.cgi?id=63802 Rémy
