[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448
--- Comment #6 from Roman ---
I will hopefully be able to answer your questions later today. This is
happening on a customer site and working remotely is quite...painful.
What I can answer at the moment:
1, 3, 4, 5. Will have to check later.
2. The problem occurs with no obvious pattern.
Yesterday, I was able to reproduce it almost always. Today, when I retried, it
took 5-10 tries to reproduce it. The Windows env was not rebooted, but I
restarted Tomcat to add some debug information myself.
6. Yes, this is the main essence of the code (I stripped away all things not
related to the actual problem):
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class ServletForAsf extends HttpServlet {
private static final long serialVersionUID = 3266630280219605924L;
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
String resourcePath = request.getPathInfo();
File resourceFile = new File(resourcePath);
ServletOutputStream outputStream = response.getOutputStream();
int bufferSize = 1 << 16;
try (InputStream in = new BufferedInputStream(new
FileInputStream(resourceFile), bufferSize)) {
final byte[] buffer = new byte[bufferSize];
int count;
long totalCount = 0;
while ((count = in.read(buffer)) != -1) {
try {
outputStream.write(buffer, 0, count);
} catch (Exception e) {
throw new IOException("Error while
transfering data. Data transferred so far: " + totalCount + ". Current buffer
size: " + count, e);
}
totalCount += count;
}
outputStream.flush();
}
}
}
7. Yes, testing a custom Tomcat would be possible. Fortunately, the environment
where this is happening is a test env and the customer is rather friendly.
In the meantime, the comment about the DefaultServlet made me think (and
debug).
It seems that the DefaultServlet does not take the same route as our Servlet
does.
When the DefaultServlet serves a file, it is not done within the Servlet
directly (as far as I can tell) but rather in
Http11Processor.processSendfile(SocketWrapperBase) line: 1359
The main thing is that this eventually calls
sd.fchannel.close();
When I follow the path of our servlet, I was not able to actually find an
actual flush() call on the underlying socket.
So, my current thinking is that the data is not flushed to the client
successfully and that is why it needs to wait for the connection timeout.
--
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #7 from Roman --- Update on question 1: Yes, I can reproduce the problem with Tomcat 9.0.48. To downgrade the server, I replaced * lib/* * bin/bootstrap.jar * bin/commons-daemon.jar * bin/tomcat9*.exe * bin/tcnative-1.dll -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #8 from Roman --- And here are the answers for the other questions: 3. APR disabled (org.apache.catalina.core.AprLifecycleListener commented out) Threads: https-jsse-nio-8443-exec-* Not reproducible 4. APR enabled, NIO2 (replaced org.apache.coyote.http11.Http11NioProtocol by org.apache.coyote.http11.Http11Nio2Protocol) Threads: https-openssl-nio2-8443-exec-* Not reproducible 5. APR disabled, NIO2 (org.apache.catalina.core.AprLifecycleListener commented out, replaced org.apache.coyote.http11.Http11NioProtocol by org.apache.coyote.http11.Http11Nio2Protocol) Threads: https-jsse-nio2-8443-exec-* Not reproducible So, that basically means that this only happens with NIO and APR. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] Pingvin235 opened a new pull request #434: Optionally inherit request parameters
Pingvin235 opened a new pull request #434: URL: https://github.com/apache/tomcat/pull/434 Greetings to All, the proposed fix makes optional taking requests parameters for internal requests. Which produced when using JSTL directives `
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #9 from Mark Thomas --- Thanks for all the information. That you still see the issue with 9.0.48 means that the response buffer flushing changes in 9.0.49 are not the root cause of this issue. That the observation of the issue is limited to NIO+OpenSSL and not (NIO+JSSE, NIO2+OpenSSL or NIO2+JSSE) suggests where the root cause may lie but it is possible that NIO+OpenSSL just happens to have the right timing to trigger an issue that could affect other combinations. Similarly, that using sendfile appears to avoids the issue suggests where the root cause may lie. I will set up a Windows 2019 VM to see if I can reproduce this. Meanwhile, could you test with 9.0.46? There were some NIO changes between 9.0.46 and 9.0.48 it would be helpful to rule in/out. And another question. Can you recreate this with debug logging enabled? Specifically with the following in logging.properties: org.apache.coyote.level = FINE org.apache.tomcat.util.net.level = FINE If yes, please provide a copy of the debug logs for when the issue occurred. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] markt-asf commented on pull request #434: Optionally inherit request parameters
markt-asf commented on pull request #434: URL: https://github.com/apache/tomcat/pull/434#issuecomment-879789821 Such an option would be in direct violation of the Servlet specification. That makes me very hesitant to agree to such a change. The Servlet specification already states that the parameters from the include take precedence. I'd rather add clarification of any ambiguities (if necessary) to the spec and align Tomcat's behaviour with that than add an option for non-compliant behaviour. Regarding the implementation, I am -1 on the proposed implementation as it is based on an environment variable. If it were implemented, It would need to be a per context configuration attribute. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #10 from Roman --- I tested with 9.0.46 and was not able to reproduce the problem. (Thread names: http-nio-8080-exec-*, APR on, NIO) I will attach the log file containing the requested FINE output. The file only contains log lines that have not been produced by our software. If you need the full log, please let me know. To reproduce the problem, I have to constantly run the cURL command. So there might be remanants from the previous run in the log, but the last entries are the ones that were written when the download was blocked. The system that I am testing on *might* be used by the customer, so there is a slight chance that there is traffic in the log that is not related to my download, but I hope this was not case here. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #11 from Roman --- Created attachment 37953 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37953&action=edit Tomcat FINE log while the downloading was blocked -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #12 from Roman --- For the latest tests, I have reduced the connection timeout to 30s (which might be good to know when reading the log file). -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #13 from Mark Thomas --- (In reply to Roman from comment #10) > I tested with 9.0.46 and was not able to reproduce the problem. > (Thread names: http-nio-8080-exec-*, APR on, NIO) That is an HTTP connector. ARe you sure you tested with HTTPS? > I will attach the log file containing the requested FINE output. The file > only contains log lines that have not been produced by our software. If you > need the full log, please let me know. > > To reproduce the problem, I have to constantly run the cURL command. So > there might be remanants from the previous run in the log, but the last > entries are the ones that were written when the download was blocked. > > The system that I am testing on *might* be used by the customer, so there is > a slight chance that there is traffic in the log that is not related to my > download, but I hope this was not case here. Thanks. Still Getting my test system set up. I'll look at the logs after that. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #14 from Roman --- > That is an HTTP connector. ARe you sure you tested with HTTPS? Yes, I just took the first exec thread name that I found in the threaddump. Sorry for the confusion. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #15 from Mark Thomas --- Debug log aren't very helpful. If I am unable to recreate this, we may need to start down the custom Tomcat build route. Which version of Java are you using? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #16 from Mark Thomas --- I haven't been able to recreate this issue locally. Also, the debug logs didn't have much information on writes - which is where the problem appears to be. Therefore, I think it is time to move to the custom Tomcat build. The source code for the first build (and any future builds) can be found at: https://github.com/markt-asf/tomcat/tree/bz-65448 The binaries can be found at: https://people.apache.org/~markt/dev/ Please can you retest with v9.0.51-bz-65448-v1 and post the resulting logs. The new logs are INFO level logs. Please aim to retain these log messages in their entirety. The timestamps, thread names and messages are equally important. v1 adds INFO level logging at the point where plaintext data is encrypted and written to the network. Depending on what the logs look like, v2 will add additional logging either further up or lower down the call stack to track down the root cause. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] Pingvin235 commented on pull request #434: Optionally inherit request parameters
Pingvin235 commented on pull request #434: URL: https://github.com/apache/tomcat/pull/434#issuecomment-880041227 Could you please point me the place in the specification, describing that behavior? For 4.0 could found only section 9.3 regarding includes, but didn't notice anything about parameters inheritance / precedence.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] Pingvin235 commented on pull request #434: Optionally inherit request parameters
Pingvin235 commented on pull request #434: URL: https://github.com/apache/tomcat/pull/434#issuecomment-880042757 > Regarding the implementation, I am -1 on the proposed implementation as it is based on an environment variable. If it were implemented, It would need to be a per context configuration attribute. Can do it this way of course case this option is in principle allowed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] markt-asf commented on pull request #434: Optionally inherit request parameters
markt-asf commented on pull request #434: URL: https://github.com/apache/tomcat/pull/434#issuecomment-880066207 You want the last paragraph of section 9.1.1. A RequestDispatcher is what performs the include or forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #17 from Roman --- Ok, I will do this tomorrow. Thanks for your help! In the meantime, this is the Java version we use: Specification: Java Virtual Machine Specification 11 (Oracle Corporation) Virtual Machine: OpenJDK 64-Bit Server VM 11.0.9.1+1 (AdoptOpenJDK) Java Vendor: AdoptOpenJDK Java Version: 11.0.9.1 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #18 from Roman --- One more thing: I did a threaddump of the server while the download was blocking and did not see any Thread that was out of the ordinary. So, I would assume that the writing to the socket may have already finished, but not yet committed (i.e., flushed). Hence maybe the waiting time for the socket timeout when suddenly the client receives the remainder of the response. This is also fit with the statistics that show that the maximum request processing time was just a few seconds, in contrast to the actual waiting time of 30 seconds (or more, depending on the timeout config). -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[GitHub] [tomcat] Pingvin235 commented on pull request #434: Optionally inherit request parameters
Pingvin235 commented on pull request #434: URL: https://github.com/apache/tomcat/pull/434#issuecomment-880204596 Thanks for pointing. Assuming this is the conflicting part: "Parameters specified in the query string used to create the RequestDispatcher take precedence over other parameters of the same name passed to the included servlet." Perhaps I understand it wrong, but if other parameters are completely excluded, do query string parameters loose their precedence? Actually that we need is to have the query string parameters only. May be you can recommend another way to archive such behavior? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #19 from Roman --- Created attachment 37955 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37955&action=edit Logs of v1, with screenshot of cURL/Log-Tail when it blocked -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 65448] Download of file via Servlet OutputStream blocks when using SSL
https://bz.apache.org/bugzilla/show_bug.cgi?id=65448 --- Comment #20 from Roman --- I have attached the full logs of the test run with v1. It took quite some tries to actually reproduce it, but I made a screenshot at the time it happened so that you can correlate the blocking with timestamps in the log. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
