check out the videos on this website TROO.TUBE don't be such a sheep/zombie/loser/NPC. Much love! https://troo.tube/videos/watch/aaa64864-52ee-4201-922f-41300032f219
On Tue, May 5, 2020 at 3:33 AM Johannes Siegert <johannes.sieg...@offerista.com> wrote: > > Hi, > > We did further tests to see where the problem exactly is. These are our > outcomes: > > The content-length is calculated correctly, a quick test with curl showed > this. > The problem is that the stream with the gzip data is not fully consumed and > afterwards not closed. > > Using the debugger with a breakpoint at > org/apache/solr/common/util/Utils.java:575 shows that it won't enter the > function readFully((entity.getContent()) most likely due to how the gzip > stream content is wrapped and extracted beforehand. > > On line org/apache/solr/common/util/Utils.java:582 the > consumeQuietly(entity) should close the stream but does not because of a > silent exception. > > This seems to be the same as it is described in > https://issues.apache.org/jira/browse/SOLR-14457 > > We saw that the problem happened also with correct GZIP responses from > jetty. Not only with non-GZIP as described within the jira issue. > > Best, > > Johannes > > Am Do., 23. Apr. 2020 um 09:55 Uhr schrieb Johannes Siegert < > johannes.sieg...@offerista.com>: > > > Hi, > > > > we want to use gzip-compression between our application and the solr > > server. > > > > We use a standalone solr server version 8.4.1 and the prepackaged jetty as > > application server. > > > > We have enabled the jetty gzip module by adding these two files: > > > > {path_to_solr}/server/modules/gzip.mod (see below the question) > > {path_to_solr}/server/etc/jetty-gzip.xml (see below the question) > > > > Within the application we use a HttpSolrServer that is configured with > > allowCompression=true. > > > > After we had released our application we saw that the number of > > connections within the TCP-state CLOSE_WAIT rising up until the application > > was not able to open new connections. > > > > > > After a long debugging session we think the problem is that the header > > "Content-Length" that is returned by the jetty is sometimes wrong when > > gzip-compression is enabled. > > > > The solrj client uses a ContentLengthInputStream, that uses the header > > "Content-Lenght" to detect if all data was received. But the InputStream > > can not be fully consumed because the value of the header "Content-Lenght" > > is higher than the actual content-length. > > > > Usually the method PoolingHttpClientConnectionManager.releaseConnection is > > called after the InputStream was fully consumed. This give the connection > > free to be reused or to be closed by the application. > > > > Due to the incorrect header "Content-Length" the > > PoolingHttpClientConnectionManager.releaseConnection method is never called > > and the connection stays active. After the connection-timeout of the jetty > > is reached, it closes the connection from the server-side and the TCP-state > > switches into CLOSE_WAIT. The client never closes the connection and so the > > number of connections in use rises up. > > > > > > Currently we try to configure the jetty gzip module to return no > > "Content-Length" if gzip-compression was used. We hope that in this case > > another InputStream implementation is used that uses the NULL-terminator to > > see when the InputStream was fully consumed. > > > > Do you have any experiences with this problem or any suggestions for us? > > > > Thanks, > > > > Johannes > > > > > > gzip.mod > > > > ----- > > > > DO NOT EDIT - See: > > https://www.eclipse.org/jetty/documentation/current/startup-modules.html > > > > [description] > > Enable GzipHandler for dynamic gzip compression > > for the entire server. > > > > [tags] > > handler > > > > [depend] > > server > > > > [xml] > > etc/jetty-gzip.xml > > > > [ini-template] > > ## Minimum content length after which gzip is enabled > > jetty.gzip.minGzipSize=2048 > > > > ## Check whether a file with *.gz extension exists > > jetty.gzip.checkGzExists=false > > > > ## Gzip compression level (-1 for default) > > jetty.gzip.compressionLevel=-1 > > > > ## User agents for which gzip is disabled > > jetty.gzip.excludedUserAgent=.*MSIE.6\.0.* > > > > ----- > > > > jetty-gzip.xml > > > > ----- > > > > <?xml version="1.0"?> > > <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" " > > http://www.eclipse.org/jetty/configure_9_3.dtd";> > > > > <!-- =============================================================== --> > > <!-- Mixin the GZIP Handler --> > > <!-- This applies the GZIP Handler to the entire server --> > > <!-- If a GZIP handler is required for an individual context, then --> > > <!-- use a context XML (see test.xml example in distribution) --> > > <!-- =============================================================== --> > > > > <Configure id="Server" class="org.eclipse.jetty.server.Server"> > > <Call name="insertHandler"> > > <Arg> > > <New id="GzipHandler" > > class="org.eclipse.jetty.server.handler.gzip.GzipHandler"> > > <Set name="minGzipSize"> > > <Property name="jetty.gzip.minGzipSize" > > deprecated="gzip.minGzipSize" default="2048" /> > > </Set> > > <Set name="checkGzExists"> > > <Property name="jetty.gzip.checkGzExists" > > deprecated="gzip.checkGzExists" default="false" /> > > </Set> > > <Set name="compressionLevel"> > > <Property name="jetty.gzip.compressionLevel" > > deprecated="gzip.compressionLevel" default="-1" /> > > </Set> > > <Set name="inflateBufferSize"> > > <Property name="jetty.gzip.inflateBufferSize" > > default="0" /> > > </Set> > > <Set name="deflaterPoolCapacity"> > > <Property name="jetty.gzip.deflaterPoolCapacity" > > default="-1" /> > > </Set> > > <Set name="syncFlush"> > > <Property name="jetty.gzip.syncFlush" default="false" > > /> > > </Set> > > > > <Set name="excludedAgentPatterns"> > > <Array type="String"> > > <Item> > > <Property name="jetty.gzip.excludedUserAgent" > > deprecated="gzip.excludedUserAgent" default=".*MSIE.6\.0.*" /> > > </Item> > > </Array> > > </Set> > > > > <Set name="includedMethodList"> > > <Property name="jetty.gzip.includedMethodList" > > default="GET,POST" /> > > </Set> > > <Set name="excludedMethodList"> > > <Property name="jetty.gzip.excludedMethodList" > > default="" /> > > </Set> > > </New> > > </Arg> > > </Call> > > </Configure> > > > > ----- > >
