[ https://issues.apache.org/jira/browse/SOLR-14896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris M. Hostetter updated SOLR-14896: -------------------------------------- Description: I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like... {noformat} HTTP/1.1 400 Illegal character VCHAR='=' Content-Type: text/html;charset=iso-8859-1 Content-Length: 70 Connection: close <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre> {noformat} ...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header *NAMES* (VCHAR is a legal token type in Header _VALUES_) ---- These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests – _even when the client is not using SolrJ_ – suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj – this seems like a jetty bug. I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to [jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka [jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. -BUT ... that issue suggests that the buffer re-use situation *ONLY* arises due to a prior "500: Response header too large" error – but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs.- was: I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like... {noformat} HTTP/1.1 400 Illegal character VCHAR='=' Content-Type: text/html;charset=iso-8859-1 Content-Length: 70 Connection: close <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre> {noformat} ...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header *NAMES* (VCHAR is a legal token type in Header _VALUES_) ---- These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests -- _even when the client is not using SolrJ_ -- suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj -- this seems like a jetty bug. I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to [jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka [jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. BUT ... that issue suggests that the buffer re-use situation *ONLY* arises due to a prior "500: Response header too large" error -- but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs. I've revised the description of this jira to strike out my concerns that this may not be directly caused by [jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] – i was missinformed, and have confirmed several situations of "500: Response header too large" errors in the solr log files at some point prior to these "400 Bad Message" errors being returned by the same node. > jetty "Bad Message 400" / "Illegal character" responses to sporadic requests > ---------------------------------------------------------------------------- > > Key: SOLR-14896 > URL: https://issues.apache.org/jira/browse/SOLR-14896 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Chris M. Hostetter > Priority: Major > > I'm getting some offline reports from users who recently upgraded to solr 8.6 > of non-reliably reproducing errors from Jetty when doing the HTTP header > parsing. these errors result in a jetty response that will typically look > something like... > {noformat} > HTTP/1.1 400 Illegal character VCHAR='=' > Content-Type: text/html;charset=iso-8859-1 > Content-Length: 70 > Connection: close > <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre> > {noformat} > ...the exact "Illegal character" can vary. Based on some review of the > solr/jetty code bases these errors come from Jetty's HttpParser class when > unexpected character types are encountered while parsing the HTTP METHOD or > Header *NAMES* (VCHAR is a legal token type in Header _VALUES_) > ---- > These errors sometimes manifest in log files as RemoteSolrExceptions due to > intra-node communication, but I've been told they can also be returned > directly by solr/jetty when clients make requests – _even when the client is > not using SolrJ_ – suggesting that whatever is causing the "malformed" > request is not specifically a bug in solr/solrj – this seems like a jetty bug. > I'm suspect that the underlying cause is jetty buffers being re-used/shared > between multiple requests, and suspicious that this is caused by / related to > [jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka > [jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. > -BUT ... that issue suggests that the buffer re-use situation *ONLY* arises > due to a prior "500: Response header too large" error – but i have not been > able to confirm that that situation has ever existed on any of the affected > servers where this "400 Illegal character" error occurs.- -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org