[ 
https://issues.apache.org/jira/browse/SOLR-14896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris M. Hostetter updated SOLR-14896:
--------------------------------------
    Description: 
I'm getting some offline reports from users who recently upgraded to solr 8.6 
of non-reliably reproducing errors from Jetty when doing the HTTP header 
parsing. these errors result in a jetty response that will typically look 
something like...
{noformat}
HTTP/1.1 400 Illegal character VCHAR='='
Content-Type: text/html;charset=iso-8859-1
Content-Length: 70
Connection: close

<h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>
{noformat}
...the exact "Illegal character" can vary. Based on some review of the 
solr/jetty code bases these errors come from Jetty's HttpParser class when 
unexpected character types are encountered while parsing the HTTP METHOD or 
Header *NAMES* (VCHAR is a legal token type in Header _VALUES_)
----
These errors sometimes manifest in log files as RemoteSolrExceptions due to 
intra-node communication, but I've been told they can also be returned directly 
by solr/jetty when clients make requests – _even when the client is not using 
SolrJ_ – suggesting that whatever is causing the "malformed" request is not 
specifically a bug in solr/solrj – this seems like a jetty bug.

I'm suspect that the underlying cause is jetty buffers being re-used/shared 
between multiple requests, and suspicious that this is caused by / related to 
[jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka 
[jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. -BUT 
... that issue suggests that the buffer re-use situation *ONLY* arises due to a 
prior "500: Response header too large" error – but i have not been able to 
confirm that that situation has ever existed on any of the affected servers 
where this "400 Illegal character" error occurs.-

  was:
I'm getting some offline reports from users who recently upgraded to solr 8.6  
of non-reliably reproducing errors from Jetty when doing the HTTP header 
parsing.  these errors result in a jetty response that will typically look 
something like...

{noformat}
HTTP/1.1 400 Illegal character VCHAR='='
Content-Type: text/html;charset=iso-8859-1
Content-Length: 70
Connection: close

<h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>
{noformat}

...the exact "Illegal character" can vary.  Based on some review of the 
solr/jetty code bases these errors come from Jetty's HttpParser class when 
unexpected character types are encountered while parsing the HTTP METHOD or 
Header *NAMES* (VCHAR is a legal token type in Header _VALUES_)

----

These errors sometimes manifest in log files as RemoteSolrExceptions due to 
intra-node communication, but I've been told they can also be returned directly 
by solr/jetty when clients make requests -- _even when the client is not using 
SolrJ_ -- suggesting that whatever is causing the "malformed" request is not 
specifically a bug in solr/solrj -- this seems like a jetty bug.

I'm suspect that the underlying cause is jetty buffers being re-used/shared 
between multiple requests, and suspicious that this is caused by / related to  
[jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka 
[jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. BUT 
... that issue suggests that the buffer re-use situation *ONLY* arises due to a 
prior "500: Response header too large" error -- but i have not been able to 
confirm that that situation has ever existed on any of the affected servers 
where this "400 Illegal character" error occurs.



I've revised the description of this jira to strike out my concerns that this 
may not be directly caused by [jetty 
issue#4936|https://github.com/eclipse/jetty.project/issues/4936] – i was 
missinformed, and have confirmed several situations of "500: Response header 
too large" errors in the solr log files at some point prior to these "400 Bad 
Message" errors being returned by the same node.

> jetty "Bad Message 400" / "Illegal character" responses to sporadic requests
> ----------------------------------------------------------------------------
>
>                 Key: SOLR-14896
>                 URL: https://issues.apache.org/jira/browse/SOLR-14896
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Chris M. Hostetter
>            Priority: Major
>
> I'm getting some offline reports from users who recently upgraded to solr 8.6 
> of non-reliably reproducing errors from Jetty when doing the HTTP header 
> parsing. these errors result in a jetty response that will typically look 
> something like...
> {noformat}
> HTTP/1.1 400 Illegal character VCHAR='='
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 70
> Connection: close
> <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>
> {noformat}
> ...the exact "Illegal character" can vary. Based on some review of the 
> solr/jetty code bases these errors come from Jetty's HttpParser class when 
> unexpected character types are encountered while parsing the HTTP METHOD or 
> Header *NAMES* (VCHAR is a legal token type in Header _VALUES_)
> ----
> These errors sometimes manifest in log files as RemoteSolrExceptions due to 
> intra-node communication, but I've been told they can also be returned 
> directly by solr/jetty when clients make requests – _even when the client is 
> not using SolrJ_ – suggesting that whatever is causing the "malformed" 
> request is not specifically a bug in solr/solrj – this seems like a jetty bug.
> I'm suspect that the underlying cause is jetty buffers being re-used/shared 
> between multiple requests, and suspicious that this is caused by / related to 
> [jetty issue#4936|https://github.com/eclipse/jetty.project/issues/4936] / aka 
> [jetty bug#564984|https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984] .. 
> -BUT ... that issue suggests that the buffer re-use situation *ONLY* arises 
> due to a prior "500: Response header too large" error – but i have not been 
> able to confirm that that situation has ever existed on any of the affected 
> servers where this "400 Illegal character" error occurs.-



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to