Hi, we are having problems with an installation of SolrCloud where a leader node kicks off an indexing and tries to replicate all the updates using the UpdateHandler.
What we get instead is an error around a wrong UTF-8 encoding from the leader trying to call the /udpate endpoint on the replica: request: http://10.40.0.25:9765/skus/update?update.chain=custom&_version_=-1459207589104451584&update.distrib=FROMLEADER&update.from=http%3A%2 F%2F10.40.0.24%3A9765%2Fskus%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) While on the replica we get this: 2014-02-05 14:00:00,226 [qtp-108] INFO org.apache.solr.update.processor.LogUpdateProcessor - [skus] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1459207589104451584&update.from= http://10.40.0.24:9765/skus/&wt=javabin&version=2&update.chain=custom<http://10.40.0.24:9765/gilt-by-sku/&wt=javabin&version=2&update.chain=custom>} {} 0 71 2014-02-05 14:00:00,227 [qtp-108] ERROR org.apache.solr.core.SolrCore - org.apache.solr.common.SolrException: *Invalid UTF-8 middle byte 0xe0 (at cha**r #1, byte #-1)* at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) I have tried to sanitize all my docs making sure all the strings are in UTF-8 but does not work. Attached there is also the HTTP conversation that produces the error. Would love to understand what is going on here :) Thank you, Ugo
2014-02-05 17:20:57,703 [updateExecutor-1-thread-1] DEBUG org.apache.http.impl.client.DefaultHttpClient - Attempt 1 to execute request 2014-02-05 17:20:57,703 [updateExecutor-1-thread-1] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Sending request: POST /skus/update?update.chain=custom&_version_=-1459220232373534720&update.distrib=FROMLEADER&update.from=http%3A%2F%2F172.16.30.1%3A9765%2Fskus%2F&wt=javabin&version=2 HTTP/1.1 2014-02-05 17:20:57,703 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "POST /skus/update?update.chain=custom&_version_=-1459220232373534720&update.distrib=FROMLEADER&update.from=http%3A%2F%2F172.16.30.1%3A9765%2Fskus%2F&wt=javabin&version=2 HTTP/1.1[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "Content-Type: application/javabin[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "Transfer-Encoding: chunked[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "Host: 172.16.30.1:9766[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "Connection: Keep-Alive[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "[\r][\n]" 2014-02-05 17:20:57,704 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> POST /skus/update?update.chain=custom&_version_=-1459220232373534720&update.distrib=FROMLEADER&update.from=http%3A%2F%2F172.16.30.1%3A9765%2Fskus%2F&wt=javabin&version=2 HTTP/1.1 2014-02-05 17:20:57,706 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0 2014-02-05 17:20:57,706 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> Content-Type: application/javabin 2014-02-05 17:20:57,706 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> Transfer-Encoding: chunked 2014-02-05 17:20:57,706 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> Host: 172.16.30.1:9766 2014-02-05 17:20:57,706 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - >> Connection: Keep-Alive 2014-02-05 17:20:57,707 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "a2[\r][\n]" 2014-02-05 17:20:57,707 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "[0x2][0xc3][0xe0]¶ms[0xa4][0xe0],update.chain&custom[0xe0])_version_4-1459220232373534720[0xe0].update.distrib*FROMLEADER[0xe0]+update.from?[0x5]http://172.16.30.1:9765/skus/[0xe0]&delByQ[0x81]#*:*[0xe0]$docs[0x0]"; 2014-02-05 17:20:57,707 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "[\r][\n]" 2014-02-05 17:20:57,708 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "0[\r][\n]" 2014-02-05 17:20:57,708 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - >> "[\r][\n]" 2014-02-05 17:20:57,784 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "HTTP/1.1 400 Bad Request[\r][\n]" 2014-02-05 17:20:57,785 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "X-SERVICE-VERSION: 47538@IRL-ML-DUBLIN.local-UNKNOWN[\r][\n]" 2014-02-05 17:20:57,785 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "X-SERVICE-ID: unknown.47538.IRL-ML-DUBLIN[\r][\n]" 2014-02-05 17:20:57,785 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "Content-Type: application/octet-stream[\r][\n]" 2014-02-05 17:20:57,785 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "Content-Length: 114[\r][\n]" 2014-02-05 17:20:57,785 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "[\r][\n]" 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Receiving response: HTTP/1.1 400 Bad Request 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - << HTTP/1.1 400 Bad Request 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - << X-SERVICE-VERSION: 47538@IRL-ML-DUBLIN.local-UNKNOWN 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - << X-SERVICE-ID: unknown.47538.IRL-ML-DUBLIN 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - << Content-Type: application/octet-stream 2014-02-05 17:20:57,786 [updateExecutor-1-thread-1] DEBUG org.apache.http.headers - << Content-Length: 114 2014-02-05 17:20:57,789 [updateExecutor-1-thread-1] DEBUG org.apache.http.impl.client.DefaultHttpClient - Connection can be kept alive indefinitely 2014-02-05 17:20:57,791 [updateExecutor-1-thread-1] ERROR org.apache.solr.update.StreamingSolrServers - errororg.apache.solr.common.SolrException: Bad Request request: http://172.16.30.1:9766/skus/update?update.chain=custom&_version_=-1459220232373534720&update.distrib=FROMLEADER&update.from=http%3A%2F%2F172.16.30.1%3A9765%2Fskus%2F&wt=javabin&version=2 at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) 2014-02-05 17:20:57,792 [updateExecutor-1-thread-1] DEBUG org.apache.http.wire - << "[0x2][0xa2][0xe0].responseHeader[0xa2][0xe0]&statusP[0x19][0xe0]%QTime^[0x2][0xe0]%error[0xa2][0xe0]#msg?[0x16]Invalid UTF-8 middle byte 0xe0 (at char #1, byte #-1)[0xe0]$codeP[0x19]"
