[ 
https://issues.apache.org/jira/browse/SOLR-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980879#comment-16980879
 ] 

Colvin Cowie commented on SOLR-13963:
-------------------------------------

[~noble.paul] Eclipse is playing up and I need to head off.

If you run the test in the patch without the extra synchronized{} block you 
(should) see it fail every time.

I've attached another version of the JBC where I've replaced the synchronized{} 
with a Reentrant lock and thrown an exception when the lock for the instance is 
held by another thread, which shows that it is happening.

If there's more detail you need after you've tried running the test, then let 
me know and I can get back to you tomorrow.

When I was debugging it earlier, I grabbed this stacktrace where two threads 
were modifying the same instance. The line numbers won't match here because the 
code was formatted differently.
{noformat}
Thread [qtp1047503754-123] (Suspended) 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec)._readStr(DataInputInputStream,
 JavaBinCodec$StringCache, int) line: 931 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readStr(DataInputInputStream,
 JavaBinCodec$StringCache, boolean) line: 920 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readExternString(DataInputInputStream)
 line: 1190 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
 line: 302 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
 line: 280 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readSolrInputDocument(DataInputInputStream)
 line: 626 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
 line: 339 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
 line: 280 
 
JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(DataInputInputStream)
 line: 321 
 JavaBinUpdateRequestCodec$StreamingCodec.readIterator(DataInputInputStream) 
line: 280 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
 line: 335 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
 line: 280 
 JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(DataInputInputStream) 
line: 235 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
 line: 300 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
 line: 280 
 JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).unmarshal(InputStream) 
line: 189 
 JavaBinUpdateRequestCodec.unmarshal(InputStream, 
JavaBinUpdateRequestCodec$StreamingUpdateHandler) line: 126 
 JavabinLoader.parseAndLoadDocs(SolrQueryRequest, SolrQueryResponse, 
InputStream, UpdateRequestProcessor) line: 123 
 JavabinLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream, 
UpdateRequestProcessor) line: 70 
 UpdateRequestHandler$1.load(SolrQueryRequest, SolrQueryResponse, 
ContentStream, UpdateRequestProcessor) line: 97 
 
UpdateRequestHandler(ContentStreamHandlerBase).handleRequestBody(SolrQueryRequest,
 SolrQueryResponse) line: 68 
 UpdateRequestHandler(RequestHandlerBase).handleRequest(SolrQueryRequest, 
SolrQueryResponse) line: 198 
 SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse) 
line: 2576 
 HttpSolrCall.execute(SolrQueryResponse) line: 803 
 HttpSolrCall.call() line: 582 
 RobustSolrDispatchFilter(SolrDispatchFilter).doFilter(ServletRequest, 
ServletResponse, FilterChain, boolean) line: 424 
 RobustSolrDispatchFilter(SolrDispatchFilter).doFilter(ServletRequest, 
ServletResponse, FilterChain) line: 351 
 ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 
1602 
 ServletHandler.doHandle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 540 
 ServletHandler(ScopedHandler).handle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 146 
 ConstraintSecurityHandler(SecurityHandler).handle(String, Request, 
HttpServletRequest, HttpServletResponse) line: 548 
 SessionHandler(HandlerWrapper).handle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 132 
 SessionHandler(ScopedHandler).nextHandle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 257 
 SessionHandler.doHandle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 1711 
 WebAppContext(ScopedHandler).nextHandle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 255 
 WebAppContext(ContextHandler).doHandle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 1347 
 ServletHandler(ScopedHandler).nextScope(String, Request, HttpServletRequest, 
HttpServletResponse) line: 203 
 ServletHandler.doScope(String, Request, HttpServletRequest, 
HttpServletResponse) line: 480 
 SessionHandler.doScope(String, Request, HttpServletRequest, 
HttpServletResponse) line: 1678 
 WebAppContext(ScopedHandler).nextScope(String, Request, HttpServletRequest, 
HttpServletResponse) line: 201 
 WebAppContext(ContextHandler).doScope(String, Request, HttpServletRequest, 
HttpServletResponse) line: 1249 
 WebAppContext(ScopedHandler).handle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 144 
 Server(HandlerWrapper).handle(String, Request, HttpServletRequest, 
HttpServletResponse) line: 132 
 Server.handle(HttpChannel) line: 505 
 HttpChannelOverHttp(HttpChannel).handle() line: 370 
 HttpConnection.onFillable() line: 267 
 AbstractConnection$ReadCallback.succeeded() line: 305 
 AbstractEndPoint$1(FillInterest).fillable() line: 103 
 ChannelEndPoint$2.run() line: 117 
 EatWhatYouKill.runTask(Runnable) line: 333 
 EatWhatYouKill.doProduce(boolean) line: 310 
 EatWhatYouKill.tryProduce(boolean) line: 168 
 EatWhatYouKill.run() line: 126 
 ReservedThreadExecutor$ReservedThread.run() line: 366 
 QueuedThreadPool.runJob(Runnable) line: 781 
 QueuedThreadPool$Runner.run() line: 917 
 Thread.run() line: 748 
 
 
Thread [updateExecutor-3-thread-4] (Suspended) 
 owns: JavaBinUpdateRequestCodec$StreamingCodec (id=203) 
 
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).lambda$1(ByteArrayUtf8CharSequence)
 line: 969 
 239370648.apply(Object) line: not available 
 ByteArrayUtf8CharSequence._getStr() line: 156 
 ByteArrayUtf8CharSequence.toString() line: 235 
 ByteArrayUtf8CharSequence.convertCharSeq(Object) line: 215 
 SolrInputField.getValue() line: 116 
 SolrInputDocument.lambda$0(BiConsumer, CharSequence, Object) line: 55 
 717260320.accept(Object, Object) line: not available 
 LinkedHashMap<K,V>.forEach(BiConsumer<? super K,? super V>) line: 684 
 
JavaBinUpdateRequestCodec$MaskCharSequenceSolrInputDoc(SolrInputDocument).writeMap(MapWriter$EntryWriter)
 line: 59 
 JavaBinCodec.writeSolrInputDocument(SolrInputDocument) line: 660 
 JavaBinCodec.writeKnownType(Object) line: 383 
 JavaBinCodec.writeVal(Object) line: 252 
 JavaBinCodec.writeMapEntry(Map$Entry) line: 815 
 JavaBinCodec.writeKnownType(Object) line: 411 
 JavaBinCodec.writeVal(Object) line: 252 
 JavaBinCodec.writeIterator(Iterator) line: 752 
 JavaBinCodec.writeKnownType(Object) line: 395 
 JavaBinCodec.writeVal(Object) line: 252 
 JavaBinCodec.writeNamedList(NamedList<?>) line: 247 
 JavaBinCodec.writeKnownType(Object) line: 354 
 JavaBinCodec.writeVal(Object) line: 252 
 JavaBinCodec.marshal(Object, OutputStream) line: 162 
 JavaBinUpdateRequestCodec.marshal(UpdateRequest, OutputStream) line: 102 
 BinaryRequestWriter.write(SolrRequest, OutputStream) line: 83 
 Http2SolrClient.send(Http2SolrClient$OutStream, SolrRequest, String) line: 340 
 ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream() line: 231 
 ConcurrentUpdateHttp2SolrClient$Runner.run() line: 176 
 InstrumentedExecutorService$InstrumentedRunnable.run() line: 181 
 ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ArrayList, List, Map, 
String, Runnable, Exception) line: 210 
 1639765351.run() line: not available 
 
ExecutorUtil$MDCAwareThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker)
 line: 1149 
 ThreadPoolExecutor$Worker.run() line: 624 
 Thread.run() line: 748{noformat}

> JavaBinCodec has concurrent modification of CharArr resulting in corrupt 
> intranode updates
> ------------------------------------------------------------------------------------------
>
>                 Key: SOLR-13963
>                 URL: https://issues.apache.org/jira/browse/SOLR-13963
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 8.3
>            Reporter: Colvin Cowie
>            Assignee: Noble Paul
>            Priority: Major
>         Attachments: JavaBinCodec.java, SOLR-13963.patch
>
>
> Discussed on the mailing list "Possible data corruption in JavaBinCodec in 
> Solr 8.3 during distributed update?"
>  
> In summary, after moving to 8.3 we had a consistent (but non-deterministic) 
> set of failing tests where the data being sent in intranode requests was 
> _sometimes_ corrupted. For example if the well formed data was
>  _'fieldName':"this is a long string"_
>  The error we saw from Solr might be that
>  unknown field _+'fieldNamis a long string"+_ 
>   
>  The change that indirectly caused to this issue to materialize was from 
> SOLR-13682 which meant that 
> org.apache.solr.common.SolrInputDocument.writeMap(EntryWriter) would call 
> org.apache.solr.common.SolrInputField.getValue() rather than 
> org.apache.solr.common.SolrInputField.getRawValue() as it had before.
>   
>  getRawValue for a string calls 
> org.apache.solr.common.util.ByteArrayUtf8CharSequence._getStr() which in this 
> context calls
>  org.apache.solr.common.util.JavaBinCodec.getStringProvider()
>  
>  JavaBinCodec has a CharArr, _arr_, which is modified in two different 
> locations, but only one of which is protected with a synchronized block
>   
>  getStringProvider() synchronizes on _arr_:
>  
> [https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966]
>   
>  but  _readStr() doesn't:
>  
> [https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930]
>   
>  The two methods are called concurrently, but wheren't prior to SOLR-13682.
>   
>  Adding a synchronized block into _readStr() around the modification of _arr_ 
> fixes the problem as far as I can see.
>  
> Also, the problem does not seem to occur when using the dynamic schema mode 
> of autoCreateFields=true in the updateRequestProcessorChain.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to