[
https://issues.apache.org/jira/browse/SOLR-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980879#comment-16980879
]
Colvin Cowie commented on SOLR-13963:
-------------------------------------
[~noble.paul] Eclipse is playing up and I need to head off.
If you run the test in the patch without the extra synchronized{} block you
(should) see it fail every time.
I've attached another version of the JBC where I've replaced the synchronized{}
with a Reentrant lock and thrown an exception when the lock for the instance is
held by another thread, which shows that it is happening.
If there's more detail you need after you've tried running the test, then let
me know and I can get back to you tomorrow.
When I was debugging it earlier, I grabbed this stacktrace where two threads
were modifying the same instance. The line numbers won't match here because the
code was formatted differently.
{noformat}
Thread [qtp1047503754-123] (Suspended)
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec)._readStr(DataInputInputStream,
JavaBinCodec$StringCache, int) line: 931
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readStr(DataInputInputStream,
JavaBinCodec$StringCache, boolean) line: 920
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readExternString(DataInputInputStream)
line: 1190
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
line: 302
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
line: 280
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readSolrInputDocument(DataInputInputStream)
line: 626
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
line: 339
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
line: 280
JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(DataInputInputStream)
line: 321
JavaBinUpdateRequestCodec$StreamingCodec.readIterator(DataInputInputStream)
line: 280
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
line: 335
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
line: 280
JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(DataInputInputStream)
line: 235
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readObject(DataInputInputStream)
line: 300
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).readVal(DataInputInputStream)
line: 280
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).unmarshal(InputStream)
line: 189
JavaBinUpdateRequestCodec.unmarshal(InputStream,
JavaBinUpdateRequestCodec$StreamingUpdateHandler) line: 126
JavabinLoader.parseAndLoadDocs(SolrQueryRequest, SolrQueryResponse,
InputStream, UpdateRequestProcessor) line: 123
JavabinLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream,
UpdateRequestProcessor) line: 70
UpdateRequestHandler$1.load(SolrQueryRequest, SolrQueryResponse,
ContentStream, UpdateRequestProcessor) line: 97
UpdateRequestHandler(ContentStreamHandlerBase).handleRequestBody(SolrQueryRequest,
SolrQueryResponse) line: 68
UpdateRequestHandler(RequestHandlerBase).handleRequest(SolrQueryRequest,
SolrQueryResponse) line: 198
SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
line: 2576
HttpSolrCall.execute(SolrQueryResponse) line: 803
HttpSolrCall.call() line: 582
RobustSolrDispatchFilter(SolrDispatchFilter).doFilter(ServletRequest,
ServletResponse, FilterChain, boolean) line: 424
RobustSolrDispatchFilter(SolrDispatchFilter).doFilter(ServletRequest,
ServletResponse, FilterChain) line: 351
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line:
1602
ServletHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 540
ServletHandler(ScopedHandler).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 146
ConstraintSecurityHandler(SecurityHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 548
SessionHandler(HandlerWrapper).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 132
SessionHandler(ScopedHandler).nextHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 257
SessionHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 1711
WebAppContext(ScopedHandler).nextHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 255
WebAppContext(ContextHandler).doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 1347
ServletHandler(ScopedHandler).nextScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 203
ServletHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 480
SessionHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 1678
WebAppContext(ScopedHandler).nextScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 201
WebAppContext(ContextHandler).doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 1249
WebAppContext(ScopedHandler).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 144
Server(HandlerWrapper).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 132
Server.handle(HttpChannel) line: 505
HttpChannelOverHttp(HttpChannel).handle() line: 370
HttpConnection.onFillable() line: 267
AbstractConnection$ReadCallback.succeeded() line: 305
AbstractEndPoint$1(FillInterest).fillable() line: 103
ChannelEndPoint$2.run() line: 117
EatWhatYouKill.runTask(Runnable) line: 333
EatWhatYouKill.doProduce(boolean) line: 310
EatWhatYouKill.tryProduce(boolean) line: 168
EatWhatYouKill.run() line: 126
ReservedThreadExecutor$ReservedThread.run() line: 366
QueuedThreadPool.runJob(Runnable) line: 781
QueuedThreadPool$Runner.run() line: 917
Thread.run() line: 748
Thread [updateExecutor-3-thread-4] (Suspended)
owns: JavaBinUpdateRequestCodec$StreamingCodec (id=203)
JavaBinUpdateRequestCodec$StreamingCodec(JavaBinCodec).lambda$1(ByteArrayUtf8CharSequence)
line: 969
239370648.apply(Object) line: not available
ByteArrayUtf8CharSequence._getStr() line: 156
ByteArrayUtf8CharSequence.toString() line: 235
ByteArrayUtf8CharSequence.convertCharSeq(Object) line: 215
SolrInputField.getValue() line: 116
SolrInputDocument.lambda$0(BiConsumer, CharSequence, Object) line: 55
717260320.accept(Object, Object) line: not available
LinkedHashMap<K,V>.forEach(BiConsumer<? super K,? super V>) line: 684
JavaBinUpdateRequestCodec$MaskCharSequenceSolrInputDoc(SolrInputDocument).writeMap(MapWriter$EntryWriter)
line: 59
JavaBinCodec.writeSolrInputDocument(SolrInputDocument) line: 660
JavaBinCodec.writeKnownType(Object) line: 383
JavaBinCodec.writeVal(Object) line: 252
JavaBinCodec.writeMapEntry(Map$Entry) line: 815
JavaBinCodec.writeKnownType(Object) line: 411
JavaBinCodec.writeVal(Object) line: 252
JavaBinCodec.writeIterator(Iterator) line: 752
JavaBinCodec.writeKnownType(Object) line: 395
JavaBinCodec.writeVal(Object) line: 252
JavaBinCodec.writeNamedList(NamedList<?>) line: 247
JavaBinCodec.writeKnownType(Object) line: 354
JavaBinCodec.writeVal(Object) line: 252
JavaBinCodec.marshal(Object, OutputStream) line: 162
JavaBinUpdateRequestCodec.marshal(UpdateRequest, OutputStream) line: 102
BinaryRequestWriter.write(SolrRequest, OutputStream) line: 83
Http2SolrClient.send(Http2SolrClient$OutStream, SolrRequest, String) line: 340
ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream() line: 231
ConcurrentUpdateHttp2SolrClient$Runner.run() line: 176
InstrumentedExecutorService$InstrumentedRunnable.run() line: 181
ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ArrayList, List, Map,
String, Runnable, Exception) line: 210
1639765351.run() line: not available
ExecutorUtil$MDCAwareThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker)
line: 1149
ThreadPoolExecutor$Worker.run() line: 624
Thread.run() line: 748{noformat}
> JavaBinCodec has concurrent modification of CharArr resulting in corrupt
> intranode updates
> ------------------------------------------------------------------------------------------
>
> Key: SOLR-13963
> URL: https://issues.apache.org/jira/browse/SOLR-13963
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 8.3
> Reporter: Colvin Cowie
> Assignee: Noble Paul
> Priority: Major
> Attachments: JavaBinCodec.java, SOLR-13963.patch
>
>
> Discussed on the mailing list "Possible data corruption in JavaBinCodec in
> Solr 8.3 during distributed update?"
>
> In summary, after moving to 8.3 we had a consistent (but non-deterministic)
> set of failing tests where the data being sent in intranode requests was
> _sometimes_ corrupted. For example if the well formed data was
> _'fieldName':"this is a long string"_
> The error we saw from Solr might be that
> unknown field _+'fieldNamis a long string"+_
>
> The change that indirectly caused to this issue to materialize was from
> SOLR-13682 which meant that
> org.apache.solr.common.SolrInputDocument.writeMap(EntryWriter) would call
> org.apache.solr.common.SolrInputField.getValue() rather than
> org.apache.solr.common.SolrInputField.getRawValue() as it had before.
>
> getRawValue for a string calls
> org.apache.solr.common.util.ByteArrayUtf8CharSequence._getStr() which in this
> context calls
> org.apache.solr.common.util.JavaBinCodec.getStringProvider()
>
> JavaBinCodec has a CharArr, _arr_, which is modified in two different
> locations, but only one of which is protected with a synchronized block
>
> getStringProvider() synchronizes on _arr_:
>
> [https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966]
>
> but _readStr() doesn't:
>
> [https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930]
>
> The two methods are called concurrently, but wheren't prior to SOLR-13682.
>
> Adding a synchronized block into _readStr() around the modification of _arr_
> fixes the problem as far as I can see.
>
> Also, the problem does not seem to occur when using the dynamic schema mode
> of autoCreateFields=true in the updateRequestProcessorChain.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]