<https://issues.apache.org/jira/browse/SOLR-13471>
Finally had a chance to get back to this and figured out what our problem
was. Thanks to Mikhail for pointing me in the right direction. For some
reason one of our custom UpdateRequestProcessors was calling
super.processAdd as the first line in our overridden processAdd method. It
looked something like this:
@Override
public void processAdd(AddUpdateCommand cmd) throws IOException {
super.processAdd(cmd);
SolrInputDocument doc = cmd.getSolrInputDocument();
// Do stuff and set field in doc
super.next.processAdd(cmd);
}
Once I got rid of the first line, everything worked fine. I suspect the
fact that we were going through a decent portion of the update request
processor chain twice was causing some unknown issues. The big issue
causing the ConcurrentModificationException, however, appeared to happen
when trying to distribute the update to a replica. It looks like it would
try and write out to the replica as we'd take the second trip through our
processor chain and thus two things would try and modify the
SolrInputDocument at the same time causing the exception. This would
additionally cause a ClassCastException (java.lang.String cannot be cast to
java.util.Map) on the replica as in issue SOLR-13471
<https://issues.apache.org/jira/browse/SOLR-13471>.
Anyway, thanks for the insight everyone,
Tim
On Fri, Nov 8, 2019 at 12:26 AM Shawn Heisey <apa...@elyograg.org> wrote:
> On 11/6/2019 8:17 AM, Tim Swetland wrote:
> > I'm currently running into a ConcurrentModificationException ingesting
> data
> > as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document,
> but
> > it definitely appears regularly in our logs. We didn't run into this
> > problem in 8.1, so I'm not sure what might have changed. I feel like this
> > is probably a bug, but if there's a workaround or if there's an idea of
> > something I might be doing wrong, please let me know.
> >
> > Stack trace:
> > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>};
> node=StdNode:
> > https://<server>/solr/coll_shard1_replica_n2/ to
> > https://<server>/solr/coll_shard1_replica_n2/
> > => java.util.ConcurrentModificationException
> > at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > java.util.ConcurrentModificationException: null
> > at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > at
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
>
> This error, as mentioned in the SOLR-8028 issue linked by Edward
> Ribeiro, sounds like you are re-using objects when you are building
> SolrInputDocument instances for your indexing.
>
> Looking at the actual code involved in the stacktrace, I think what's
> happening is that at the same time SolrJ is converting a
> SolrInputDocument object to the javabin format so it can be sent to
> Solr, something else has modified that SolrInputDocument object.
>
> You should never re-use SolrJ objects that you construct for indexing.
> A brand new SolrInputDocument instance should be created every time you
> need one, and any objects that go into its creation should also be new
> objects. This is especially important when the code is multi-threaded,
> but there are certain code constructs that can cause this to happen even
> in code that does not create multiple threads. Also, code that uses
> CloudSolrClient can very easily become multi-threaded even if the user's
> code isn't.
>
> In Solr 8.2, clients and the server were updated to use HTTP/2. I did
> not closely follow the work for this, but it would have required some
> pretty major changes to SolrJ, changes that very well could have altered
> the precise timing of operations. Your additional note says you are
> also seeing this problem with 8.1, which does not surprise me. The
> precise timing of the 8.1 code might have been such that the problem was
> far less noticeable.
>
> If you can share your code that uses SolrJ, we *might* be able to help
> you narrow down what's happening and get it fixed.
>
> Thanks,
> Shawn
>
