I opened now https://issues.apache.org/jira/browse/SOLR-13239 for the
problem I observed.
Well, who can really be sure about those things. But I would assume it
should still be ok. The number of watchers should still not be gigantic.
I have setups with about 2000 collections each but far less JVMs. ZK
distributes the watches over the all nodes, which should also include
observer nodes.
Said that an alternative could be to refresh the cache asynchronously to
the call detecting it to be outdated. Wouldn't the worst case be that a
request gets send to a Solr node that has to forward the request to the
correct node? The chance for the cache entry to be wrong after just one
minute is however quite low. So in most cases the request would still be
send to the correct node without having to wait for the cache update and
without potentially blocking other requests. In a performance test we
saw quite a few threads being blocked at this point.
regards,
Hendrik
On 09.02.2019 20:40, Erick Erickson wrote:
Jason's comments are exactly why there _is_ a state.json per
collection rather than the single clusterstate.json in the original
implementation.
Hendrik:
yes, please do open a JIRA for the condition you observed,
especially if you can point to the suspect code. There have
been intermittent issues with collection creation in the test
shells.
About the watchers.
bq. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK.
Consider an installation I have witnessed with 450K replicas scattered
over 100s of collections and 100s of JVMs. Each JVM may have one
or more CloudSolrClients. Are you _sure_ ZK can handle that kind
of watch load? The current architecture allows there to be many fewer
watches set, partially to deal with this scale. And even at this scale,
an incoming request to a node that does _not_ host _any_ replica of
the target collection needs to be able to forward the request, but doesn't
need to know much else about the target collections.
Best,
Erick
On Fri, Feb 8, 2019 at 5:23 PM Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote:
Hi Jason,
thanks for your answer. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK. I would assume that the Solr server instances need to
monitor those nodes to be up to date on the cluster state. Using
org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher
you can even add a watch for that using the SolrJ API. At least for the
currently watched collections the client should thus actually already
have the correct information available. The access to that would likely
be a bit ugly though.
The CloudSolrClient also allows to set a watch on /collections using
org.apache.solr.common.cloud.ZkStateReader.registerCloudCollectionsListener.
This is actually another thing I just ran into. As the code has a watch
on /collections the listener gets informed about new collections as soon
as the "directory" for the collection is being created. If the listener
does then straight away try to access the collection info via
zkStateReader.getClusterState() the DocCollection can be returned as
null as the DocCollection is build on the information stored in the
state.json file, which might not exist yet. I'm trying to monitor the
Solr cluster state and thus ran into this. Not sure if I should open a
Jira for that.
regards,
Hendrik
On 08.02.2019 23:20, Jason Gerlowski wrote:
Hi Henrik,
I'll try to answer, and let others correct me if I stray. I wasn't
around when CloudSolrClient was written, so take this with a grain of
salt:
"Why does the client need that timeout?....Wouldn't it make sense to
use a watch?"
You could probably write a CloudSolrClient that uses watch(es) to keep
track of changing collection state. But I suspect you'd need a
watch-per-collection, instead of just a single watch.
Modern versions of Solr store the state for each collection in
individual "state.json" ZK nodes
("/solr/collections/<collection_name>/state.json"). To catch changes
to all of these collections, you'd need to watch each of those nodes.
Which wouldn't scale well for users who want lots of collections. I
suspect this was one of the concerns that nudged the author(s) to use
a cache-based approach.
(Even when all collection state was stored in a single ZK node, a
watch-based CloudSolrClient would likely have scaling issues for the
many-collection use case. The client would need to recalculate its
state information for _all_ collections any time that _any_ of the
collections changed, since it has no way to tell which collection was
changed.)
Best,
Jason
On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote:
Hi,
when I perform a query using the CloudSolrClient the code first
retrieves the DocCollection to determine to which instance the query
should be send [1]. getDocCollection [2] does a lookup in a cache, which
has a 60s expiration time [3]. When a DocCollection has to be reloaded
this is guarded by a lock [4]. Per default there are 3 locks, which can
cause some congestion. The main question though is why does the client
need that timeout? According to this [5] comment the code does not use a
watch. Wouldn't it make sense to use a watch? I thought the big
advantage of the CloudSolrClient is that is knows were to send requests
to, so that no extra hop needs to be done on the server side. Having to
query ZooKeeper though for the current state does however take some of
that advantage.
regards,
Hendrik
[1]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849
[2]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1180
[3]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L162
[4]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1200
[5]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L821
