Thanks Emir - responses inline below
>Can you please confirm that Solr nodes are aware of entire ZK ensemble?
Can you explain how I could find that out - I looked into the logs and the
Admin UI and didn¹t see
A way to examine if the Solr nodes saw the entire ensemble
>Can you give more info how it is deployed - ZK on separate servers?
3 Zookeeper nodes (separate servers)
3 Shards and 3 replicas (again each node on separate servers)
Deployed in same AWS region and same AZ
>What is load on Solr when it happens?
Very minimal
Although it seems to occur around or after the time I do a Curl command to
update a field or field type in a collection.
See curl request below
>Do you see any errors in Solr logs?
I see the following in the log file of one of the servers that dropped out
of the cluster
597534321 INFO
(zkCallback-4-thread-1094-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533348 INFO
(zkCallback-4-thread-1079-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574579 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1074-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576496 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597577450 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533345 INFO
(zkCallback-4-thread-1093-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533342 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1084-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597644246 INFO
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597674162 WARN
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader ZooKeeper watch triggered, but Solr cannot
talk to ZK
597628388 ERROR
(zkCallback-4-thread-881-processing-n:52.91.90.134:8983_solr-EventThread)
[ ] o.a.z.ClientCnxn Error while calling watcher
java.lang.OutOfMemoryError: Java heap space
597593838 INFO
(zkCallback-4-thread-1103-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
So it looks like it ran out of memory . . . Strange but I thought my
collections were pretty small.
Any idea why a replace-field-type call might cause an OutOfMemoryException?
-Frank
Example Curl request that causes Solr nodes to appear as ³Gone²
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-field-type" : {
"name":"text_ws",
"class":"solr.TextField",
"positionIncrementGap":"100",
"indexAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.LowerCaseFilterFactory" },
{ "class":"solr.EdgeNGramFilterFactory",
"minGramSize":"2", "maxGramSize":"15" }
]
},
"queryAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.SynonymFilterFactory",
"synonyms":"synonyms.txt", "ignoreCase":true, "expand":true },
{ "class":"solr.LowerCaseFilterFactory" }
]
}
}
}' http://aa.bb.cc.dd:8983/solr/my_collection_name/schema
On 12/1/15, 7:45 AM, "Emir Arnautovic" <emir.arnauto...@sematext.com>
wrote:
>Hi Frank,
>Can you please confirm that Solr nodes are aware of entire ZK ensemble?
>Can you give more info how it is deployed - ZK on separate servers? What
>is load on Solr when it happens? Do you see any errors in Solr logs?
>
>Thanks,
>Emir
>
