Hi Frank,
Seems like OOM is reason why Solr failed. What is size of your
collection and heap size. You could do thread and heap dump and see what
is taking memory. Not sure how threads are named but hope that
thread-1103 does not mean there are 1103 threads reading ZK.
Regards,
Emir
On 01.12.2015 16:21, Kelly, Frank wrote:
Thanks Emir - responses inline below
Can you please confirm that Solr nodes are aware of entire ZK ensemble?
Can you explain how I could find that out - I looked into the logs and the
Admin UI and didn¹t see
A way to examine if the Solr nodes saw the entire ensemble
Did you provide list of ZK on startup?
Can you give more info how it is deployed - ZK on separate servers?
3 Zookeeper nodes (separate servers)
3 Shards and 3 replicas (again each node on separate servers)
Deployed in same AWS region and same AZ
What is load on Solr when it happens?
Very minimal
Although it seems to occur around or after the time I do a Curl command to
update a field or field type in a collection.
See curl request below
Do you see any errors in Solr logs?
I see the following in the log file of one of the servers that dropped out
of the cluster
597534321 INFO
(zkCallback-4-thread-1094-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533348 INFO
(zkCallback-4-thread-1079-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574579 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1074-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576496 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597577450 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533345 INFO
(zkCallback-4-thread-1093-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533342 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1084-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597644246 INFO
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597674162 WARN
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader ZooKeeper watch triggered, but Solr cannot
talk to ZK
597628388 ERROR
(zkCallback-4-thread-881-processing-n:52.91.90.134:8983_solr-EventThread)
[ ] o.a.z.ClientCnxn Error while calling watcher
java.lang.OutOfMemoryError: Java heap space
597593838 INFO
(zkCallback-4-thread-1103-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
So it looks like it ran out of memory . . . Strange but I thought my
collections were pretty small.
Any idea why a replace-field-type call might cause an OutOfMemoryException?
-Frank
Example Curl request that causes Solr nodes to appear as ³Gone²
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-field-type" : {
"name":"text_ws",
"class":"solr.TextField",
"positionIncrementGap":"100",
"indexAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.LowerCaseFilterFactory" },
{ "class":"solr.EdgeNGramFilterFactory",
"minGramSize":"2", "maxGramSize":"15" }
]
},
"queryAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.SynonymFilterFactory",
"synonyms":"synonyms.txt", "ignoreCase":true, "expand":true },
{ "class":"solr.LowerCaseFilterFactory" }
]
}
}
}' http://aa.bb.cc.dd:8983/solr/my_collection_name/schema
On 12/1/15, 7:45 AM, "Emir Arnautovic" <emir.arnauto...@sematext.com>
wrote:
Hi Frank,
Can you please confirm that Solr nodes are aware of entire ZK ensemble?
Can you give more info how it is deployed - ZK on separate servers? What
is load on Solr when it happens? Do you see any errors in Solr logs?
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
