Hi Erick,
thx!
the idea behind is to have a dedicated Kubernetes deployment for each
collection. So e.g. if I need more solr nodes for particular collection I would
just scale the Kubernetes deployment and solr should automatically add new
replicas to these new nodes.
does it makes sense?
Br,
Anton
On 12.08.20, 14:05, "Erick Erickson" <erickerick...@gmail.com> wrote:
Autoscaling may be overkill, Is this a one-time thing or something you need
automated?
Because for a one-time event, it’s simpler/faster just to specify
createNodeSet with the CREATE command that lists the new machines you want
the
collection to be placed on.
Note that there’s the special value EMPTY, if you use that then you can
ADDREPLICA
to build out your shards with the “node” parameter to place each one exactly
where you want.
Best,
Erick
> On Aug 12, 2020, at 7:47 AM, Anton Pfennig <anton.pfen...@itolium.com>
wrote:
>
> Hi guys,
>
> in my solr setup as SolrCloud (8.3.1) I’m using 5 nodes for one
collection (say “collection1”, one on each node.
> Now I would like to add a new collection on the same solr cluster but
additionally the new collection (say “collection2”) should be only replicated
on only nodes with sysprop.channel=mysysprop.
>
> Whole setup runs on GKE (Kubernetes)
>
> So If i scale and add additionally 5 nodes wich correctly sysprop.channel
being set and cluster gets a new node, autoscaling dies with
nullpointerexception trying to fetch metrics from new node. (see logs attached).
>
> This is not an IO issue because the nodes and zookeeper can talk to each
other.
> And if I call /metrics on these new nodes I also see the right properties.
>
> Also I see no violations in the suggester.
>
> Please help 😊
>
> THX
> Anton
>
> Attached files:
>
> *
autoscaling.json<https://eco.dev.search.d-p.io/solr-v3/admin/zookeeper?detail=true&path=%2Fautoscaling.json>
> * Logs
>
>
>
> {
> "policies":{"my-channel-policy[{
> "replica":"<100",
> "shard":"#EACH",
> "collection":"collection2",
> "nodeset":{"sysprop.channel":"mychannel"}
> }]},
> "cluster-preferences":[
> {
> "minimize":"cores",
> "precision":1},
> {"maximize":"freedisk"}],
> "triggers":{
> ".auto_add_replicas":{
> "name":".auto_add_replicas",
> "event":"nodeLost",
> "waitFor":120,
> "enabled":true,
> "actions":[
> {
> "name":"auto_add_replicas_plan",
> "class":"solr.AutoAddReplicasPlanAction"},
> {
> "name":"execute_plan",
> "class":"solr.ExecutePlanAction"}]},
> ".scheduled_maintenance":{
> "name":".scheduled_maintenance",
> "event":"scheduled",
> "startTime":"NOW",
> "every":"+1DAY",
> "enabled":true,
> "actions":[
> {
> "name":"inactive_shard_plan",
> "class":"solr.InactiveShardPlanAction"},
> {
> "name":"inactive_markers_plan",
> "class":"solr.InactiveMarkersPlanAction"},
> {
> "name":"execute_plan",
> "class":"solr.ExecutePlanAction"}]},
> "node_added_trigger":{
> "event":"nodeAdded",
> "waitFor":20,
> "enabled":true,
> "actions":[
> {
> "name":"compute_plan",
> "class":"solr.ComputePlanAction"},
> {
> "name":"execute_plan",
> "class":"solr.ExecutePlanAction"}]}},
> "listeners":{
> ".auto_add_replicas.system":{
> "beforeAction":[],
> "afterAction":[],
> "stage":[
> "STARTED",
> "ABORTED",
> "SUCCEEDED",
> "FAILED",
> "BEFORE_ACTION",
> "AFTER_ACTION",
> "IGNORED"],
> "trigger":".auto_add_replicas",
> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener"},
> ".scheduled_maintenance.system":{
> "beforeAction":[],
> "afterAction":[],
> "stage":[
> "STARTED",
> "ABORTED",
> "SUCCEEDED",
> "FAILED",
> "BEFORE_ACTION",
> "AFTER_ACTION",
> "IGNORED"],
> "trigger":".scheduled_maintenance",
> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener"},
> "node_added_trigger.system":{
> "beforeAction":[],
> "afterAction":[],
> "stage":[
> "STARTED",
> "ABORTED",
> "SUCCEEDED",
> "FAILED",
> "BEFORE_ACTION",
> "AFTER_ACTION",
> "IGNORED"],
> "trigger":"node_added_trigger",
> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener"}},
> "properties":{}}
>
>
>
> 2020-07-29 14:54:52.372 WARN (AutoscalingActionExecutor-8-thread-1) [
] o.a.s.c.a.ScheduledTriggers Exception executing actions =>
org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing
action for trigger event: {
> "id":"ca5c53772b355Teaxi61tbu5rj7uqysz5vrcgll",
> org.apache.solr.cloud.autoscaling.TriggerActionException: Error
processing action for trigger event: {
> "id":"ca5c53772b355Teaxi61tbu5rj7uqysz5vrcgll",
> "source":"node_added_trigger",
> "eventTime":3559966177932117,
> "eventType":"NODEADDED",
> "properties":{
> "eventTimes":[3559966177932117],
> "preferredOperation":"movereplica",
> "_enqueue_time_":3559967181279816,
>
"nodeNames":["solr-sede-v3-1.solr-headless-v3.search-preprod-europe-west4-b.svc.cluster.local:8983_solr"],
> "replicaType":"NRT"}}
> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:327)
~[?:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:?]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
> at java.lang.Thread.run(Unknown Source) [?:?]
> Caused by: org.apache.solr.common.SolrException: Unexpected exception
while processing event: {
> "id":"ca5c53772b355Teaxi61tbu5rj7uqysz5vrcgll",
> "source":"node_added_trigger",
> "eventTime":3559966177932117,
> "eventType":"NODEADDED",
> "properties":{
> "eventTimes":[3559966177932117],
> "preferredOperation":"movereplica",
> "_enqueue_time_":3559967181279816,
>
"nodeNames":["solr-sede-v3-1.solr-headless-v3.search-preprod-europe-west4-b.svc.cluster.local:8983_solr"],
> "replicaType":"NRT"}}
> at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:161)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324)
~[?:?]
> ... 6 more
> Caused by: org.apache.solr.common.SolrException:
org.apache.solr.common.SolrException: Error getting remote info
> at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:78)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[?:?]
> at org.apache.solr.client.solrj.cloud.autoscaling.Row.<init>(Row.java:71)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.<init>(Policy.java:575)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:396)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:358)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.createSession(PolicyHelper.java:492)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.get(PolicyHelper.java:457)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getSession(PolicyHelper.java:513)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:90)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324)
~[?:?]
> ... 6 more
> Caused by: org.apache.solr.common.SolrException: Error getting remote info
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:364)
~[?:?]
> at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[?:?]
> at org.apache.solr.client.solrj.cloud.autoscaling.Row.<init>(Row.java:71)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.<init>(Policy.java:575)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:396)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:358)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.createSession(PolicyHelper.java:492)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.get(PolicyHelper.java:457)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getSession(PolicyHelper.java:513)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:90)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324)
~[?:?]
> ... 6 more
> Caused by: java.lang.NullPointerException
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:338)
~[?:?]
> at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[?:?]
> at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[?:?]
> at org.apache.solr.client.solrj.cloud.autoscaling.Row.<init>(Row.java:71)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.<init>(Policy.java:575)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:396)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.createSession(Policy.java:358)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.createSession(PolicyHelper.java:492)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.get(PolicyHelper.java:457)
~[?:?]
> at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getSession(PolicyHelper.java:513)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:90)
~[?:?]
> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324)
~[?:?]
> ... 6 more
>
>
>
