[ https://issues.apache.org/jira/browse/SOLR-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102323#comment-17102323 ]
Bernd Wahlen edited comment on SOLR-14463 at 5/8/20, 7:21 AM: -------------------------------------------------------------- You are right zk_followers is missing, but cluster is working @see zk_synced_followers. So what do you think? Workaroud it in solr, open a zookeeper bug (zk_followers is still in the documentation of 3.6.1) or both? {code:java} zk_version 3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on 04/21/2020 15:01 GMT zk_server_state leader zk_peer_state leading - broadcast zk_ephemerals_count 19 zk_num_alive_connections 6 zk_avg_latency 0.4362 zk_min_proposal_size 36 zk_outstanding_requests 0 zk_znode_count 138 zk_global_sessions 15 zk_last_client_response_size 16 zk_packets_sent 38309 zk_packets_received 38264 zk_max_proposal_size 1314 zk_max_client_response_size 70313 zk_synced_followers 2 zk_connection_drop_probability 0.0 zk_watch_count 25 zk_synced_non_voting_followers 0 zk_synced_observers 0 zk_min_latency 0 zk_max_file_descriptor_count 1048576 zk_approximate_data_size 308288 zk_open_file_descriptor_count 67 zk_last_proposal_size 36 zk_pending_syncs 0 zk_leader_uptime 61771446 zk_local_sessions 0 zk_uptime 61771582 zk_max_latency 2738 zk_outstanding_tls_handshake 0 zk_learners 2 zk_min_client_response_size 16 zk_quorum_size 3 zk_proposal_count 121 zk_outstanding_changes_removed 124 zk_stale_requests_dropped 0 zk_large_requests_rejected 0 zk_connection_rejected 0 zk_sessionless_connections_expired 0 zk_looking_count 2 zk_dead_watchers_queued 0 zk_stale_requests 0 zk_connection_drop_count 1 zk_learner_proposal_received_count 71 zk_digest_mismatches_count 0 zk_dead_watchers_cleared 0 zk_response_packet_cache_hits 1048 zk_bytes_received_count 701380 zk_add_dead_watcher_stall_time 0 zk_request_throttle_wait_count 0 zk_response_packet_cache_misses 47 zk_ensemble_auth_success 0 zk_prep_processor_request_queued 38168 zk_learner_commit_received_count 71 zk_stale_replies 0 zk_connection_request_count 7 zk_ensemble_auth_fail 0 zk_diff_count 2 zk_response_packet_get_children_cache_misses 0 zk_connection_revalidate_count 5 zk_quit_leading_due_to_disloyal_voter 0 zk_snap_count 0 zk_unrecoverable_error_count 0 zk_commit_count 121 zk_stale_sessions_expired 0 zk_response_packet_get_children_cache_hits 0 zk_sync_processor_request_queued 195 zk_outstanding_changes_queued 124 zk_request_commit_queued 195 zk_ensemble_auth_skip 0 zk_tls_handshake_exceeded 0 zk_revalidate_count 10 zk_avg_node_created_watch_count 0.0 zk_min_node_created_watch_count 0 zk_max_node_created_watch_count 0 zk_cnt_node_created_watch_count 0 zk_sum_node_created_watch_count 0 zk_avg_session_queues_drained 0.6205 zk_min_session_queues_drained 0 zk_max_session_queues_drained 1 zk_cnt_session_queues_drained 195 zk_sum_session_queues_drained 121 zk_avg_write_commit_proc_req_queued 0.0117 zk_min_write_commit_proc_req_queued 0 zk_max_write_commit_proc_req_queued 7 zk_cnt_write_commit_proc_req_queued 37821 zk_sum_write_commit_proc_req_queued 444 zk_avg_connection_token_deficit 0.0 zk_min_connection_token_deficit 0 zk_max_connection_token_deficit 0 zk_cnt_connection_token_deficit 7 zk_sum_connection_token_deficit 0 zk_avg_read_commit_proc_req_queued 1.0047 zk_min_read_commit_proc_req_queued 0 zk_max_read_commit_proc_req_queued 5 zk_cnt_read_commit_proc_req_queued 37821 zk_sum_read_commit_proc_req_queued 37998 zk_avg_node_deleted_watch_count 0.0 zk_min_node_deleted_watch_count 0 zk_max_node_deleted_watch_count 0 zk_cnt_node_deleted_watch_count 0 zk_sum_node_deleted_watch_count 0 zk_avg_startup_txns_load_time 19.0 zk_min_startup_txns_load_time 19 zk_max_startup_txns_load_time 19 zk_cnt_startup_txns_load_time 1 zk_sum_startup_txns_load_time 19 zk_avg_sync_processor_queue_size 0.132 zk_min_sync_processor_queue_size 0 zk_max_sync_processor_queue_size 3 zk_cnt_sync_processor_queue_size 197 zk_sum_sync_processor_queue_size 26 zk_avg_follower_sync_time 33.0 zk_min_follower_sync_time 33 zk_max_follower_sync_time 33 zk_cnt_follower_sync_time 1 zk_sum_follower_sync_time 33 zk_avg_prep_processor_queue_size 0.0103 zk_min_prep_processor_queue_size 0 zk_max_prep_processor_queue_size 4 zk_cnt_prep_processor_queue_size 38169 zk_sum_prep_processor_queue_size 393 zk_avg_fsynctime 0.052 zk_min_fsynctime 0 zk_max_fsynctime 4 zk_cnt_fsynctime 173 zk_sum_fsynctime 9 zk_avg_reads_issued_from_session_queue 0.0308 zk_min_reads_issued_from_session_queue 0 zk_max_reads_issued_from_session_queue 2 zk_cnt_reads_issued_from_session_queue 195 zk_sum_reads_issued_from_session_queue 6 zk_avg_snapshottime 6.0 zk_min_snapshottime 6 zk_max_snapshottime 6 zk_cnt_snapshottime 1 zk_sum_snapshottime 6 zk_avg_startup_txns_loaded 137.0 zk_min_startup_txns_loaded 137 zk_max_startup_txns_loaded 137 zk_cnt_startup_txns_loaded 1 zk_sum_startup_txns_loaded 137 zk_avg_reads_after_write_in_session_queue 0.0496 zk_min_reads_after_write_in_session_queue 0 zk_max_reads_after_write_in_session_queue 2 zk_cnt_reads_after_write_in_session_queue 121 zk_sum_reads_after_write_in_session_queue 6 zk_avg_requests_in_session_queue 1.9395 zk_min_requests_in_session_queue 1 zk_max_requests_in_session_queue 7 zk_cnt_requests_in_session_queue 248 zk_sum_requests_in_session_queue 481 zk_avg_write_commit_proc_issued 1.0 zk_min_write_commit_proc_issued 1 zk_max_write_commit_proc_issued 1 zk_cnt_write_commit_proc_issued 195 zk_sum_write_commit_proc_issued 195 zk_avg_prep_process_time 0.0047 zk_min_prep_process_time 0 zk_max_prep_process_time 12 zk_cnt_prep_process_time 38168 zk_sum_prep_process_time 179 zk_avg_pending_session_queue_size 1.0083 zk_min_pending_session_queue_size 1 zk_max_pending_session_queue_size 2 zk_cnt_pending_session_queue_size 121 zk_sum_pending_session_queue_size 122 zk_avg_time_waiting_empty_pool_in_commit_processor_read_ms 0.0 zk_min_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_max_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_cnt_time_waiting_empty_pool_in_commit_processor_read_ms 195 zk_sum_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_avg_commit_process_time 0.0169 zk_min_commit_process_time 0 zk_max_commit_process_time 4 zk_cnt_commit_process_time 37821 zk_sum_commit_process_time 639 zk_avg_dbinittime 39.0 zk_min_dbinittime 39 zk_max_dbinittime 39 zk_cnt_dbinittime 1 zk_sum_dbinittime 39 zk_avg_netty_queued_buffer_capacity 0.0 zk_min_netty_queued_buffer_capacity 0 zk_max_netty_queued_buffer_capacity 0 zk_cnt_netty_queued_buffer_capacity 0 zk_sum_netty_queued_buffer_capacity 0 zk_avg_election_time 125.5 zk_min_election_time 33 zk_max_election_time 218 zk_cnt_election_time 2 zk_sum_election_time 251 zk_avg_commit_commit_proc_req_queued 0.0054 zk_min_commit_commit_proc_req_queued 0 zk_max_commit_commit_proc_req_queued 4 zk_cnt_commit_commit_proc_req_queued 37821 zk_sum_commit_commit_proc_req_queued 204 zk_avg_sync_processor_batch_size 1.1272 zk_min_sync_processor_batch_size 1 zk_max_sync_processor_batch_size 5 zk_cnt_sync_processor_batch_size 173 zk_sum_sync_processor_batch_size 195 zk_avg_node_children_watch_count 6.0 zk_min_node_children_watch_count 6 zk_max_node_children_watch_count 6 zk_cnt_node_children_watch_count 1 zk_sum_node_children_watch_count 6 zk_avg_write_batch_time_in_commit_processor 0.2051 zk_min_write_batch_time_in_commit_processor 0 zk_max_write_batch_time_in_commit_processor 4 zk_cnt_write_batch_time_in_commit_processor 195 zk_sum_write_batch_time_in_commit_processor 40 zk_avg_read_commit_proc_issued 1.0041 zk_min_read_commit_proc_issued 0 zk_max_read_commit_proc_issued 5 zk_cnt_read_commit_proc_issued 37821 zk_sum_read_commit_proc_issued 37975 zk_avg_concurrent_request_processing_in_commit_processor 2.0 zk_min_concurrent_request_processing_in_commit_processor 2 zk_max_concurrent_request_processing_in_commit_processor 2 zk_cnt_concurrent_request_processing_in_commit_processor 1 zk_sum_concurrent_request_processing_in_commit_processor 2 zk_avg_node_changed_watch_count 1.0 zk_min_node_changed_watch_count 1 zk_max_node_changed_watch_count 1 zk_cnt_node_changed_watch_count 16 zk_sum_node_changed_watch_count 16 zk_avg_sync_process_time 0.1077 zk_min_sync_process_time 0 zk_max_sync_process_time 2 zk_cnt_sync_process_time 195 zk_sum_sync_process_time 21 zk_avg_startup_snap_load_time 9.0 zk_min_startup_snap_load_time 9 zk_max_startup_snap_load_time 9 zk_cnt_startup_snap_load_time 1 zk_sum_startup_snap_load_time 9 zk_avg_prep_processor_queue_time_ms 0.0718 zk_min_prep_processor_queue_time_ms 0 zk_max_prep_processor_queue_time_ms 12 zk_cnt_prep_processor_queue_time_ms 38168 zk_sum_prep_processor_queue_time_ms 2742 zk_p50_prep_processor_queue_time_ms 0 zk_p95_prep_processor_queue_time_ms 1 zk_p99_prep_processor_queue_time_ms 1 zk_p999_prep_processor_queue_time_ms 1 zk_avg_close_session_prep_time 0.0 zk_min_close_session_prep_time 0 zk_max_close_session_prep_time 0 zk_cnt_close_session_prep_time 1 zk_sum_close_session_prep_time 0 zk_p50_close_session_prep_time 0 zk_p95_close_session_prep_time 0 zk_p99_close_session_prep_time 0 zk_p999_close_session_prep_time 0 zk_avg_read_commitproc_time_ms 0.2513 zk_min_read_commitproc_time_ms 0 zk_max_read_commitproc_time_ms 2738 zk_cnt_read_commitproc_time_ms 37981 zk_sum_read_commitproc_time_ms 9543 zk_p50_read_commitproc_time_ms 0 zk_p95_read_commitproc_time_ms 1 zk_p99_read_commitproc_time_ms 1 zk_p999_read_commitproc_time_ms 1 zk_avg_updatelatency 1.5143 zk_min_updatelatency 0 zk_max_updatelatency 5 zk_cnt_updatelatency 35 zk_sum_updatelatency 53 zk_p50_updatelatency 1 zk_p95_updatelatency 3 zk_p99_updatelatency 5 zk_p999_updatelatency 5 zk_avg_local_write_committed_time_ms 0.0579 zk_min_local_write_committed_time_ms 0 zk_max_local_write_committed_time_ms 1 zk_cnt_local_write_committed_time_ms 121 zk_sum_local_write_committed_time_ms 7 zk_p50_local_write_committed_time_ms 0 zk_p95_local_write_committed_time_ms 1 zk_p99_local_write_committed_time_ms 1 zk_p999_local_write_committed_time_ms 1 zk_avg_readlatency 0.4352 zk_min_readlatency 0 zk_max_readlatency 2738 zk_cnt_readlatency 37981 zk_sum_readlatency 16528 zk_p50_readlatency 0 zk_p95_readlatency 1 zk_p99_readlatency 1 zk_p999_readlatency 1 zk_avg_quorum_ack_latency 1.0909 zk_min_quorum_ack_latency 0 zk_max_quorum_ack_latency 7 zk_cnt_quorum_ack_latency 121 zk_sum_quorum_ack_latency 132 zk_p50_quorum_ack_latency 1 zk_p95_quorum_ack_latency 2 zk_p99_quorum_ack_latency 6 zk_p999_quorum_ack_latency 7 zk_avg_om_commit_process_time_ms 0.0 zk_min_om_commit_process_time_ms 0 zk_max_om_commit_process_time_ms 0 zk_cnt_om_commit_process_time_ms 0 zk_sum_om_commit_process_time_ms 0 zk_p50_om_commit_process_time_ms 0 zk_p95_om_commit_process_time_ms 0 zk_p99_om_commit_process_time_ms 0 zk_p999_om_commit_process_time_ms 0 zk_avg_read_final_proc_time_ms 0.0255 zk_min_read_final_proc_time_ms 0 zk_max_read_final_proc_time_ms 7 zk_cnt_read_final_proc_time_ms 37981 zk_sum_read_final_proc_time_ms 968 zk_p50_read_final_proc_time_ms 0 zk_p95_read_final_proc_time_ms 0 zk_p99_read_final_proc_time_ms 1 zk_p999_read_final_proc_time_ms 1 zk_avg_commit_propagation_latency 5965.7297 zk_min_commit_propagation_latency 2 zk_max_commit_propagation_latency 147105 zk_cnt_commit_propagation_latency 74 zk_sum_commit_propagation_latency 441464 zk_p50_commit_propagation_latency 3 zk_p95_commit_propagation_latency 36772 zk_p99_commit_propagation_latency 147105 zk_p999_commit_propagation_latency 147105 zk_avg_dead_watchers_cleaner_latency 0.0 zk_min_dead_watchers_cleaner_latency 0 zk_max_dead_watchers_cleaner_latency 0 zk_cnt_dead_watchers_cleaner_latency 0 zk_sum_dead_watchers_cleaner_latency 0 zk_p50_dead_watchers_cleaner_latency 0 zk_p95_dead_watchers_cleaner_latency 0 zk_p99_dead_watchers_cleaner_latency 0 zk_p999_dead_watchers_cleaner_latency 0 zk_avg_write_final_proc_time_ms 0.2 zk_min_write_final_proc_time_ms 0 zk_max_write_final_proc_time_ms 4 zk_cnt_write_final_proc_time_ms 195 zk_sum_write_final_proc_time_ms 39 zk_p50_write_final_proc_time_ms 0 zk_p95_write_final_proc_time_ms 1 zk_p99_write_final_proc_time_ms 4 zk_p999_write_final_proc_time_ms 4 zk_avg_proposal_ack_creation_latency 2264.2769 zk_min_proposal_ack_creation_latency 0 zk_max_proposal_ack_creation_latency 147107 zk_cnt_proposal_ack_creation_latency 195 zk_sum_proposal_ack_creation_latency 441534 zk_p50_proposal_ack_creation_latency 1 zk_p95_proposal_ack_creation_latency 3 zk_p99_proposal_ack_creation_latency 147104 zk_p999_proposal_ack_creation_latency 147107 zk_avg_proposal_latency 1.9014 zk_min_proposal_latency 1 zk_max_proposal_latency 3 zk_cnt_proposal_latency 71 zk_sum_proposal_latency 135 zk_p50_proposal_latency 2 zk_p95_proposal_latency 2 zk_p99_proposal_latency 3 zk_p999_proposal_latency 3 zk_avg_om_proposal_process_time_ms 0.0 zk_min_om_proposal_process_time_ms 0 zk_max_om_proposal_process_time_ms 0 zk_cnt_om_proposal_process_time_ms 0 zk_sum_om_proposal_process_time_ms 0 zk_p50_om_proposal_process_time_ms 0 zk_p95_om_proposal_process_time_ms 0 zk_p99_om_proposal_process_time_ms 0 zk_p999_om_proposal_process_time_ms 0 zk_avg_sync_processor_queue_and_flush_time_ms 0.3846 zk_min_sync_processor_queue_and_flush_time_ms 0 zk_max_sync_processor_queue_and_flush_time_ms 4 zk_cnt_sync_processor_queue_and_flush_time_ms 195 zk_sum_sync_processor_queue_and_flush_time_ms 75 zk_p50_sync_processor_queue_and_flush_time_ms 0 zk_p95_sync_processor_queue_and_flush_time_ms 1 zk_p99_sync_processor_queue_and_flush_time_ms 4 zk_p999_sync_processor_queue_and_flush_time_ms 4 zk_avg_propagation_latency 2264.8564 zk_min_propagation_latency 0 zk_max_propagation_latency 147106 zk_cnt_propagation_latency 195 zk_sum_propagation_latency 441647 zk_p50_propagation_latency 2 zk_p95_propagation_latency 4 zk_p99_propagation_latency 147103 zk_p999_propagation_latency 147106 zk_avg_server_write_committed_time_ms 0.0676 zk_min_server_write_committed_time_ms 0 zk_max_server_write_committed_time_ms 1 zk_cnt_server_write_committed_time_ms 74 zk_sum_server_write_committed_time_ms 5 zk_p50_server_write_committed_time_ms 0 zk_p95_server_write_committed_time_ms 1 zk_p99_server_write_committed_time_ms 1 zk_p999_server_write_committed_time_ms 1 zk_avg_sync_processor_queue_time_ms 0.0923 zk_min_sync_processor_queue_time_ms 0 zk_max_sync_processor_queue_time_ms 2 zk_cnt_sync_processor_queue_time_ms 195 zk_sum_sync_processor_queue_time_ms 18 zk_p50_sync_processor_queue_time_ms 0 zk_p95_sync_processor_queue_time_ms 1 zk_p99_sync_processor_queue_time_ms 2 zk_p999_sync_processor_queue_time_ms 2 zk_avg_sync_processor_queue_flush_time_ms 0.1792 zk_min_sync_processor_queue_flush_time_ms 0 zk_max_sync_processor_queue_flush_time_ms 4 zk_cnt_sync_processor_queue_flush_time_ms 173 zk_sum_sync_processor_queue_flush_time_ms 31 zk_p50_sync_processor_queue_flush_time_ms 0 zk_p95_sync_processor_queue_flush_time_ms 1 zk_p99_sync_processor_queue_flush_time_ms 3 zk_p999_sync_processor_queue_flush_time_ms 4 zk_avg_write_commitproc_time_ms 0.9008 zk_min_write_commitproc_time_ms 0 zk_max_write_commitproc_time_ms 3 zk_cnt_write_commitproc_time_ms 121 zk_sum_write_commitproc_time_ms 109 zk_p50_write_commitproc_time_ms 1 zk_p95_write_commitproc_time_ms 2 zk_p99_write_commitproc_time_ms 3 zk_p999_write_commitproc_time_ms 3 zk_avg_1_learner_handler_qp_size 0.001 zk_min_1_learner_handler_qp_size 0 zk_max_1_learner_handler_qp_size 2 zk_cnt_1_learner_handler_qp_size 62146 zk_sum_1_learner_handler_qp_size 64 zk_avg_3_learner_handler_qp_size 0.0641 zk_min_3_learner_handler_qp_size 0 zk_max_3_learner_handler_qp_size 88 zk_cnt_3_learner_handler_qp_size 61998 zk_sum_3_learner_handler_qp_size 3977 zk_avg_configs_read_per_namespace 3896.2239 zk_min_configs_read_per_namespace 84 zk_max_configs_read_per_namespace 70324 zk_cnt_configs_read_per_namespace 134 zk_sum_configs_read_per_namespace 522094 zk_avg_overseer_read_per_namespace 91.9 zk_min_overseer_read_per_namespace 83 zk_max_overseer_read_per_namespace 102 zk_cnt_overseer_read_per_namespace 10 zk_sum_overseer_read_per_namespace 919 zk_avg_aliases.json_read_per_namespace 81.0 zk_min_aliases.json_read_per_namespace 81 zk_max_aliases.json_read_per_namespace 81 zk_cnt_aliases.json_read_per_namespace 3 zk_sum_aliases.json_read_per_namespace 243 zk_avg_collections_read_per_namespace 119.9851 zk_min_collections_read_per_namespace 80 zk_max_collections_read_per_namespace 1326 zk_cnt_collections_read_per_namespace 5288 zk_sum_collections_read_per_namespace 634481 zk_avg_live_nodes_read_per_namespace 134.5556 zk_min_live_nodes_read_per_namespace 79 zk_max_live_nodes_read_per_namespace 154 zk_cnt_live_nodes_read_per_namespace 9 zk_sum_live_nodes_read_per_namespace 1211 zk_avg_overseer_elect_read_per_namespace 155.9591 zk_min_overseer_elect_read_per_namespace 83 zk_max_overseer_elect_read_per_namespace 262 zk_cnt_overseer_elect_read_per_namespace 928 zk_sum_overseer_elect_read_per_namespace 144730 zk_avg_security.json_read_per_namespace 82.5 zk_min_security.json_read_per_namespace 82 zk_max_security.json_read_per_namespace 84 zk_cnt_security.json_read_per_namespace 4 zk_sum_security.json_read_per_namespace 330 zk_avg_clusterstate.json_read_per_namespace 86.6667 zk_min_clusterstate.json_read_per_namespace 86 zk_max_clusterstate.json_read_per_namespace 88 zk_cnt_clusterstate.json_read_per_namespace 3 zk_sum_clusterstate.json_read_per_namespace 260 zk_avg_autoscaling_read_per_namespace 89.75 zk_min_autoscaling_read_per_namespace 87 zk_max_autoscaling_read_per_namespace 93 zk_cnt_autoscaling_read_per_namespace 4 zk_sum_autoscaling_read_per_namespace 359 zk_avg_autoscaling.json_read_per_namespace 857.0 zk_min_autoscaling.json_read_per_namespace 85 zk_max_autoscaling.json_read_per_namespace 1629 zk_cnt_autoscaling.json_read_per_namespace 2 zk_sum_autoscaling.json_read_per_namespace 1714 zk_avg_zookeeper_write_per_namespace 176.0 zk_min_zookeeper_write_per_namespace 176 zk_max_zookeeper_write_per_namespace 176 zk_cnt_zookeeper_write_per_namespace 2 zk_sum_zookeeper_write_per_namespace 352 zk_avg_overseer_write_per_namespace 144.4526 zk_min_overseer_write_per_namespace 29 zk_max_overseer_write_per_namespace 378 zk_cnt_overseer_write_per_namespace 95 zk_sum_overseer_write_per_namespace 13723 zk_avg_collections_write_per_namespace 651.7949 zk_min_collections_write_per_namespace 35 zk_max_collections_write_per_namespace 1258 zk_cnt_collections_write_per_namespace 117 zk_sum_collections_write_per_namespace 76260 zk_avg_overseer_elect_write_per_namespace 76.5 zk_min_overseer_elect_write_per_namespace 22 zk_max_overseer_elect_write_per_namespace 88 zk_cnt_overseer_elect_write_per_namespace 10 zk_sum_overseer_elect_write_per_namespace 765 zk_avg_live_nodes_write_per_namespace 37.0 zk_min_live_nodes_write_per_namespace 37 zk_max_live_nodes_write_per_namespace 37 zk_cnt_live_nodes_write_per_namespace 8 zk_sum_live_nodes_write_per_namespace 296 zk_avg_autoscaling_write_per_namespace 181.7692 zk_min_autoscaling_write_per_namespace 52 zk_max_autoscaling_write_per_namespace 379 zk_cnt_autoscaling_write_per_namespace 26 zk_sum_autoscaling_write_per_namespace 4726 zk_avg_1_learner_handler_qp_time_ms 0.3492 zk_min_1_learner_handler_qp_time_ms 0 zk_max_1_learner_handler_qp_time_ms 22 zk_cnt_1_learner_handler_qp_time_ms 63 zk_sum_1_learner_handler_qp_time_ms 22 zk_p50_1_learner_handler_qp_time_ms 0 zk_p95_1_learner_handler_qp_time_ms 0 zk_p99_1_learner_handler_qp_time_ms 22 zk_p999_1_learner_handler_qp_time_ms 22 zk_avg_3_learner_handler_qp_time_ms 0.0161 zk_min_3_learner_handler_qp_time_ms 0 zk_max_3_learner_handler_qp_time_ms 1 zk_cnt_3_learner_handler_qp_time_ms 62 zk_sum_3_learner_handler_qp_time_ms 1 zk_p50_3_learner_handler_qp_time_ms 0 zk_p95_3_learner_handler_qp_time_ms 0 zk_p99_3_learner_handler_qp_time_ms 1 zk_p999_3_learner_handler_qp_time_ms 1 {code} was (Author: bwahlen): You are right zk_followers is missing, but cluster is working @see zk_synced_followers {code:java} zk_version 3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on 04/21/2020 15:01 GMT zk_server_state leader zk_peer_state leading - broadcast zk_ephemerals_count 19 zk_num_alive_connections 6 zk_avg_latency 0.4362 zk_min_proposal_size 36 zk_outstanding_requests 0 zk_znode_count 138 zk_global_sessions 15 zk_last_client_response_size 16 zk_packets_sent 38309 zk_packets_received 38264 zk_max_proposal_size 1314 zk_max_client_response_size 70313 zk_synced_followers 2 zk_connection_drop_probability 0.0 zk_watch_count 25 zk_synced_non_voting_followers 0 zk_synced_observers 0 zk_min_latency 0 zk_max_file_descriptor_count 1048576 zk_approximate_data_size 308288 zk_open_file_descriptor_count 67 zk_last_proposal_size 36 zk_pending_syncs 0 zk_leader_uptime 61771446 zk_local_sessions 0 zk_uptime 61771582 zk_max_latency 2738 zk_outstanding_tls_handshake 0 zk_learners 2 zk_min_client_response_size 16 zk_quorum_size 3 zk_proposal_count 121 zk_outstanding_changes_removed 124 zk_stale_requests_dropped 0 zk_large_requests_rejected 0 zk_connection_rejected 0 zk_sessionless_connections_expired 0 zk_looking_count 2 zk_dead_watchers_queued 0 zk_stale_requests 0 zk_connection_drop_count 1 zk_learner_proposal_received_count 71 zk_digest_mismatches_count 0 zk_dead_watchers_cleared 0 zk_response_packet_cache_hits 1048 zk_bytes_received_count 701380 zk_add_dead_watcher_stall_time 0 zk_request_throttle_wait_count 0 zk_response_packet_cache_misses 47 zk_ensemble_auth_success 0 zk_prep_processor_request_queued 38168 zk_learner_commit_received_count 71 zk_stale_replies 0 zk_connection_request_count 7 zk_ensemble_auth_fail 0 zk_diff_count 2 zk_response_packet_get_children_cache_misses 0 zk_connection_revalidate_count 5 zk_quit_leading_due_to_disloyal_voter 0 zk_snap_count 0 zk_unrecoverable_error_count 0 zk_commit_count 121 zk_stale_sessions_expired 0 zk_response_packet_get_children_cache_hits 0 zk_sync_processor_request_queued 195 zk_outstanding_changes_queued 124 zk_request_commit_queued 195 zk_ensemble_auth_skip 0 zk_tls_handshake_exceeded 0 zk_revalidate_count 10 zk_avg_node_created_watch_count 0.0 zk_min_node_created_watch_count 0 zk_max_node_created_watch_count 0 zk_cnt_node_created_watch_count 0 zk_sum_node_created_watch_count 0 zk_avg_session_queues_drained 0.6205 zk_min_session_queues_drained 0 zk_max_session_queues_drained 1 zk_cnt_session_queues_drained 195 zk_sum_session_queues_drained 121 zk_avg_write_commit_proc_req_queued 0.0117 zk_min_write_commit_proc_req_queued 0 zk_max_write_commit_proc_req_queued 7 zk_cnt_write_commit_proc_req_queued 37821 zk_sum_write_commit_proc_req_queued 444 zk_avg_connection_token_deficit 0.0 zk_min_connection_token_deficit 0 zk_max_connection_token_deficit 0 zk_cnt_connection_token_deficit 7 zk_sum_connection_token_deficit 0 zk_avg_read_commit_proc_req_queued 1.0047 zk_min_read_commit_proc_req_queued 0 zk_max_read_commit_proc_req_queued 5 zk_cnt_read_commit_proc_req_queued 37821 zk_sum_read_commit_proc_req_queued 37998 zk_avg_node_deleted_watch_count 0.0 zk_min_node_deleted_watch_count 0 zk_max_node_deleted_watch_count 0 zk_cnt_node_deleted_watch_count 0 zk_sum_node_deleted_watch_count 0 zk_avg_startup_txns_load_time 19.0 zk_min_startup_txns_load_time 19 zk_max_startup_txns_load_time 19 zk_cnt_startup_txns_load_time 1 zk_sum_startup_txns_load_time 19 zk_avg_sync_processor_queue_size 0.132 zk_min_sync_processor_queue_size 0 zk_max_sync_processor_queue_size 3 zk_cnt_sync_processor_queue_size 197 zk_sum_sync_processor_queue_size 26 zk_avg_follower_sync_time 33.0 zk_min_follower_sync_time 33 zk_max_follower_sync_time 33 zk_cnt_follower_sync_time 1 zk_sum_follower_sync_time 33 zk_avg_prep_processor_queue_size 0.0103 zk_min_prep_processor_queue_size 0 zk_max_prep_processor_queue_size 4 zk_cnt_prep_processor_queue_size 38169 zk_sum_prep_processor_queue_size 393 zk_avg_fsynctime 0.052 zk_min_fsynctime 0 zk_max_fsynctime 4 zk_cnt_fsynctime 173 zk_sum_fsynctime 9 zk_avg_reads_issued_from_session_queue 0.0308 zk_min_reads_issued_from_session_queue 0 zk_max_reads_issued_from_session_queue 2 zk_cnt_reads_issued_from_session_queue 195 zk_sum_reads_issued_from_session_queue 6 zk_avg_snapshottime 6.0 zk_min_snapshottime 6 zk_max_snapshottime 6 zk_cnt_snapshottime 1 zk_sum_snapshottime 6 zk_avg_startup_txns_loaded 137.0 zk_min_startup_txns_loaded 137 zk_max_startup_txns_loaded 137 zk_cnt_startup_txns_loaded 1 zk_sum_startup_txns_loaded 137 zk_avg_reads_after_write_in_session_queue 0.0496 zk_min_reads_after_write_in_session_queue 0 zk_max_reads_after_write_in_session_queue 2 zk_cnt_reads_after_write_in_session_queue 121 zk_sum_reads_after_write_in_session_queue 6 zk_avg_requests_in_session_queue 1.9395 zk_min_requests_in_session_queue 1 zk_max_requests_in_session_queue 7 zk_cnt_requests_in_session_queue 248 zk_sum_requests_in_session_queue 481 zk_avg_write_commit_proc_issued 1.0 zk_min_write_commit_proc_issued 1 zk_max_write_commit_proc_issued 1 zk_cnt_write_commit_proc_issued 195 zk_sum_write_commit_proc_issued 195 zk_avg_prep_process_time 0.0047 zk_min_prep_process_time 0 zk_max_prep_process_time 12 zk_cnt_prep_process_time 38168 zk_sum_prep_process_time 179 zk_avg_pending_session_queue_size 1.0083 zk_min_pending_session_queue_size 1 zk_max_pending_session_queue_size 2 zk_cnt_pending_session_queue_size 121 zk_sum_pending_session_queue_size 122 zk_avg_time_waiting_empty_pool_in_commit_processor_read_ms 0.0 zk_min_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_max_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_cnt_time_waiting_empty_pool_in_commit_processor_read_ms 195 zk_sum_time_waiting_empty_pool_in_commit_processor_read_ms 0 zk_avg_commit_process_time 0.0169 zk_min_commit_process_time 0 zk_max_commit_process_time 4 zk_cnt_commit_process_time 37821 zk_sum_commit_process_time 639 zk_avg_dbinittime 39.0 zk_min_dbinittime 39 zk_max_dbinittime 39 zk_cnt_dbinittime 1 zk_sum_dbinittime 39 zk_avg_netty_queued_buffer_capacity 0.0 zk_min_netty_queued_buffer_capacity 0 zk_max_netty_queued_buffer_capacity 0 zk_cnt_netty_queued_buffer_capacity 0 zk_sum_netty_queued_buffer_capacity 0 zk_avg_election_time 125.5 zk_min_election_time 33 zk_max_election_time 218 zk_cnt_election_time 2 zk_sum_election_time 251 zk_avg_commit_commit_proc_req_queued 0.0054 zk_min_commit_commit_proc_req_queued 0 zk_max_commit_commit_proc_req_queued 4 zk_cnt_commit_commit_proc_req_queued 37821 zk_sum_commit_commit_proc_req_queued 204 zk_avg_sync_processor_batch_size 1.1272 zk_min_sync_processor_batch_size 1 zk_max_sync_processor_batch_size 5 zk_cnt_sync_processor_batch_size 173 zk_sum_sync_processor_batch_size 195 zk_avg_node_children_watch_count 6.0 zk_min_node_children_watch_count 6 zk_max_node_children_watch_count 6 zk_cnt_node_children_watch_count 1 zk_sum_node_children_watch_count 6 zk_avg_write_batch_time_in_commit_processor 0.2051 zk_min_write_batch_time_in_commit_processor 0 zk_max_write_batch_time_in_commit_processor 4 zk_cnt_write_batch_time_in_commit_processor 195 zk_sum_write_batch_time_in_commit_processor 40 zk_avg_read_commit_proc_issued 1.0041 zk_min_read_commit_proc_issued 0 zk_max_read_commit_proc_issued 5 zk_cnt_read_commit_proc_issued 37821 zk_sum_read_commit_proc_issued 37975 zk_avg_concurrent_request_processing_in_commit_processor 2.0 zk_min_concurrent_request_processing_in_commit_processor 2 zk_max_concurrent_request_processing_in_commit_processor 2 zk_cnt_concurrent_request_processing_in_commit_processor 1 zk_sum_concurrent_request_processing_in_commit_processor 2 zk_avg_node_changed_watch_count 1.0 zk_min_node_changed_watch_count 1 zk_max_node_changed_watch_count 1 zk_cnt_node_changed_watch_count 16 zk_sum_node_changed_watch_count 16 zk_avg_sync_process_time 0.1077 zk_min_sync_process_time 0 zk_max_sync_process_time 2 zk_cnt_sync_process_time 195 zk_sum_sync_process_time 21 zk_avg_startup_snap_load_time 9.0 zk_min_startup_snap_load_time 9 zk_max_startup_snap_load_time 9 zk_cnt_startup_snap_load_time 1 zk_sum_startup_snap_load_time 9 zk_avg_prep_processor_queue_time_ms 0.0718 zk_min_prep_processor_queue_time_ms 0 zk_max_prep_processor_queue_time_ms 12 zk_cnt_prep_processor_queue_time_ms 38168 zk_sum_prep_processor_queue_time_ms 2742 zk_p50_prep_processor_queue_time_ms 0 zk_p95_prep_processor_queue_time_ms 1 zk_p99_prep_processor_queue_time_ms 1 zk_p999_prep_processor_queue_time_ms 1 zk_avg_close_session_prep_time 0.0 zk_min_close_session_prep_time 0 zk_max_close_session_prep_time 0 zk_cnt_close_session_prep_time 1 zk_sum_close_session_prep_time 0 zk_p50_close_session_prep_time 0 zk_p95_close_session_prep_time 0 zk_p99_close_session_prep_time 0 zk_p999_close_session_prep_time 0 zk_avg_read_commitproc_time_ms 0.2513 zk_min_read_commitproc_time_ms 0 zk_max_read_commitproc_time_ms 2738 zk_cnt_read_commitproc_time_ms 37981 zk_sum_read_commitproc_time_ms 9543 zk_p50_read_commitproc_time_ms 0 zk_p95_read_commitproc_time_ms 1 zk_p99_read_commitproc_time_ms 1 zk_p999_read_commitproc_time_ms 1 zk_avg_updatelatency 1.5143 zk_min_updatelatency 0 zk_max_updatelatency 5 zk_cnt_updatelatency 35 zk_sum_updatelatency 53 zk_p50_updatelatency 1 zk_p95_updatelatency 3 zk_p99_updatelatency 5 zk_p999_updatelatency 5 zk_avg_local_write_committed_time_ms 0.0579 zk_min_local_write_committed_time_ms 0 zk_max_local_write_committed_time_ms 1 zk_cnt_local_write_committed_time_ms 121 zk_sum_local_write_committed_time_ms 7 zk_p50_local_write_committed_time_ms 0 zk_p95_local_write_committed_time_ms 1 zk_p99_local_write_committed_time_ms 1 zk_p999_local_write_committed_time_ms 1 zk_avg_readlatency 0.4352 zk_min_readlatency 0 zk_max_readlatency 2738 zk_cnt_readlatency 37981 zk_sum_readlatency 16528 zk_p50_readlatency 0 zk_p95_readlatency 1 zk_p99_readlatency 1 zk_p999_readlatency 1 zk_avg_quorum_ack_latency 1.0909 zk_min_quorum_ack_latency 0 zk_max_quorum_ack_latency 7 zk_cnt_quorum_ack_latency 121 zk_sum_quorum_ack_latency 132 zk_p50_quorum_ack_latency 1 zk_p95_quorum_ack_latency 2 zk_p99_quorum_ack_latency 6 zk_p999_quorum_ack_latency 7 zk_avg_om_commit_process_time_ms 0.0 zk_min_om_commit_process_time_ms 0 zk_max_om_commit_process_time_ms 0 zk_cnt_om_commit_process_time_ms 0 zk_sum_om_commit_process_time_ms 0 zk_p50_om_commit_process_time_ms 0 zk_p95_om_commit_process_time_ms 0 zk_p99_om_commit_process_time_ms 0 zk_p999_om_commit_process_time_ms 0 zk_avg_read_final_proc_time_ms 0.0255 zk_min_read_final_proc_time_ms 0 zk_max_read_final_proc_time_ms 7 zk_cnt_read_final_proc_time_ms 37981 zk_sum_read_final_proc_time_ms 968 zk_p50_read_final_proc_time_ms 0 zk_p95_read_final_proc_time_ms 0 zk_p99_read_final_proc_time_ms 1 zk_p999_read_final_proc_time_ms 1 zk_avg_commit_propagation_latency 5965.7297 zk_min_commit_propagation_latency 2 zk_max_commit_propagation_latency 147105 zk_cnt_commit_propagation_latency 74 zk_sum_commit_propagation_latency 441464 zk_p50_commit_propagation_latency 3 zk_p95_commit_propagation_latency 36772 zk_p99_commit_propagation_latency 147105 zk_p999_commit_propagation_latency 147105 zk_avg_dead_watchers_cleaner_latency 0.0 zk_min_dead_watchers_cleaner_latency 0 zk_max_dead_watchers_cleaner_latency 0 zk_cnt_dead_watchers_cleaner_latency 0 zk_sum_dead_watchers_cleaner_latency 0 zk_p50_dead_watchers_cleaner_latency 0 zk_p95_dead_watchers_cleaner_latency 0 zk_p99_dead_watchers_cleaner_latency 0 zk_p999_dead_watchers_cleaner_latency 0 zk_avg_write_final_proc_time_ms 0.2 zk_min_write_final_proc_time_ms 0 zk_max_write_final_proc_time_ms 4 zk_cnt_write_final_proc_time_ms 195 zk_sum_write_final_proc_time_ms 39 zk_p50_write_final_proc_time_ms 0 zk_p95_write_final_proc_time_ms 1 zk_p99_write_final_proc_time_ms 4 zk_p999_write_final_proc_time_ms 4 zk_avg_proposal_ack_creation_latency 2264.2769 zk_min_proposal_ack_creation_latency 0 zk_max_proposal_ack_creation_latency 147107 zk_cnt_proposal_ack_creation_latency 195 zk_sum_proposal_ack_creation_latency 441534 zk_p50_proposal_ack_creation_latency 1 zk_p95_proposal_ack_creation_latency 3 zk_p99_proposal_ack_creation_latency 147104 zk_p999_proposal_ack_creation_latency 147107 zk_avg_proposal_latency 1.9014 zk_min_proposal_latency 1 zk_max_proposal_latency 3 zk_cnt_proposal_latency 71 zk_sum_proposal_latency 135 zk_p50_proposal_latency 2 zk_p95_proposal_latency 2 zk_p99_proposal_latency 3 zk_p999_proposal_latency 3 zk_avg_om_proposal_process_time_ms 0.0 zk_min_om_proposal_process_time_ms 0 zk_max_om_proposal_process_time_ms 0 zk_cnt_om_proposal_process_time_ms 0 zk_sum_om_proposal_process_time_ms 0 zk_p50_om_proposal_process_time_ms 0 zk_p95_om_proposal_process_time_ms 0 zk_p99_om_proposal_process_time_ms 0 zk_p999_om_proposal_process_time_ms 0 zk_avg_sync_processor_queue_and_flush_time_ms 0.3846 zk_min_sync_processor_queue_and_flush_time_ms 0 zk_max_sync_processor_queue_and_flush_time_ms 4 zk_cnt_sync_processor_queue_and_flush_time_ms 195 zk_sum_sync_processor_queue_and_flush_time_ms 75 zk_p50_sync_processor_queue_and_flush_time_ms 0 zk_p95_sync_processor_queue_and_flush_time_ms 1 zk_p99_sync_processor_queue_and_flush_time_ms 4 zk_p999_sync_processor_queue_and_flush_time_ms 4 zk_avg_propagation_latency 2264.8564 zk_min_propagation_latency 0 zk_max_propagation_latency 147106 zk_cnt_propagation_latency 195 zk_sum_propagation_latency 441647 zk_p50_propagation_latency 2 zk_p95_propagation_latency 4 zk_p99_propagation_latency 147103 zk_p999_propagation_latency 147106 zk_avg_server_write_committed_time_ms 0.0676 zk_min_server_write_committed_time_ms 0 zk_max_server_write_committed_time_ms 1 zk_cnt_server_write_committed_time_ms 74 zk_sum_server_write_committed_time_ms 5 zk_p50_server_write_committed_time_ms 0 zk_p95_server_write_committed_time_ms 1 zk_p99_server_write_committed_time_ms 1 zk_p999_server_write_committed_time_ms 1 zk_avg_sync_processor_queue_time_ms 0.0923 zk_min_sync_processor_queue_time_ms 0 zk_max_sync_processor_queue_time_ms 2 zk_cnt_sync_processor_queue_time_ms 195 zk_sum_sync_processor_queue_time_ms 18 zk_p50_sync_processor_queue_time_ms 0 zk_p95_sync_processor_queue_time_ms 1 zk_p99_sync_processor_queue_time_ms 2 zk_p999_sync_processor_queue_time_ms 2 zk_avg_sync_processor_queue_flush_time_ms 0.1792 zk_min_sync_processor_queue_flush_time_ms 0 zk_max_sync_processor_queue_flush_time_ms 4 zk_cnt_sync_processor_queue_flush_time_ms 173 zk_sum_sync_processor_queue_flush_time_ms 31 zk_p50_sync_processor_queue_flush_time_ms 0 zk_p95_sync_processor_queue_flush_time_ms 1 zk_p99_sync_processor_queue_flush_time_ms 3 zk_p999_sync_processor_queue_flush_time_ms 4 zk_avg_write_commitproc_time_ms 0.9008 zk_min_write_commitproc_time_ms 0 zk_max_write_commitproc_time_ms 3 zk_cnt_write_commitproc_time_ms 121 zk_sum_write_commitproc_time_ms 109 zk_p50_write_commitproc_time_ms 1 zk_p95_write_commitproc_time_ms 2 zk_p99_write_commitproc_time_ms 3 zk_p999_write_commitproc_time_ms 3 zk_avg_1_learner_handler_qp_size 0.001 zk_min_1_learner_handler_qp_size 0 zk_max_1_learner_handler_qp_size 2 zk_cnt_1_learner_handler_qp_size 62146 zk_sum_1_learner_handler_qp_size 64 zk_avg_3_learner_handler_qp_size 0.0641 zk_min_3_learner_handler_qp_size 0 zk_max_3_learner_handler_qp_size 88 zk_cnt_3_learner_handler_qp_size 61998 zk_sum_3_learner_handler_qp_size 3977 zk_avg_configs_read_per_namespace 3896.2239 zk_min_configs_read_per_namespace 84 zk_max_configs_read_per_namespace 70324 zk_cnt_configs_read_per_namespace 134 zk_sum_configs_read_per_namespace 522094 zk_avg_overseer_read_per_namespace 91.9 zk_min_overseer_read_per_namespace 83 zk_max_overseer_read_per_namespace 102 zk_cnt_overseer_read_per_namespace 10 zk_sum_overseer_read_per_namespace 919 zk_avg_aliases.json_read_per_namespace 81.0 zk_min_aliases.json_read_per_namespace 81 zk_max_aliases.json_read_per_namespace 81 zk_cnt_aliases.json_read_per_namespace 3 zk_sum_aliases.json_read_per_namespace 243 zk_avg_collections_read_per_namespace 119.9851 zk_min_collections_read_per_namespace 80 zk_max_collections_read_per_namespace 1326 zk_cnt_collections_read_per_namespace 5288 zk_sum_collections_read_per_namespace 634481 zk_avg_live_nodes_read_per_namespace 134.5556 zk_min_live_nodes_read_per_namespace 79 zk_max_live_nodes_read_per_namespace 154 zk_cnt_live_nodes_read_per_namespace 9 zk_sum_live_nodes_read_per_namespace 1211 zk_avg_overseer_elect_read_per_namespace 155.9591 zk_min_overseer_elect_read_per_namespace 83 zk_max_overseer_elect_read_per_namespace 262 zk_cnt_overseer_elect_read_per_namespace 928 zk_sum_overseer_elect_read_per_namespace 144730 zk_avg_security.json_read_per_namespace 82.5 zk_min_security.json_read_per_namespace 82 zk_max_security.json_read_per_namespace 84 zk_cnt_security.json_read_per_namespace 4 zk_sum_security.json_read_per_namespace 330 zk_avg_clusterstate.json_read_per_namespace 86.6667 zk_min_clusterstate.json_read_per_namespace 86 zk_max_clusterstate.json_read_per_namespace 88 zk_cnt_clusterstate.json_read_per_namespace 3 zk_sum_clusterstate.json_read_per_namespace 260 zk_avg_autoscaling_read_per_namespace 89.75 zk_min_autoscaling_read_per_namespace 87 zk_max_autoscaling_read_per_namespace 93 zk_cnt_autoscaling_read_per_namespace 4 zk_sum_autoscaling_read_per_namespace 359 zk_avg_autoscaling.json_read_per_namespace 857.0 zk_min_autoscaling.json_read_per_namespace 85 zk_max_autoscaling.json_read_per_namespace 1629 zk_cnt_autoscaling.json_read_per_namespace 2 zk_sum_autoscaling.json_read_per_namespace 1714 zk_avg_zookeeper_write_per_namespace 176.0 zk_min_zookeeper_write_per_namespace 176 zk_max_zookeeper_write_per_namespace 176 zk_cnt_zookeeper_write_per_namespace 2 zk_sum_zookeeper_write_per_namespace 352 zk_avg_overseer_write_per_namespace 144.4526 zk_min_overseer_write_per_namespace 29 zk_max_overseer_write_per_namespace 378 zk_cnt_overseer_write_per_namespace 95 zk_sum_overseer_write_per_namespace 13723 zk_avg_collections_write_per_namespace 651.7949 zk_min_collections_write_per_namespace 35 zk_max_collections_write_per_namespace 1258 zk_cnt_collections_write_per_namespace 117 zk_sum_collections_write_per_namespace 76260 zk_avg_overseer_elect_write_per_namespace 76.5 zk_min_overseer_elect_write_per_namespace 22 zk_max_overseer_elect_write_per_namespace 88 zk_cnt_overseer_elect_write_per_namespace 10 zk_sum_overseer_elect_write_per_namespace 765 zk_avg_live_nodes_write_per_namespace 37.0 zk_min_live_nodes_write_per_namespace 37 zk_max_live_nodes_write_per_namespace 37 zk_cnt_live_nodes_write_per_namespace 8 zk_sum_live_nodes_write_per_namespace 296 zk_avg_autoscaling_write_per_namespace 181.7692 zk_min_autoscaling_write_per_namespace 52 zk_max_autoscaling_write_per_namespace 379 zk_cnt_autoscaling_write_per_namespace 26 zk_sum_autoscaling_write_per_namespace 4726 zk_avg_1_learner_handler_qp_time_ms 0.3492 zk_min_1_learner_handler_qp_time_ms 0 zk_max_1_learner_handler_qp_time_ms 22 zk_cnt_1_learner_handler_qp_time_ms 63 zk_sum_1_learner_handler_qp_time_ms 22 zk_p50_1_learner_handler_qp_time_ms 0 zk_p95_1_learner_handler_qp_time_ms 0 zk_p99_1_learner_handler_qp_time_ms 22 zk_p999_1_learner_handler_qp_time_ms 22 zk_avg_3_learner_handler_qp_time_ms 0.0161 zk_min_3_learner_handler_qp_time_ms 0 zk_max_3_learner_handler_qp_time_ms 1 zk_cnt_3_learner_handler_qp_time_ms 62 zk_sum_3_learner_handler_qp_time_ms 1 zk_p50_3_learner_handler_qp_time_ms 0 zk_p95_3_learner_handler_qp_time_ms 0 zk_p99_3_learner_handler_qp_time_ms 1 zk_p999_3_learner_handler_qp_time_ms 1 {code} > solr admin ui: zkstatus: For input string: "null" with zk 3.6.x > --------------------------------------------------------------- > > Key: SOLR-14463 > URL: https://issues.apache.org/jira/browse/SOLR-14463 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Affects Versions: 8.5.1 > Reporter: Bernd Wahlen > Assignee: Jan Høydahl > Priority: Major > > When i select in solr 8.5.1 webinterface: Cloud - ZK Status i got > For input string: "null" > This happens after upgrading the leader to zookeeper 3.6.1 (upgrading the > followers before was not a problem). From my view configuration is running, > just the status web page doesn't. > I remember that there was similar problem before with solr/zk version > incompatibilities. From zookeeper documentation 3.6.1 should be fully > compatible with 3.5 clients. > Annoyingly there is no official zk 3.6.1 docker container available, but i > think it is easy to reproduce. > and from solr.log file: > {code:java} > 2020-05-07 15:58:33.194 ERROR (qtp1940055334-231) [ ] > o.a.s.h.RequestHandlerBase java.lang.NumberFormatException: For input string: > "null" > at > java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.base/java.lang.Integer.parseInt(Integer.java:652) > at java.base/java.lang.Integer.parseInt(Integer.java:770) > at > org.apache.solr.handler.admin.ZookeeperStatusHandler.getZkStatus(ZookeeperStatusHandler.java:116) > at > org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:78) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) > at > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842) > at > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) > at > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at > org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.base/java.lang.Thread.run(Thread.java:834) > {code} > Others have the same isues: > https://www.reddit.com/r/Solr/comments/fld4l6/issue_with_zk_status_page/ -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org