pabrahamusa opened a new issue #6449: URL: https://github.com/apache/incubator-pinot/issues/6449
My EKS node running pinot server ran out of map areas and I bumped it to 262144 from default. When the error came pinot stopped consuming segments. I updated the max map in os and restarted the server. After starting the realtime table was showing Bad status. However it recovered eventually. I am seeing following message after that and wondering is this due to some hidden problem somewhere in pinot server ? ``` my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/16 18:50:29.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(f307bcc1_DEFAULT)] Event f307bcc1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. ``` ``` my-pinot-server-1 server 2021/01/16 18:50:51.938 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__162__20210112T1932Z my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__163__20210112T1935Z my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__164__20210112T1937Z my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__165__20210112T1940Z my-pinot-server-1 server 2021/01/16 18:50:51.939 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__166__20210112T1944Z my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__167__20210112T1947Z my-pinot-server-1 server 2021/01/16 18:50:51.940 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__168__20210112T1950Z my-pinot-server-1 server 2021/01/16 18:50:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__169__20210112T1953Z n config. ``` ``` my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.autotune.initialRows' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'streamType' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.zk.broker.url' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config. my-pinot-server-1 server 2021/01/16 19:01:16.722 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config. ``` ``` my-pinot-server-1 server 2021-01-16T19:01:25.952+0000: 672.343: Total time for which application threads were stopped: 0.0003745 seconds, Stopping threads took: 0.0000540 seconds my-pinot-server-1 server 2021-01-16T19:01:26.158+0000: 672.549: Application time: 0.2053891 seconds my-pinot-server-1 server 2021-01-16T19:01:26.159+0000: 672.550: Total time for which application threads were stopped: 0.0015166 seconds, Stopping threads took: 0.0000398 seconds my-pinot-server-1 server 2021-01-16T19:01:26.227+0000: 672.618: Application time: 0.0673477 seconds my-pinot-server-1 server 2021-01-16T19:01:26.228+0000: 672.619: Total time for which application threads were stopped: 0.0015950 seconds, Stopping threads took: 0.0000456 seconds my-pinot-server-1 server 2021-01-16T19:01:26.242+0000: 672.633: Application time: 0.0142821 seconds my-pinot-server-1 server 2021-01-16T19:01:26.243+0000: 672.634: Total time for which application threads were stopped: 0.0003874 seconds, Stopping threads took: 0.0000330 seconds my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Application time: 0.0067833 seconds my-pinot-server-1 server 2021-01-16T19:01:26.250+0000: 672.641: Total time for which application threads were stopped: 0.0003493 seconds, Stopping threads took: 0.0000236 seconds my-pinot-server-1 server 2021-01-16T19:01:26.251+0000: 672.642: Application time: 0.0010896 seconds my-pinot-server-1 server 2021-01-16T19:01:26.253+0000: 672.643: Total time for which application threads were stopped: 0.0014404 seconds, Stopping threads took: 0.0000105 seconds my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Application time: 1.0000987 seconds my-pinot-server-1 server 2021-01-16T19:01:27.253+0000: 673.644: Total time for which application threads were stopped: 0.0004062 seconds, Stopping threads took: 0.0000532 seconds my-pinot-server-1 server 2021-01-16T19:01:30.253+0000: 676.644: Application time: 3.0002441 seconds my-pinot-server-1 server 2021-01-16T19:01:30.254+0000: 676.645: Total time for which application threads were stopped: 0.0003946 seconds, Stopping threads took: 0.0000391 seconds my-pinot-server-1 server 2021-01-16T19:02:17.596+0000: 723.987: Application time: 47.3426464 seconds my-pinot-server-1 server 2021-01-16T19:02:17.597+0000: 723.988: Total time for which application threads were stopped: 0.0004295 seconds, Stopping threads took: 0.0000559 seconds ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
