pabrahamusa commented on issue #6449: URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
@fx19880617 Thanks for the details. I just keep monitoring the system and the memory for the System was going through the edge. I noticed that once it happens and If i leave it for say couple of days following errors will appear and Pinot server will stop ingestion completely. Is there any way to fix this issue?  ``` sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.5, Avg: 1.6, Max: 1.8, Diff: 0.3, Sum: 9.4] 2021/01/21 17:02:38.123 ERROR [ServerSegmentCompletionProtocolHandler] [sblog__0__32545__20210121T1702Z] Could not send request http://sb-pinot-controller-0.sb-pinot-controller-headless.logging.svc.cluster.local:9000/segmentCommit?name=sblog__0__32545__20210121T1702Z&offset=44504082&instance=Server_sb-pinot-server-1.sb-pinot-server-headless.logging.svc.cluster.local_8098&buildTimeMillis=191&memoryUsedBytes=256504&segmentSizeBytes=2763568&rowCount=1000&streamPartitionMsgOffset=44504082 sb-pinot-server-1 sb-pinot-server-0server server [Update RS (ms): Min: 5.6, Avg: 5.7, Max: 5.7, Diff: 0.1, Sum: 34.1] sb-pinot-controller-1 shaded.org.apache.http.NoHttpResponseException: sb-pinot-controller-0.sb-pinot-controller-headless.logging.svc.cluster.local:9000 failed to respond sb-pinot-server-1sb-pinot-server-0 sb-pinot-broker-0 server [Processed Buffers: Min: 17, Avg: 27.3, Max: 34, Diff: 17, Sum: 164] at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-a36d1c04c720c0f47dfec078c24c84c98ed665c4] sb-pinot-broker-0 sb-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. broker [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-1 sb-pinot-broker-0 server broker [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 2] sb-pinot-broker-0 sb-pinot-controller-1 server [Choose CSet: 0.0 ms] sb-pinot-server-0 broker controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. sb-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. ... 16 more sb-pinot-controller-1server [Ref Proc: 3.4 ms] controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-server-1 server 2021/01/21 17:04:50.874 ERROR [LLRealtimeSegmentDataManager_sblog__0__32545__20210121T1702Z] [sblog__0__32545__20210121T1702Z] Holding after response from Controller: {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"NOT_SENT","offset":-1} sb-pinot-server-1 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. server sb-pinot-server-0 server [Ref Enq: 0.0 ms] sb-pinot-server-1 controller 2021-01-21T17:25:03.929+0000: 1350.323: Application time: 0.0590331 seconds sb-pinot-server-0 sb-pinot-controller-1 server controller 2021-01-21T17:25:03.930+0000: 1350.323: [GC pause (G1 Evacuation Pause) (mixed), 0.0406644 secs] server [Times: user=0.26 sys=0.00, real=0.05 secs] sb-pinot-server-0 2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29726__20210121T1102Z server 2021-01-21T17:25:05.778+0000: 166.983: Total time for which application threads were stopped: 0.0500843 seconds, Stopping threads took: 0.0000282 seconds sb-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: [GC concurrent-mark-end, 1.4470120 secs] sb-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: Application time: 0.4400702 seconds sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29727__20210121T1102Z sb-pinot-server-1 server 2021-01-21T17:25:06.219+0000: 167.424: [GC remark 2021-01-21T17:25:06.219+0000: 167.424: [Finalize Marking, 0.0002085 secs] 2021-01-21T17:25:06.219+0000: 167.424: [GC ref-proc, 0.0851322 secs] 2021-01-21T17:25:06.304+0000: 167.509: [Unloading, 0.0080895 secs], 0.0963586 secs] 2021/01/21 17:22:51.941 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29728__20210121T1102Z sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29729__20210121T1102Z sb-pinot-server-1sb-pinot-server-0 server sb-pinot-controller-1 [Times: user=0.11 sys=0.04, real=0.10 secs] sb-pinot-server-0server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29730__20210121T1102Z sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29731__20210121T1102Z sb-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29732__20210121T1102Z controller [Parallel Time: 39.8 ms, GC Workers: 4] sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350323.3, Avg: 1350323.3, Max: 1350323.3, Diff: 0.0] server 2021-01-21T17:25:06.315+0000: 167.520: Total time for which application threads were stopped: 0.0972538 seconds, Stopping threads took: 0.0003432 seconds sb-pinot-server-0 server sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 0.9, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.0] sb-pinot-controller-1 sb-pinot-server-1 2021-01-21T17:25:06.315+0000: 167.520: Application time: 0.0001148 seconds servercontroller [Update RS (ms): Min: 15.6, Avg: 15.6, Max: 15.6, Diff: 0.0, Sum: 62.4] sb-pinot-controller-1 sb-pinot-server-0 servercontroller [Processed Buffers: Min: 76, Avg: 77.8, Max: 79, Diff: 3, Sum: 311] sb-pinot-controller-1 controller [Scan RS (ms): Min: 1.7, Avg: 1.7, Max: 1.7, Diff: 0.1, Sum: 6.8] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 2021-01-21T17:25:06.316+0000: 167.521: [GC cleanup 3168M->3047M(4096M), 0.0040580 secs] sb-pinot-server-0 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29733__20210121T1102Z sb-pinot-server-1server [Times: user=0.02 sys=0.00, real=0.00 secs] sb-pinot-server-0 server server 2021/01/21 17:22:51.942 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29734__20210121T1103Z controller [Object Copy (ms): Min: 21.3, Avg: 21.4, Max: 21.5, Diff: 0.2, Sum: 85.6] sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29735__20210121T1103Z sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29736__20210121T1103Z controller2021-01-21T17:25:06.320+0000: 167.525: Total time for which application threads were stopped: 0.0046889 seconds, Stopping threads took: 0.0001052 seconds sb-pinot-server-0 sb-pinot-server-1server 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-start] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-controller-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29737__20210121T1103Z controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29738__20210121T1103Z sb-pinot-controller-1 server sb-pinot-server-1 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-end, 0.0001522 secs] sb-pinot-server-0server server controller [GC Worker Total (ms): Min: 39.7, Avg: 39.7, Max: 39.7, Diff: 0.0, Sum: 158.9] 2021/01/21 17:22:51.943 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29739__20210121T1103Z sb-pinot-controller-1 2021-01-21T17:25:06.929+0000: 168.134: Application time: 0.6090475 seconds sb-pinot-server-0 sb-pinot-server-1 server controller [GC Worker End (ms): Min: 1350363.0, Avg: 1350363.0, Max: 1350363.0, Diff: 0.0] sb-pinot-controller-1 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29740__20210121T1103Z sb-pinot-server-1 serverserver 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29741__20210121T1103Z sb-pinot-server-1 controller [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controller 2021-01-21T17:25:06.930+0000: 168.135: [GC pause (G1 Evacuation Pause) (young), 0.0357839 secs] sb-pinot-server-0 server [Parallel Time: 32.1 ms, GC Workers: 6] sb-pinot-server-0 server [GC Worker Start (ms): Min: 168134.9, Avg: 168134.9, Max: 168134.9, Diff: 0.1] [Code Root Purge: 0.0 ms] sb-pinot-controller-1sb-pinot-server-0 server controller [Clear CT: 0.1 ms] sb-pinot-controller-1 controller [Other: 0.8 ms] sb-pinot-controller-1 controller server 2021/01/21 17:22:51.944 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29742__20210121T1104Z sb-pinot-server-1 [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.5, Diff: 0.4, Sum: 7.9] sb-pinot-server-0 server [Choose CSet: 0.1 ms] sb-pinot-controller-1 controller [Ref Proc: 0.2 ms] sb-pinot-controller-1 controller [Ref Enq: 0.0 ms] sb-pinot-controller-1 controller server [Update RS (ms): Min: 14.8, Avg: 14.9, Max: 15.1, Diff: 0.3, Sum: 89.2] sb-pinot-server-0 server [Processed Buffers: Min: 107, Avg: 122.2, Max: 132, Diff: 25, Sum: 733] sb-pinot-server-0 server [Scan RS (ms): Min: 1.9, Avg: 2.1, Max: 2.2, Diff: 0.3, Sum: 12.5] sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Redirty Cards: 0.2 ms] sb-pinot-controller-1 controller 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29743__20210121T1104Z sb-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29744__20210121T1104Z [Humongous Register: 0.0 ms] sb-pinot-controller-1 sb-pinot-server-0 server [Object Copy (ms): Min: 13.5, Avg: 13.7, Max: 13.8, Diff: 0.3, Sum: 82.1] sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] controller [Humongous Reclaim: 0.0 ms] sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29745__20210121T1104Z sb-pinot-server-0 sb-pinot-server-1 [Free CSet: 0.1 ms] sb-pinot-controller-1 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.3] serversb-pinot-server-0controller server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29746__20210121T1104Z sb-pinot-server-1 [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 466.6M(1024.0M)->412.2M(1024.0M)] sb-pinot-controller-1 server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29747__20210121T1104Z sb-pinot-server-1 server 2021/01/21 17:22:51.946 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29748__20210121T1105Z [GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.1, Diff: 0.1, Sum: 192.1] sb-pinot-server-1 sb-pinot-server-0 server 2021/01/21 17:22:51.947 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29749__20210121T1105Z sb-pinot-server-1 server server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29750__20210121T1105Z controller [Times: user=0.16 sys=0.00, real=0.04 secs] [GC Worker End (ms): Min: 168166.9, Avg: 168166.9, Max: 168166.9, Diff: 0.1] sb-pinot-controller-1 sb-pinot-server-1 controller 2021-01-21T17:25:03.971+0000: 1350.364: Total time for which application threads were stopped: 0.0412316 seconds, Stopping threads took: 0.0000680 seconds sb-pinot-controller-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29751__20210121T1105Z sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29752__20210121T1105Z sb-pinot-server-1 serversb-pinot-server-0 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29753__20210121T1105Z server [Code Root Fixup: 0.0 ms] sb-pinot-server-0 controller 2021-01-21T17:25:04.025+0000: 1350.418: Application time: 0.0544185 seconds sb-pinot-controller-1 controllerserver 2021-01-21T17:25:04.025+0000: 1350.419: [GC pause (G1 Evacuation Pause) (mixed), 0.0437685 secs] [Code Root Purge: 0.0 ms] sb-pinot-server-0 server [Clear CT: 0.5 ms] sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29754__20210121T1105Z sb-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29755__20210121T1105Z sb-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29756__20210121T1106Z sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29757__20210121T1106Z server [Other: 3.2 ms] sb-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29758__20210121T1106Z sb-pinot-server-1 sb-pinot-server-0 controller server [Choose CSet: 0.0 ms] [Parallel Time: 43.0 ms, GC Workers: 4] sb-pinot-server-0 server [Ref Proc: 1.6 ms] serversb-pinot-server-0 server [Ref Enq: 0.0 ms] 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29759__20210121T1106Z sb-pinot-server-1 sb-pinot-server-0 serverserver [Redirty Cards: 0.1 ms] sb-pinot-controller-1 sb-pinot-server-0 controller 2021/01/21 17:22:51.949 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29760__20210121T1106Z sb-pinot-server-1 server 2021/01/21 17:22:51.950 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29761__20210121T1106Z sb-pinot-server-1 [GC Worker Start (ms): Min: 1350418.9, Avg: 1350418.9, Max: 1350418.9, Diff: 0.0] sb-pinot-controller-1 server controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.1] sb-pinot-controller-1 controller [Update RS (ms): Min: 23.6, Avg: 23.6, Max: 23.6, Diff: 0.0, Sum: 94.5] sb-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 106.8, Max: 109, Diff: 6, Sum: 427] server2021/01/21 17:22:51.950 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29762__20210121T1106Z sb-pinot-server-1 server 2021/01/21 17:22:51.951 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29763__20210121T1107Z sb-pinot-server-1 [Humongous Register: 0.2 ms] server 2021/01/21 17:22:51.951 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29764__20210121T1107Z sb-pinot-server-1 sb-pinot-controller-1 server controller [Scan RS (ms): Min: 0.2, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.8] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29765__20210121T1107Z controller sb-pinot-server-0 sb-pinot-server-1 server server [Humongous Reclaim: 0.3 ms] [Object Copy (ms): Min: 18.0, Avg: 18.1, Max: 18.2, Diff: 0.1, Sum: 72.4] sb-pinot-server-0 sb-pinot-controller-1 controller server [Free CSet: 0.8 ms] sb-pinot-server-0 [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29766__20210121T1107Z sb-pinot-server-1 server server2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29767__20210121T1107Z sb-pinot-server-1 controller server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29768__20210121T1107Z sb-pinot-server-1 [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 43.0, Avg: 43.0, Max: 43.0, Diff: 0.0, Sum: 171.9] server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29769__20210121T1107Z sb-pinot-server-1 [Eden: 900.0M(900.0M)->0.0B(167.0M) Survivors: 32768.0K->37888.0K Heap: 3705.3M(4096.0M)->2630.9M(4096.0M)] sb-pinot-server-0 server sb-pinot-controller-1 controller server [GC Worker End (ms): Min: 1350461.9, Avg: 1350461.9, Max: 1350461.9, Diff: 0.0] sb-pinot-controller-1 controller 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29770__20210121T1108Z sb-pinot-server-1 [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms] sb-pinot-controller-1 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29771__20210121T1108Z sb-pinot-server-1 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29772__20210121T1108Z [Times: user=0.19 sys=0.00, real=0.03 secs] sb-pinot-server-1sb-pinot-server-0 server 2021/01/21 17:22:51.952 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29773__20210121T1108Z serversb-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29774__20210121T1108Z sb-pinot-server-1 controllerserver 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29775__20210121T1108Z [Clear CT: 0.1 ms] sb-pinot-controller-1 controller 2021-01-21T17:25:06.966+0000: 168.170: Total time for which application threads were stopped: 0.0366911 seconds, Stopping threads took: 0.0002408 seconds sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29776__20210121T1108Z sb-pinot-server-1 server [Other: 0.6 ms] sb-pinot-controller-1 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29777__20210121T1109Z sb-pinot-server-1 controller [Choose CSet: 0.0 ms] sb-pinot-controller-1 server controller 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29778__20210121T1109Z sb-pinot-server-1 2021-01-21T17:25:07.002+0000: 168.207: Application time: 0.0364979 seconds sb-pinot-server-0 server 2021-01-21T17:25:07.004+0000: 168.208: [GC pause (G1 Evacuation Pause) (mixed), 0.0506602 secs] sb-pinot-server-0 server [Parallel Time: 47.3 ms, GC Workers: 6] [Ref Proc: 0.1 ms] sb-pinot-server-0 server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29779__20210121T1109Z server [GC Worker Start (ms): Min: 168208.9, Avg: 168208.9, Max: 168208.9, Diff: 0.1] sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.4, Max: 1.6, Diff: 0.3, Sum: 8.3] sb-pinot-server-0 sb-pinot-controller-1 controller [Ref Enq: 0.0 ms] sb-pinot-controller-1 controller [Redirty Cards: 0.1 ms] sb-pinot-controller-1 controllersb-pinot-server-1 [Humongous Register: 0.0 ms] sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] sb-pinot-controller-1server 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29780__20210121T1109Z sb-pinot-server-1 server server [Update RS (ms): Min: 3.5, Avg: 12.9, Max: 31.6, Diff: 28.1, Sum: 77.6] sb-pinot-server-0 2021/01/21 17:22:51.953 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29781__20210121T1109Z sb-pinot-server-1 controller server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29782__20210121T1109Z sb-pinot-server-1 serverserver [Processed Buffers: Min: 6, Avg: 14.3, Max: 31, Diff: 25, Sum: 86] 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29783__20210121T1109Z sb-pinot-server-0 server [Scan RS (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: 0.8, Sum: 2.8] sb-pinot-server-1sb-pinot-server-0 [Free CSet: 0.1 ms] sb-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 470.8M(1024.0M)->444.5M(1024.0M)] sb-pinot-controller-1 controller [Times: user=0.17 sys=0.00, real=0.05 secs] sb-pinot-controller-1server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 controller server [Object Copy (ms): Min: 14.1, Avg: 32.3, Max: 41.4, Diff: 27.4, Sum: 193.9] sb-pinot-server-0 2021-01-21T17:25:04.069+0000: 1350.463: Total time for which application threads were stopped: 0.0443644 seconds, Stopping threads took: 0.0001010 seconds sb-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29784__20210121T1109Z sb-pinot-server-1server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29785__20210121T1110Z sb-pinot-server-1controller server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29786__20210121T1110Z sb-pinot-server-1 serversb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29787__20210121T1110Z sb-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29788__20210121T1110Z sb-pinot-server-1 2021-01-21T17:25:04.093+0000: 1350.486: Application time: 0.0234818 seconds sb-pinot-controller-1 [GC Worker Total (ms): Min: 47.1, Avg: 47.1, Max: 47.1, Diff: 0.1, Sum: 282.8] sb-pinot-server-0 server [GC Worker End (ms): Min: 168256.0, Avg: 168256.0, Max: 168256.1, Diff: 0.0] sb-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29789__20210121T1110Z sb-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29790__20210121T1110Z controller sb-pinot-server-1 server 2021-01-21T17:25:04.093+0000: 1350.487: Total time for which application threads were stopped: 0.0004505 seconds, Stopping threads took: 0.0000400 seconds sb-pinot-controller-1 server [Code Root Fixup: 0.0 ms] sb-pinot-server-0 server [Code Root Purge: 0.0 ms] sb-pinot-server-0 server [Clear CT: 0.5 ms] sb-pinot-server-0 server [Other: 2.9 ms] sb-pinot-server-0 server [Choose CSet: 0.2 ms] sb-pinot-server-0 server [Ref Proc: 1.1 ms] sb-pinot-server-02021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29791__20210121T1110Z sb-pinot-server-1 controller 2021-01-21T17:25:04.093+0000: 1350.487: Application time: 0.0000469 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.094+0000: 1350.487: [GC pause (GCLocker Initiated GC) (young), 0.0285995 secs] sb-pinot-controller-1 server controller [Parallel Time: 27.9 ms, GC Workers: 4] sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350487.1, Avg: 1350487.1, Max: 1350487.1, Diff: 0.0] 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29792__20210121T1111Z sb-pinot-server-1 server [Ref Enq: 0.0 ms] sb-pinot-server-0 sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.2, Sum: 4.1] server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29793__20210121T1111Z sb-pinot-server-1 server sb-pinot-controller-1 server controller [Update RS (ms): Min: 21.6, Avg: 21.7, Max: 21.9, Diff: 0.2, Sum: 86.8] 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29794__20210121T1111Z sb-pinot-controller-1 sb-pinot-server-1 controller [Processed Buffers: Min: 85, Avg: 88.0, Max: 90, Diff: 5, Sum: 352] sb-pinot-controller-1 [Redirty Cards: 0.1 ms] controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29795__20210121T1111Z sb-pinot-controller-1 sb-pinot-server-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29796__20210121T1111Z sb-pinot-server-1 server 2021/01/21 17:22:51.955 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29797__20210121T1111Z sb-pinot-server-1 server sb-pinot-server-0 controllerserver [Humongous Register: 0.1 ms] 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29798__20210121T1111Z sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29799__20210121T1111Z sb-pinot-server-1 sb-pinot-server-0 server [Object Copy (ms): Min: 4.9, Avg: 5.1, Max: 5.2, Diff: 0.3, Sum: 20.4] [Humongous Reclaim: 0.1 ms] sb-pinot-controller-1 sb-pinot-server-0 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29800__20210121T1111Z sb-pinot-server-1 server server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29801__20210121T1111Z sb-pinot-server-1 server sb-pinot-controller-1 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29802__20210121T1112Z controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-server-1 [Free CSet: 0.9 ms] sb-pinot-server-0 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29803__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29804__20210121T1112Z sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29805__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29806__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29807__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29808__20210121T1112Z sb-pinot-server-1 server server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29809__20210121T1112Z controller [GC Worker Total (ms): Min: 27.8, Avg: 27.9, Max: 27.9, Diff: 0.0, Sum: 111.4] sb-pinot-controller-1 [Eden: 167.0M(167.0M)->0.0B(200.0M) Survivors: 37888.0K->4096.0K Heap: 2797.9M(4096.0M)->2222.6M(4096.0M)] controller [GC Worker End (ms): Min: 1350515.0, Avg: 1350515.0, Max: 1350515.0, Diff: 0.0] sb-pinot-controller-1 sb-pinot-server-0 server [Times: user=0.14 sys=0.00, real=0.05 secs] sb-pinot-server-0 controller [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms] sb-pinot-controller-1 sb-pinot-server-1 controller [Clear CT: 0.1 ms] sb-pinot-controller-1 controller [Other: 0.6 ms] sb-pinot-controller-1 controller [Choose CSet: 0.0 ms] server server 2021-01-21T17:25:07.054+0000: 168.259: Total time for which application threads were stopped: 0.0522104 seconds, Stopping threads took: 0.0008706 seconds sb-pinot-server-0 server 2021-01-21T17:25:07.304+0000: 168.509: Application time: 0.2499433 seconds sb-pinot-server-0 sb-pinot-controller-1 server 2021-01-21T17:25:07.305+0000: 168.510: [GC pause (G1 Evacuation Pause) (mixed), 0.0317508 secs] controller [Ref Proc: 0.1 ms] sb-pinot-controller-1 controller [Ref Enq: 0.0 ms] sb-pinot-server-0 server sb-pinot-controller-1 controller [Redirty Cards: 0.1 ms] [Parallel Time: 29.1 ms, GC Workers: 6] sb-pinot-controller-1 controller 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29810__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29811__20210121T1112Z [Humongous Register: 0.0 ms] sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] sb-pinot-controller-1 controller sb-pinot-server-0 server [GC Worker Start (ms): Min: 168510.7, Avg: 168510.7, Max: 168510.7, Diff: 0.1] sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.4, Diff: 0.2, Sum: 7.9] sb-pinot-server-1 [Free CSet: 0.1 ms] sb-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(91136.0K) Survivors: 7168.0K->2048.0K Heap: 493.8M(1024.0M)->450.0M(1024.0M)] sb-pinot-server-0 sb-pinot-controller-1 server controller [Times: user=0.11 sys=0.00, real=0.03 secs] sb-pinot-controller-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29812__20210121T1112Z sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29813__20210121T1112Z controller 2021-01-21T17:25:04.122+0000: 1350.516: Total time for which application threads were stopped: 0.0289920 seconds, Stopping threads took: 0.0000117 seconds sb-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29814__20210121T1112Z sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29815__20210121T1113Z sb-pinot-server-1 [Update RS (ms): Min: 22.7, Avg: 22.7, Max: 22.8, Diff: 0.1, Sum: 136.4] sb-pinot-server-0 server controller server 2021/01/21 17:22:51.957 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29816__20210121T1113Z [Processed Buffers: Min: 90, Avg: 107.3, Max: 116, Diff: 26, Sum: 644] sb-pinot-server-0 server [Scan RS (ms): Min: 1.2, Avg: 1.2, Max: 1.2, Diff: 0.1, Sum: 7.3] sb-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29817__20210121T1113Z sb-pinot-server-1 sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Object Copy (ms): Min: 3.5, Avg: 3.7, Max: 3.8, Diff: 0.2, Sum: 22.1] server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29818__20210121T1113Z sb-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29819__20210121T1113Z sb-pinot-server-1 2021-01-21T17:25:04.125+0000: 1350.518: Application time: 0.0027123 seconds sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller server 2021-01-21T17:25:04.125+0000: 1350.519: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0110860 secs] sb-pinot-controller-1 sb-pinot-server-0 2021/01/21 17:22:51.958 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29820__20210121T1113Z server [Termination Attempts: Min: 1, Avg: 1.3, Max: 2, Diff: 1, Sum: 8] sb-pinot-server-0 server sb-pinot-server-1 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] controller [Parallel Time: 10.5 ms, GC Workers: 4] sb-pinot-controller-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29821__20210121T1114Z sb-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29822__20210121T1114Z controller [GC Worker Start (ms): Min: 1350519.1, Avg: 1350519.1, Max: 1350519.1, Diff: 0.0] sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.2, Max: 1.4, Diff: 0.2, Sum: 5.0] sb-pinot-controller-1 sb-pinot-server-1 sb-pinot-server-0 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29823__20210121T1114Z sb-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29824__20210121T1114Z serversb-pinot-server-1controller [Update RS (ms): Min: 8.3, Avg: 8.3, Max: 8.3, Diff: 0.0, Sum: 33.2] sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 28.9, Avg: 29.0, Max: 29.0, Diff: 0.0, Sum: 173.8] sb-pinot-server-0 [Processed Buffers: Min: 37, Avg: 38.8, Max: 41, Diff: 4, Sum: 155] sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server [GC Worker End (ms): Min: 168539.7, Avg: 168539.7, Max: 168539.7, Diff: 0.0] sb-pinot-server-0 sb-pinot-controller-1 controller [Object Copy (ms): Min: 0.7, Avg: 0.8, Max: 0.9, Diff: 0.2, Sum: 3.3] sb-pinot-controller-1 servercontroller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 5] sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29825__20210121T1114Z server [Code Root Fixup: 0.0 ms] sb-pinot-server-0 server [Code Root Purge: 0.0 ms] sb-pinot-server-0 [GC Worker Total (ms): Min: 10.4, Avg: 10.4, Max: 10.4, Diff: 0.0, Sum: 41.5] sb-pinot-controller-1 sb-pinot-server-1 server controller [GC Worker End (ms): Min: 1350529.5, Avg: 1350529.5, Max: 1350529.5, Diff: 0.0] server [Clear CT: 0.4 ms] 2021/01/21 17:22:51.959 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29826__20210121T1114Z sb-pinot-server-0 sb-pinot-server-1 sb-pinot-controller-1 controller server [Code Root Fixup: 0.0 ms] [Other: 2.3 ms] sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms] sb-pinot-controller-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29827__20210121T1114Z sb-pinot-server-1 server sb-pinot-server-0 server [Choose CSet: 0.2 ms] sb-pinot-server-0 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29828__20210121T1114Z controller sb-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29829__20210121T1115Z sb-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29830__20210121T1115Z sb-pinot-server-1 server [Ref Proc: 0.3 ms] [Clear CT: 0.1 ms] sb-pinot-server-0 server sb-pinot-controller-1 controller [Other: 0.5 ms] sb-pinot-controller-1 controller 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29831__20210121T1115Z sb-pinot-server-1 serverserver [Ref Enq: 0.0 ms] sb-pinot-server-0 2021/01/21 17:22:51.960 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29832__20210121T1115Z server sb-pinot-server-1 server 2021/01/21 17:22:51.961 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29833__20210121T1115Z sb-pinot-server-1 server [Redirty Cards: 0.1 ms] sb-pinot-server-0 [Choose CSet: 0.0 ms] 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29834__20210121T1115Z sb-pinot-server-1 server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29835__20210121T1115Z sb-pinot-server-1 sb-pinot-controller-1 controller server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29836__20210121T1115Z sb-pinot-server-1 server [Humongous Register: 0.1 ms] server 2021/01/21 17:22:51.962 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29837__20210121T1115Z sb-pinot-server-0 sb-pinot-server-1 serverserver 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29838__20210121T1116Z sb-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29839__20210121T1116Z sb-pinot-server-1 [Humongous Reclaim: 0.1 ms] server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29840__20210121T1116Z sb-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29841__20210121T1116Z sb-pinot-server-0 [Ref Proc: 0.1 ms] server [Free CSet: 1.1 ms] sb-pinot-server-0 server [Eden: 200.0M(200.0M)->0.0B(195.0M) Survivors: 4096.0K->9216.0K Heap: 2471.9M(4096.0M)->1820.5M(4096.0M)] sb-pinot-server-0 server sb-pinot-server-1 [Times: user=0.18 sys=0.01, real=0.03 secs] server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29842__20210121T1116Z sb-pinot-server-1 server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29843__20210121T1116Z sb-pinot-controller-1 sb-pinot-server-0 sb-pinot-server-1server 2021-01-21T17:25:07.337+0000: 168.542: Total time for which application threads were stopped: 0.0329319 seconds, Stopping threads took: 0.0005259 seconds sb-pinot-server-0 server controller [Ref Enq: 0.0 ms] sb-pinot-controller-1server 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29844__20210121T1116Z sb-pinot-server-1 server controller 2021-01-21T17:25:07.632+0000: 168.836: Application time: 0.2943829 seconds sb-pinot-server-0 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29845__20210121T1117Z sb-pinot-server-1 server server 2021-01-21T17:25:07.632+0000: 168.837: [GC pause (G1 Evacuation Pause) (mixed), 0.0321493 secs] [Redirty Cards: 0.1 ms] sb-pinot-server-0 server [Parallel Time: 28.6 ms, GC Workers: 6] sb-pinot-server-0 server [GC Worker Start (ms): Min: 168838.0, Avg: 168838.0, Max: 168838.0, Diff: 0.1] sb-pinot-controller-1 controller [Humongous Register: 0.0 ms] sb-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] sb-pinot-server-0 sb-pinot-controller-1 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.3, Max: 1.5, Diff: 0.2, Sum: 8.1] sb-pinot-server-0 server [Update RS (ms): Min: 9.7, Avg: 9.8, Max: 9.8, Diff: 0.1, Sum: 58.6] sb-pinot-server-0 controllerserver [Free CSet: 0.0 ms] [Processed Buffers: Min: 67, Avg: 79.5, Max: 89, Diff: 22, Sum: 477] 2021/01/21 17:22:51.964 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29846__20210121T1117Z sb-pinot-server-1 sb-pinot-server-0 sb-pinot-controller-1 server [Scan RS (ms): Min: 1.8, Avg: 1.8, Max: 1.9, Diff: 0.1, Sum: 10.9] server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29847__20210121T1117Z controller [Eden: 5120.0K(91136.0K)->0.0B(152.0M) Survivors: 2048.0K->3072.0K Heap: 461.0M(1024.0M)->453.5M(1024.0M)] sb-pinot-server-0 sb-pinot-controller-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Object Copy (ms): Min: 15.4, Avg: 15.5, Max: 15.6, Diff: 0.2, Sum: 92.9] sb-pinot-server-0 serversb-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29848__20210121T1117Z sb-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29849__20210121T1117Z sb-pinot-server-1 controllerserver [Times: user=0.04 sys=0.00, real=0.01 secs] sb-pinot-controller-1 controller2021/01/21 17:22:51.965 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29850__20210121T1117Z 2021-01-21T17:25:04.137+0000: 1350.530: [GC concurrent-root-region-scan-start] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 sb-pinot-server-0 controller 2021-01-21T17:25:04.137+0000: 1350.530: Total time for which application threads were stopped: 0.0116309 seconds, Stopping threads took: 0.0001002 seconds server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] sb-pinot-server-0 sb-pinot-server-1sb-pinot-controller-1 controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-root-region-scan-end, 0.0031610 secs] server 2021/01/21 17:22:51.966 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29851__20210121T1117Z sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.966 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29852__20210121T1118Z sb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29853__20210121T1118Z server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-server-1controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-mark-start] sb-pinot-server-0 server [GC Worker Total (ms): Min: 28.4, Avg: 28.4, Max: 28.5, Diff: 0.1, Sum: 170.6] sb-pinot-server-0 server sb-pinot-controller-1 controller server 2021-01-21T17:25:04.256+0000: 1350.649: Application time: 0.1189493 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.256+0000: 1350.649: [GC pause (G1 Evacuation Pause) (young), 0.0236359 secs] sb-pinot-controller-1 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29854__20210121T1118Z controller [Parallel Time: 23.0 ms, GC Workers: 4] sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350649.5, Avg: 1350649.5, Max: 1350649.5, Diff: 0.1] [GC Worker End (ms): Min: 168866.4, Avg: 168866.4, Max: 168866.4, Diff: 0.0] sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29855__20210121T1118Z [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.2, Diff: 0.2, Sum: 4.2] sb-pinot-controller-1 controller [Update RS (ms): Min: 6.5, Avg: 6.6, Max: 6.6, Diff: 0.1, Sum: 26.2] sb-pinot-controller-1 controller [Processed Buffers: Min: 65, Avg: 70.0, Max: 74, Diff: 9, Sum: 280] sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Object Copy (ms): Min: 15.2, Avg: 15.2, Max: 15.3, Diff: 0.2, Sum: 61.0] sb-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-controller-1 controllersb-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29856__20210121T1118Z sb-pinot-server-0 sb-pinot-server-1server [Code Root Fixup: 0.0 ms] sb-pinot-server-0 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 22.9, Avg: 22.9, Max: 22.9, Diff: 0.0, Sum: 91.6] server server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29857__20210121T1118Z sb-pinot-server-1 [Code Root Purge: 0.0 ms] sb-pinot-server-0 server [Clear CT: 0.4 ms] sb-pinot-server-0 server [Other: 3.2 ms] sb-pinot-server-0 server [Choose CSet: 0.4 ms] sb-pinot-server-0 server [Ref Proc: 0.8 ms] sb-pinot-server-0 server [Ref Enq: 0.0 ms] sb-pinot-server-0 server [Redirty Cards: 0.1 ms] sb-pinot-server-0 server [Humongous Register: 0.1 ms] sb-pinot-server-0 server [Humongous Reclaim: 0.1 ms] sb-pinot-server-0 server [Free CSet: 1.2 ms] sb-pinot-server-0 server [Eden: 195.0M(195.0M)->0.0B(192.0M) Survivors: 9216.0K->12288.0K Heap: 2064.8M(4096.0M)->1443.5M(4096.0M)] sb-pinot-server-0 server [Times: user=0.17 sys=0.00, real=0.03 secs] sb-pinot-server-0 server 2021-01-21T17:25:07.665+0000: 168.870: Total time for which application threads were stopped: 0.0330694 seconds, Stopping threads took: 0.0002922 seconds sb-pinot-server-0 server 2021-01-21T17:25:07.829+0000: 169.034: Application time: 0.1644559 seconds sb-pinot-server-0 server 2021-01-21T17:25:07.838+0000: 169.043: [GC pause (G1 Evacuation Pause) (mixed), 0.0260221 secs] sb-pinot-server-0 server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29858__20210121T1118Z sb-pinot-server-1sb-pinot-controller-1 server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29859__20210121T1119Z sb-pinot-server-1 controller [GC Worker End (ms): Min: 1350672.4, Avg: 1350672.4, Max: 1350672.4, Diff: 0.0] sb-pinot-controller-1 controller [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controller server 2021/01/21 17:22:51.969 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29860__20210121T1119Z sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29861__20210121T1119Z [Code Root Purge: 0.0 ms] sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29862__20210121T1119Z sb-pinot-server-1 sb-pinot-controller-1 controller [Clear CT: 0.1 ms] sb-pinot-controller-1server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29863__20210121T1119Z sb-pinot-server-1 server server [Parallel Time: 23.6 ms, GC Workers: 6] sb-pinot-server-0 server [GC Worker Start (ms): Min: 169043.2, Avg: 169043.2, Max: 169043.3, Diff: 0.1] sb-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.5, Max: 1.6, Diff: 0.2, Sum: 8.9] sb-pinot-server-0 server [Update RS (ms): Min: 13.8, Avg: 13.8, Max: 13.8, Diff: 0.1, Sum: 82.8] sb-pinot-server-0 server [Processed Buffers: Min: 50, Avg: 54.3, Max: 58, Diff: 8, Sum: 326] sb-pinot-server-0 server [Scan RS (ms): Min: 0.2, Avg: 0.3, Max: 0.3, Diff: 0.1, Sum: 1.6] 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29864__20210121T1119Z sb-pinot-server-1sb-pinot-server-0 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29865__20210121T1119Z sb-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29866__20210121T1120Z sb-pinot-server-1 server controller2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29867__20210121T1120Z sb-pinot-server-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Object Copy (ms): Min: 7.7, Avg: 7.9, Max: 8.0, Diff: 0.2, Sum: 47.2] sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.2] sb-pinot-server-0 server [GC Worker Total (ms): Min: 23.4, Avg: 23.4, Max: 23.5, Diff: 0.1, Sum: 140.7] [Other: 0.6 ms] sb-pinot-controller-1 server sb-pinot-server-0 server controller [Choose CSet: 0.0 ms] sb-pinot-controller-1 controller [Ref Proc: 0.1 ms] [GC Worker End (ms): Min: 169066.7, Avg: 169066.7, Max: 169066.7, Diff: 0.0] sb-pinot-server-0 2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29868__20210121T1120Z sb-pinot-server-1 server 2021/01/21 17:22:51.971 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29869__20210121T1120Z server sb-pinot-server-1 [Code Root Fixup: 0.0 ms] sb-pinot-server-0 server [Code Root Purge: 0.0 ms] sb-pinot-server-0 server server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29870__20210121T1120Z sb-pinot-server-1 sb-pinot-controller-1 controller [Ref Enq: 0.0 ms] [Clear CT: 0.2 ms] sb-pinot-controller-1 controller sb-pinot-server-0 server [Redirty Cards: 0.1 ms] sb-pinot-controller-1 [Other: 2.2 ms] sb-pinot-server-0 controller server [Choose CSet: 0.0 ms] [Humongous Register: 0.0 ms] sb-pinot-server-0 server [Ref Proc: 1.0 ms] sb-pinot-server-0 server [Ref Enq: 0.0 ms] sb-pinot-server-0 server [Redirty Cards: 0.1 ms] sb-pinot-server-0 sb-pinot-controller-1 controller server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29871__20210121T1120Z sb-pinot-server-1 server [Humongous Register: 0.2 ms] sb-pinot-server-0 server [Humongous Reclaim: 0.1 ms] sb-pinot-server-0 server [Free CSet: 0.3 ms] server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29872__20210121T1120Z sb-pinot-server-1 server 2021/01/21 17:22:51.972 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29873__20210121T1120Z sb-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29874__20210121T1120Z sb-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29875__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.974 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29876__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29877__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29878__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29879__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29880__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29881__20210121T1121Z sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29882__20210121T1121Z sb-pinot-server-1 [Humongous Reclaim: 0.1 ms] sb-pinot-controller-1 controller [Free CSet: 0.1 ms] sb-pinot-controller-1 controller [Eden: 152.0M(152.0M)->0.0B(298.0M) Survivors: 3072.0K->20480.0K Heap: 638.1M(1024.0M)->482.3M(1024.0M)] sb-pinot-controller-1 controller [Times: user=0.09 sys=0.00, real=0.03 secs] sb-pinot-controller-1 controller 2021-01-21T17:25:04.280+0000: 1350.673: Total time for which application threads were stopped: 0.0241836 seconds, Stopping threads took: 0.0000443 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.955: [GC concurrent-mark-end, 0.4221306 secs] sb-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29883__20210121T1122Z sb-pinot-server-0 sb-pinot-server-1server [Eden: 192.0M(192.0M)->0.0B(2155.0M) Survivors: 12288.0K->23552.0K Heap: 1650.1M(4096.0M)->1449.3M(4096.0M)] sb-pinot-server-0 controllerserver [Times: user=0.14 sys=0.00, real=0.03 secs] sb-pinot-server-0 server 2021-01-21T17:25:07.864+0000: 169.069: Total time for which application threads were stopped: 0.0349989 seconds, Stopping threads took: 0.0082034 seconds sb-pinot-server-0 server 2021-01-21T17:25:09.808+0000: 171.013: Application time: 1.9435757 seconds sb-pinot-server-0 serverserver 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29884__20210121T1122Z sb-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29885__20210121T1122Z 2021-01-21T17:25:09.813+0000: 171.018: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0428457 secs] sb-pinot-server-0 2021-01-21T17:25:04.562+0000: 1350.955: Application time: 0.2822203 seconds server [Parallel Time: 37.6 ms, GC Workers: 6] sb-pinot-server-0 sb-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.956: [GC remark 2021-01-21T17:25:04.562+0000: 1350.956: [Finalize Marking, 0.0002598 secs] 2021-01-21T17:25:04.563+0000: 1350.956: [GC ref-proc, 0.0004448 secs] 2021-01-21T17:25:04.563+0000: 1350.957: [Unloading, 0.0095395 secs], 0.0108947 secs] sb-pinot-controller-1 sb-pinot-server-1 controller [Times: user=0.04 sys=0.00, real=0.01 secs] sb-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Total time for which application threads were stopped: 0.0114314 seconds, Stopping threads took: 0.0000926 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Application time: 0.0000628 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.574+0000: 1350.967: [GC cleanup 774M->655M(1024M), 0.0018691 secs] sb-pinot-controller-1 controller [Times: user=0.01 sys=0.00, real=0.00 secs] sb-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29886__20210121T1122Z controller sb-pinot-server-1 server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29887__20210121T1122Z server [GC Worker Start (ms): Min: 171018.3, Avg: 171018.4, Max: 171018.4, Diff: 0.1] sb-pinot-server-0 sb-pinot-server-1server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.4, Max: 1.6, Diff: 0.2, Sum: 8.6] sb-pinot-server-0 server [Update RS (ms): Min: 20.6, Avg: 20.7, Max: 20.8, Diff: 0.1, Sum: 124.3] sb-pinot-server-0 server [Processed Buffers: Min: 144, Avg: 149.8, Max: 159, Diff: 15, Sum: 899] sb-pinot-server-0 2021-01-21T17:25:04.576+0000: 1350.969: Total time for which application threads were stopped: 0.0023571 seconds, Stopping threads took: 0.0000678 seconds sb-pinot-controller-1 server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29888__20210121T1122Z sb-pinot-server-1 controllerserver [Scan RS (ms): Min: 0.4, Avg: 0.4, Max: 0.5, Diff: 0.1, Sum: 2.6] server 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-start] sb-pinot-controller-1 sb-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-server-0 server [Object Copy (ms): Min: 14.5, Avg: 14.7, Max: 14.8, Diff: 0.2, Sum: 88.2] sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4] sb-pinot-server-0 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29889__20210121T1122Z sb-pinot-server-1 controller 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-end, 0.0001739 secs] server 2021/01/21 17:22:51.978 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29890__20210121T1123Z sb-pinot-controller-1 server [GC Worker Total (ms): Min: 37.3, Avg: 37.4, Max: 37.4, Diff: 0.1, Sum: 224.2] sb-pinot-server-0 server [GC Worker End (ms): Min: 171055.7, Avg: 171055.7, Max: 171055.8, Diff: 0.1] sb-pinot-server-0 server [Code Root Fixup: 0.0 ms] sb-pinot-server-0 server controller 2021-01-21T17:25:04.654+0000: 1351.048: Application time: 0.0785958 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.655+0000: 1351.048: [GC pause (G1 Evacuation Pause) (young), 0.0666046 secs] sb-pinot-server-1 [Code Root Purge: 0.0 ms] server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29891__20210121T1123Z sb-pinot-server-1 sb-pinot-controller-1 controller [Parallel Time: 65.5 ms, GC Workers: 4] sb-pinot-controller-1 sb-pinot-server-0 server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29892__20210121T1123Z server [Clear CT: 0.5 ms] sb-pinot-server-0 server [Other: 4.8 ms] sb-pinot-server-0 server [Choose CSet: 0.0 ms] sb-pinot-server-0 server [Ref Proc: 2.2 ms] sb-pinot-server-0 server [Ref Enq: 0.0 ms] sb-pinot-server-0 server [Redirty Cards: 0.1 ms] sb-pinot-server-0 server [Humongous Register: 0.1 ms] sb-pinot-server-0 server [Humongous Reclaim: 0.6 ms] controller sb-pinot-server-0 [GC Worker Start (ms): Min: 1351048.4, Avg: 1351048.4, Max: 1351048.4, Diff: 0.0] server [Free CSet: 1.0 ms] sb-pinot-controller-1 sb-pinot-server-1 sb-pinot-server-0 server [Eden: 1386.0M(2155.0M)->0.0B(2134.0M) Survivors: 23552.0K->48128.0K Heap: 3188.4M(4096.0M)->1474.4M(4096.0M)] sb-pinot-server-0 server [Times: user=0.23 sys=0.00, real=0.04 secs] sb-pinot-server-0 server controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.6] sb-pinot-controller-1 controller server 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29893__20210121T1123Z 2021/01/21 17:25:10.641 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=sblog__0__13120__20210119T1725Z, expected=ONLINE, found=OFFLINE, creationTime=1611249776061, modifiedTime=1611249910406, version=1030, waitingFor=CurrentStateMatch, resource=sblog_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;; sb-pinot-server-0 [Update RS (ms): Min: 16.0, Avg: 16.0, Max: 16.0, Diff: 0.1, Sum: 63.9] server 2021-01-21T17:25:09.856+0000: 2021-01-21T17:25:09.856+0000: 171.061: 171.061: [GC concurrent-root-region-scan-start] sb-pinot-server-0 server sb-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 111.0, Max: 117, Diff: 14, Sum: 444] sb-pinot-server-1 server Total time for which application threads were stopped: 0.0480709 seconds, Stopping threads took: 0.0044806 seconds sb-pinot-server-0 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-root-region-scan-end, 0.0198202 secs] 2021/01/21 17:22:51.979 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29894__20210121T1123Z sb-pinot-server-1 sb-pinot-controller-1 controller [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.1, Diff: 0.0, Sum: 0.3] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1server 2021/01/21 17:22:51.980 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29895__20210121T1123Z sb-pinot-server-1 server 2021/01/21 17:22:51.980 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29896__20210121T1123Z sb-pinot-server-1 controller [Object Copy (ms): Min: 48.1, Avg: 48.2, Max: 48.3, Diff: 0.2, Sum: 192.8] server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29897__20210121T1124Z sb-pinot-server-1 sb-pinot-server-0serversb-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 65.4, Avg: 65.4, Max: 65.4, Diff: 0.0, Sum: 261.7] sb-pinot-controller-1 controller [GC Worker End (ms): Min: 1351113.8, Avg: 1351113.8, Max: 1351113.8, Diff: 0.0] sb-pinot-controller-1 controller [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controller [Code Root Purge: 0.0 ms] sb-pinot-controller-1 controller [Clear CT: 0.1 ms] sb-pinot-controller-1 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-mark-start] sb-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: [GC concurrent-mark-end, 1.3475822 secs] 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29898__20210121T1124Z sb-pinot-server-1 sb-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: Application time: 1.3674839 seconds server sb-pinot-server-0 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29899__20210121T1124Z sb-pinot-server-1 controller [Other: 1.0 ms] sb-pinot-controller-1 controllerserver 2021-01-21T17:25:11.228+0000: 172.433: [GC remark 2021-01-21T17:25:11.228+0000: 172.433: [Finalize Marking, 0.0003440 secs] 2021-01-21T17:25:11.229+0000: 172.434: [GC ref-proc, 0.0850389 secs] 2021-01-21T17:25:11.314+0000: 172.519: [Unloading, 0.0079844 secs], 0.0957425 secs] sb-pinot-server-0 server [Times: user=0.11 sys=0.03, real=0.10 secs] sb-pinot-server-0 server 2021-01-21T17:25:11.324+0000: 172.529: Total time for which application threads were stopped: 0.1009431 seconds, Stopping threads took: 0.0045995 seconds server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29900__20210121T1124Z [Choose CSet: 0.0 ms] sb-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29901__20210121T1124Z sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29902__20210121T1124Z sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29903__20210121T1125Z sb-pinot-server-1 controller2021-01-21T17:25:11.324+0000: 172.529: Application time: 0.0000758 seconds sb-pinot-server-0 server server 2021-01-21T17:25:11.325+0000: 172.530: [GC cleanup 2780M->2780M(4096M), 0.0035921 secs] sb-pinot-server-0 [Ref Proc: 0.2 ms] sb-pinot-controller-1 controller server [Ref Enq: 0.0 ms] sb-pinot-controller-1 controller [Redirty Cards: 0.3 ms] [Times: user=0.02 sys=0.00, real=0.00 secs] sb-pinot-controller-1 2021/01/21 17:22:51.981 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29904__20210121T1125Z sb-pinot-server-1 serversb-pinot-server-0 server 2021-01-21T17:25:11.328+0000: 172.533: Total time for which application threads were stopped: 0.0041608 seconds, Stopping threads took: 0.0000167 seconds sb-pinot-server-0 server 2021/01/21 17:22:51.982 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29905__20210121T1125Z sb-pinot-server-1 2021-01-21T17:25:11.481+0000: 172.686: Application time: 0.1529441 seconds servercontrollersb-pinot-server-0 server 2021-01-21T17:25:11.482+0000: 172.687: Total time for which application threads were stopped: 0.0005239 seconds, Stopping threads took: 0.0000613 seconds sb-pinot-server-0 [Humongous Register: 0.0 ms] serversb-pinot-controller-1 2021/01/21 17:22:51.982 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29906__20210121T1125Z sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29907__20210121T1125Z sb-pinot-server-1 controller server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29908__20210121T1125Z sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29909__20210121T1125Z [Humongous Reclaim: 0.1 ms] sb-pinot-controller-1sb-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29910__20210121T1125Z sb-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29911__20210121T1126Z controller [Free CSet: 0.2 ms] sb-pinot-controller-1 controller [Eden: 298.0M(298.0M)->0.0B(11264.0K) Survivors: 20480.0K->40960.0K Heap: 700.6M(1024.0M)->429.5M(1024.0M)] sb-pinot-controller-1 controller [Times: user=0.26 sys=0.00, real=0.07 secs] sb-pinot-controller-1 controller sb-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29912__20210121T1126Z sb-pinot-server-1 server 2021-01-21T17:25:11.489+0000: 172.694: Application time: 0.0071873 seconds sb-pinot-server-0 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29913__20210121T1126Z sb-pinot-server-1 server 2021-01-21T17:25:11.490+0000: 172.694: Total time for which application threads were stopped: 0.0004430 seconds, Stopping threads took: 0.0000272 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.551+0000: 172.755: Application time: 0.0610437 seconds 2021-01-21T17:25:04.722+0000: 1351.115: Total time for which application threads were stopped: 0.0671829 seconds, Stopping threads took: 0.0000684 seconds sb-pinot-controller-1 sb-pinot-server-0controller 2021-01-21T17:25:04.737+0000: 1351.130: Application time: 0.0150208 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.737+0000: 1351.130: [GC pause (G1 Evacuation Pause) (mixed), 0.0775762 secs] server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29914__20210121T1126Z server 2021-01-21T17:25:11.552+0000: 172.757: Total time for which application threads were stopped: 0.0013581 seconds, Stopping threads took: 0.0000241 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.710+0000: 172.915: Application time: 0.1581764 seconds sb-pinot-server-0 sb-pinot-server-1 server sb-pinot-controller-1 controller [Parallel Time: 76.1 ms, GC Workers: 4] sb-pinot-controller-1 controller [GC Worker Start (ms): Min: 1351130.8, Avg: 1351130.8, Max: 1351130.8, Diff: 0.1] sb-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.2] sb-pinot-controller-1 controller [Update RS (ms): Min: 31.3, Avg: 31.4, Max: 31.4, Diff: 0.1, Sum: 125.4] sb-pinot-controller-1 controller [Processed Buffers: Min: 124, Avg: 125.0, Max: 126, Diff: 2, Sum: 500] sb-pinot-controller-1 server controller 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29915__20210121T1126Z sb-pinot-server-1 2021-01-21T17:25:11.718+0000: 172.923: Total time for which application threads were stopped: 0.0080070 seconds, Stopping threads took: 0.0074375 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.724+0000: 172.928: Application time: 0.0054288 seconds sb-pinot-server-0server 2021/01/21 17:22:51.984 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29916__20210121T1126Z sb-pinot-server-1 server 2021/01/21 17:22:51.986 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29917__20210121T1126Z sb-pinot-server-1 [Scan RS (ms): Min: 13.4, Avg: 13.5, Max: 13.7, Diff: 0.3, Sum: 54.1] sb-pinot-controller-1 controllerserver [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5] 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29918__20210121T1127Z sb-pinot-controller-1 sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29919__20210121T1127Z controller [Object Copy (ms): Min: 29.8, Avg: 29.9, Max: 30.1, Diff: 0.3, Sum: 119.8] sb-pinot-controller-1 server controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29920__20210121T1127Z sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29921__20210121T1127Z 2021-01-21T17:25:11.728+0000: 172.933: Total time for which application threads were stopped: 0.0044722 seconds, Stopping threads took: 0.0031338 seconds sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29922__20210121T1127Z sb-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29923__20210121T1127Z sb-pinot-controller-1 sb-pinot-server-0 server controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] sb-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] sb-pinot-controller-1 2021-01-21T17:25:11.732+0000: 172.937: Application time: 0.0044179 seconds sb-pinot-server-0 controllerserver sb-pinot-server-1 [GC Worker Total (ms): Min: 76.0, Avg: 76.0, Max: 76.0, Diff: 0.0, Sum: 304.1] sb-pinot-controller-1 server 2021/01/21 17:22:51.987 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29924__20210121T1127Z sb-pinot-server-1 controller [GC Worker End (ms): Min: 1351206.8, Avg: 1351206.8, Max: 1351206.8, Diff: 0.0] server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29925__20210121T1127Z sb-pinot-server-1 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0004491 seconds, Stopping threads took: 0.0000215 seconds sb-pinot-server-0 server sb-pinot-controller-1 controller 2021-01-21T17:25:11.733+0000: 172.938: Application time: 0.0000363 seconds sb-pinot-server-0 [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 controllerserver 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0003160 seconds, Stopping threads took: 0.0000079 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Application time: 0.0124480 seconds [Code Root Purge: 0.0 ms] sb-pinot-controller-1 server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29926__20210121T1128Z sb-pinot-server-1 controller [Clear CT: 0.1 ms] server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29927__20210121T1128Z sb-pinot-server-1 server sb-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Total time for which application threads were stopped: 0.0003742 seconds, Stopping threads took: 0.0000195 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.759+0000: 172.964: Application time: 0.0131737 seconds sb-pinot-server-02021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29928__20210121T1128Z sb-pinot-server-1 server 2021/01/21 17:22:51.988 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29929__20210121T1128Z sb-pinot-server-1 sb-pinot-controller-1server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29930__20210121T1128Z sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29931__20210121T1128Z sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29932__20210121T1128Z sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29933__20210121T1129Z sb-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29934__20210121T1129Z server sb-pinot-server-1 controller [Other: 1.3 ms] sb-pinot-controller-1 controller [Choose CSet: 0.2 ms] sb-pinot-controller-1 controller [Ref Proc: 0.1 ms] sb-pinot-controller-1 controller [Ref Enq: 0.0 ms] sb-pinot-controller-1 2021-01-21T17:25:11.760+0000: 172.965: Total time for which application threads were stopped: 0.0007156 seconds, Stopping threads took: 0.0002863 seconds controller [Redirty Cards: 0.4 ms] sb-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Application time: 0.0087255 seconds sb-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Total time for which application threads were stopped: 0.0007774 seconds, Stopping threads took: 0.0002464 seconds sb-pinot-controller-1 controller [Humongous Register: 0.0 ms] sb-pinot-controller-1 sb-pinot-server-0 server 2021-01-21T17:25:11.771+0000: 172.976: Application time: 0.0015455 seconds controller sb-pinot-server-0 server 2021-01-21T17:25:11.773+0000: 172.977: Total time for which application threads were stopped: 0.0015423 seconds, Stopping threads took: 0.0002360 seconds sb-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: Application time: 0.5611696 seconds sb-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: [GC pause (G1 Evacuation Pause) (young), 0.2155960 secs] sb-pinot-server-0 [Humongous Reclaim: 0.0 ms] serversb-pinot-controller-1 controller [Free CSet: 0.4 ms] sb-pinot-controller-1 controller [Eden: 11264.0K(11264.0K)->0.0B(48128.0K) Survivors: 40960.0K->4096.0K Heap: 455.0M(1024.0M)->354.2M(1024.0M)] sb-pinot-controller-1 [Parallel Time: 208.2 ms, GC Workers: 6] sb-pinot-server-0 server server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29935__20210121T1129Z [GC Worker Start (ms): Min: 173539.5, Avg: 173539.5, Max: 173539.5, Diff: 0.1] sb-pinot-server-1 controller server sb-pinot-server-0 [Times: user=0.30 sys=0.00, real=0.07 secs] server [Ext Root Scanning (ms): Min: 1.1, Avg: 1.3, Max: 1.6, Diff: 0.4, Sum: 7.7] 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29936__20210121T1129Z sb-pinot-server-1 sb-pinot-controller-1 controller 2021-01-21T17:25:04.815+0000: 1351.208: Total time for which application threads were stopped: 0.0781603 seconds, Stopping threads took: 0.0000425 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.841+0000: 1351.235: Application time: 0.0263536 seconds sb-pinot-controller-1 controller 2021-01-21T17:25:04.842+0000: 1351.235: [GC pause (G1 Evacuation Pause) (mixed), 0.0820991 secs] server sb-pinot-controller-1 controller 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29937__20210121T1129Z sb-pinot-server-1 [Parallel Time: 81.1 ms, GC Workers: 4] sb-pinot-server-0 sb-pinot-controller-1 server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29938__20210121T1129Z controller [GC Worker Start (ms): Min: 1351235.2, Avg: 1351235.2, Max: 1351235.2, Diff: 0.0] sb-pinot-controller-1 servercontroller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.3] sb-pinot-controller-1 [Update RS (ms): Min: 15.5, Avg: 15.5, Max: 15.6, Diff: 0.1, Sum: 93.3] sb-pinot-server-0 server [Processed Buffers: Min: 108, Avg: 112.2, Max: 115, Diff: 7, Sum: 673] sb-pinot-server-0 server [Scan RS (ms): Min: 166.2, Avg: 166.3, Max: 166.5, Diff: 0.2, Sum: 998.0] sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29939__20210121T1129Z sb-pinot-server-1 controller server [Update RS (ms): Min: 35.6, Avg: 35.7, Max: 35.9, Diff: 0.4, Sum: 142.9] sb-pinot-controller-1 controller [Processed Buffers: Min: 150, Avg: 155.0, Max: 159, Diff: 9, Sum: 620] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2] sb-pinot-controller-1 controller [Scan RS (ms): Min: 5.9, Avg: 14.1, Max: 22.5, Diff: 16.6, Sum: 56.4] sb-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 4.2, Max: 16.4, Diff: 16.4, Sum: 16.8] server 2021/01/21 17:22:51.990 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29940__20210121T1129Z sb-pinot-server-1 server 2021/01/21 17:22:51.991 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29941__20210121T1129Z sb-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29942__20210121T1129Z sb-pinot-controller-1 sb-pinot-server-1 controller sb-pinot-server-0 [Object Copy (ms): Min: 21.7, Avg: 25.9, Max: 38.2, Diff: 16.6, Sum: 103.5] server server sb-pinot-controller-1 controller 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29943__20210121T1130Z sb-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29944__20210121T1130Z sb-pinot-server-1 [Object Copy (ms): Min: 24.4, Avg: 24.8, Max: 25.0, Diff: 0.6, Sum: 148.7] sb-pinot-server-0 server server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] sb-pinot-server-02021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29945__20210121T1130Z sb-pinot-server-1 server [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 7] sb-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4] sb-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server [GC Worker Total (ms): Min: 208.0, Avg: 208.0, Max: 208.1, Diff: 0.1, Sum: 1248.2] 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29946__20210121T1130Z sb-pinot-server-1 sb-pinot-controller-1 server 2021/01/21 17:22:51.992 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29947__20210121T1130Z sb-pinot-server-1 sb-pinot-server-0 server [GC Worker End (ms): Min: 173747.5, Avg: 173747.5, Max: 173747.6, Diff: 0.1] server sb-pinot-server-0 server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29948__20210121T1130Z sb-pinot-server-1 controller [Termination Attempts: Min: 1, Avg: 5.8, Max: 10, Diff: 9, Sum: 23] sb-pinot-controller-1 [Code Root Fixup: 0.0 ms] sb-pinot-server-0 server controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29949__20210121T1130Z sb-pinot-server-1 server 2021/01/21 17:22:51.994 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29950__20210121T1130Z sb-pinot-server-1 server 2021/01/21 17:22:51.995 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29951__20210121T1130Z sb-pinot-server-1 server sb-pinot-controller-1 controller [GC Worker Total (ms): Min: 81.0, Avg: 81.0, Max: 81.0, Diff: 0.0, Sum: 324.0] [Code Root Purge: 0.0 ms] 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29952__20210121T1130Z sb-pinot-server-1 sb-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29953__20210121T1130Z sb-pinot-controller-1 controller [GC Worker End (ms): Min: 1351316.2, Avg: 1351316.2, Max: 1351316.2, Diff: 0.0] sb-pinot-controller-1 [Clear CT: 0.9 ms] controller sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29954__20210121T1130Z sb-pinot-server-0 sb-pinot-server-1 server [Other: 6.5 ms] [Code Root Fixup: 0.0 ms] sb-pinot-controller-1 sb-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29955__20210121T1130Z [Choose CSet: 0.0 ms] sb-pinot-server-1 server sb-pinot-server-0 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29956__20210121T1131Z server [Ref Proc: 3.8 ms] sb-pinot-server-0 server [Ref Enq: 0.0 ms] sb-pinot-server-0 server [Redirty Cards: 0.2 ms] sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29957__20210121T1131Z sb-pinot-server-0 sb-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29958__20210121T1131Z sb-pinot-server-1 controller [Code Root Purge: 0.0 ms] sb-pinot-controller-1 controller [Clear CT: 0.1 ms] server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29959__20210121T1131Z server sb-pinot-server-1 server [Humongous Register: 0.2 ms] sb-pinot-server-0 sb-pinot-controller-1 controller 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29960__20210121T1131Z sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29961__20210121T1131Z sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29962__20210121T1132Z sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29963__20210121T1132Z sb-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29964__20210121T1132Z [Other: 0.8 ms] server [Humongous Reclaim: 0.7 ms] sb-pinot-server-0 sb-pinot-controller-1 server [Free CSet: 1.4 ms] controller sb-pinot-server-0 server sb-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29965__20210121T1132Z [Choose CSet: 0.1 ms] sb-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29966__20210121T1132Z sb-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29967__20210121T1132Z sb-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [sblog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/sblog_REALTIME/consumers/sblog__0__29968__20210121T1132Z ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
