pabrahamusa edited a comment on issue #6449: URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-764818323
@fx19880617 Thanks for the details. I just keep monitoring the system and the memory for the System was going through the edge. I noticed that once it happens and If i leave it for say couple of days following errors will appear and Pinot server will stop ingestion completely. Is there any way to fix this issue?  ``` my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.5, Avg: 1.6, Max: 1.8, Diff: 0.3, Sum: 9.4] 2021/01/21 17:02:38.123 ERROR [ServerSegmentCompletionProtocolHandler] [mylog__0__32545__20210121T1702Z] Could not send request http://my-pinot-controller-0.my-pinot-controller-headless.logging.svc.cluster.local:9000/segmentCommit?name=mylog__0__32545__20210121T1702Z&offset=44504082&instance=Server_my-pinot-server-1.my-pinot-server-headless.logging.svc.cluster.local_8098&buildTimeMillis=191&memoryUsedBytes=256504&segmentSizeBytes=2763568&rowCount=1000&streamPartitionMsgOffset=44504082 my-pinot-server-1 my-pinot-server-0server server [Update RS (ms): Min: 5.6, Avg: 5.7, Max: 5.7, Diff: 0.1, Sum: 34.1] my-pinot-controller-1 shaded.org.apache.http.NoHttpResponseException: my-pinot-controller-0.my-pinot-controller-headless.logging.svc.cluster.local:9000 failed to respond my-pinot-server-1my-pinot-server-0 my-pinot-broker-0 server [Processed Buffers: Min: 17, Avg: 27.3, Max: 34, Diff: 17, Sum: 164] at shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-a36d1c04c720c0f47dfec078c24c84c98ed665c4] my-pinot-broker-0 my-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. broker [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-1 my-pinot-broker-0 server broker [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 2] my-pinot-broker-0 my-pinot-controller-1 server [Choose CSet: 0.0 ms] my-pinot-server-0 broker controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. my-pinot-controller-1 controller 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. ... 16 more my-pinot-controller-1server [Ref Proc: 3.4 ms] controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-server-1 server 2021/01/21 17:04:50.874 ERROR [LLRealtimeSegmentDataManager_mylog__0__32545__20210121T1702Z] [mylog__0__32545__20210121T1702Z] Holding after response from Controller: {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"NOT_SENT","offset":-1} my-pinot-server-1 2021/01/21 17:03:09.455 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(bbae15c7_DEFAULT)] Event bbae15c7_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time. server my-pinot-server-0 server [Ref Enq: 0.0 ms] my-pinot-server-1 controller 2021-01-21T17:25:03.929+0000: 1350.323: Application time: 0.0590331 seconds my-pinot-server-0 my-pinot-controller-1 server controller 2021-01-21T17:25:03.930+0000: 1350.323: [GC pause (G1 Evacuation Pause) (mixed), 0.0406644 secs] server [Times: user=0.26 sys=0.00, real=0.05 secs] my-pinot-server-0 2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29726__20210121T1102Z server 2021-01-21T17:25:05.778+0000: 166.983: Total time for which application threads were stopped: 0.0500843 seconds, Stopping threads took: 0.0000282 seconds my-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: [GC concurrent-mark-end, 1.4470120 secs] my-pinot-server-0 server 2021-01-21T17:25:06.218+0000: 167.423: Application time: 0.4400702 seconds my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29727__20210121T1102Z my-pinot-server-1 server 2021-01-21T17:25:06.219+0000: 167.424: [GC remark 2021-01-21T17:25:06.219+0000: 167.424: [Finalize Marking, 0.0002085 secs] 2021-01-21T17:25:06.219+0000: 167.424: [GC ref-proc, 0.0851322 secs] 2021-01-21T17:25:06.304+0000: 167.509: [Unloading, 0.0080895 secs], 0.0963586 secs] 2021/01/21 17:22:51.941 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29728__20210121T1102Z my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29729__20210121T1102Z my-pinot-server-1my-pinot-server-0 server my-pinot-controller-1 [Times: user=0.11 sys=0.04, real=0.10 secs] my-pinot-server-0server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29730__20210121T1102Z my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29731__20210121T1102Z my-pinot-server-1 server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29732__20210121T1102Z controller [Parallel Time: 39.8 ms, GC Workers: 4] my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350323.3, Avg: 1350323.3, Max: 1350323.3, Diff: 0.0] server 2021-01-21T17:25:06.315+0000: 167.520: Total time for which application threads were stopped: 0.0972538 seconds, Stopping threads took: 0.0003432 seconds my-pinot-server-0 server my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 0.9, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.0] my-pinot-controller-1 my-pinot-server-1 2021-01-21T17:25:06.315+0000: 167.520: Application time: 0.0001148 seconds servercontroller [Update RS (ms): Min: 15.6, Avg: 15.6, Max: 15.6, Diff: 0.0, Sum: 62.4] my-pinot-controller-1 my-pinot-server-0 servercontroller [Processed Buffers: Min: 76, Avg: 77.8, Max: 79, Diff: 3, Sum: 311] my-pinot-controller-1 controller [Scan RS (ms): Min: 1.7, Avg: 1.7, Max: 1.7, Diff: 0.1, Sum: 6.8] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 2021-01-21T17:25:06.316+0000: 167.521: [GC cleanup 3168M->3047M(4096M), 0.0040580 secs] my-pinot-server-0 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29733__20210121T1102Z my-pinot-server-1server [Times: user=0.02 sys=0.00, real=0.00 secs] my-pinot-server-0 server server 2021/01/21 17:22:51.942 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29734__20210121T1103Z controller [Object Copy (ms): Min: 21.3, Avg: 21.4, Max: 21.5, Diff: 0.2, Sum: 85.6] my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29735__20210121T1103Z my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29736__20210121T1103Z controller2021-01-21T17:25:06.320+0000: 167.525: Total time for which application threads were stopped: 0.0046889 seconds, Stopping threads took: 0.0001052 seconds my-pinot-server-0 my-pinot-server-1server 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-start] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-controller-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29737__20210121T1103Z controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29738__20210121T1103Z my-pinot-controller-1 server my-pinot-server-1 2021-01-21T17:25:06.320+0000: 167.525: [GC concurrent-cleanup-end, 0.0001522 secs] my-pinot-server-0server server controller [GC Worker Total (ms): Min: 39.7, Avg: 39.7, Max: 39.7, Diff: 0.0, Sum: 158.9] 2021/01/21 17:22:51.943 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29739__20210121T1103Z my-pinot-controller-1 2021-01-21T17:25:06.929+0000: 168.134: Application time: 0.6090475 seconds my-pinot-server-0 my-pinot-server-1 server controller [GC Worker End (ms): Min: 1350363.0, Avg: 1350363.0, Max: 1350363.0, Diff: 0.0] my-pinot-controller-1 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29740__20210121T1103Z my-pinot-server-1 serverserver 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29741__20210121T1103Z my-pinot-server-1 controller [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controller 2021-01-21T17:25:06.930+0000: 168.135: [GC pause (G1 Evacuation Pause) (young), 0.0357839 secs] my-pinot-server-0 server [Parallel Time: 32.1 ms, GC Workers: 6] my-pinot-server-0 server [GC Worker Start (ms): Min: 168134.9, Avg: 168134.9, Max: 168134.9, Diff: 0.1] [Code Root Purge: 0.0 ms] my-pinot-controller-1my-pinot-server-0 server controller [Clear CT: 0.1 ms] my-pinot-controller-1 controller [Other: 0.8 ms] my-pinot-controller-1 controller server 2021/01/21 17:22:51.944 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29742__20210121T1104Z my-pinot-server-1 [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.5, Diff: 0.4, Sum: 7.9] my-pinot-server-0 server [Choose CSet: 0.1 ms] my-pinot-controller-1 controller [Ref Proc: 0.2 ms] my-pinot-controller-1 controller [Ref Enq: 0.0 ms] my-pinot-controller-1 controller server [Update RS (ms): Min: 14.8, Avg: 14.9, Max: 15.1, Diff: 0.3, Sum: 89.2] my-pinot-server-0 server [Processed Buffers: Min: 107, Avg: 122.2, Max: 132, Diff: 25, Sum: 733] my-pinot-server-0 server [Scan RS (ms): Min: 1.9, Avg: 2.1, Max: 2.2, Diff: 0.3, Sum: 12.5] my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Redirty Cards: 0.2 ms] my-pinot-controller-1 controller 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29743__20210121T1104Z my-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29744__20210121T1104Z [Humongous Register: 0.0 ms] my-pinot-controller-1 my-pinot-server-0 server [Object Copy (ms): Min: 13.5, Avg: 13.7, Max: 13.8, Diff: 0.3, Sum: 82.1] my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] controller [Humongous Reclaim: 0.0 ms] my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.945 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29745__20210121T1104Z my-pinot-server-0 my-pinot-server-1 [Free CSet: 0.1 ms] my-pinot-controller-1 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.3] servermy-pinot-server-0controller server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29746__20210121T1104Z my-pinot-server-1 [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 466.6M(1024.0M)->412.2M(1024.0M)] my-pinot-controller-1 server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29747__20210121T1104Z my-pinot-server-1 server 2021/01/21 17:22:51.946 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29748__20210121T1105Z [GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.1, Diff: 0.1, Sum: 192.1] my-pinot-server-1 my-pinot-server-0 server 2021/01/21 17:22:51.947 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29749__20210121T1105Z my-pinot-server-1 server server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29750__20210121T1105Z controller [Times: user=0.16 sys=0.00, real=0.04 secs] [GC Worker End (ms): Min: 168166.9, Avg: 168166.9, Max: 168166.9, Diff: 0.1] my-pinot-controller-1 my-pinot-server-1 controller 2021-01-21T17:25:03.971+0000: 1350.364: Total time for which application threads were stopped: 0.0412316 seconds, Stopping threads took: 0.0000680 seconds my-pinot-controller-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29751__20210121T1105Z my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29752__20210121T1105Z my-pinot-server-1 servermy-pinot-server-0 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29753__20210121T1105Z server [Code Root Fixup: 0.0 ms] my-pinot-server-0 controller 2021-01-21T17:25:04.025+0000: 1350.418: Application time: 0.0544185 seconds my-pinot-controller-1 controllerserver 2021-01-21T17:25:04.025+0000: 1350.419: [GC pause (G1 Evacuation Pause) (mixed), 0.0437685 secs] [Code Root Purge: 0.0 ms] my-pinot-server-0 server [Clear CT: 0.5 ms] my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29754__20210121T1105Z my-pinot-server-1 server 2021/01/21 17:22:51.948 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29755__20210121T1105Z my-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29756__20210121T1106Z my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29757__20210121T1106Z server [Other: 3.2 ms] my-pinot-server-1 server 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29758__20210121T1106Z my-pinot-server-1 my-pinot-server-0 controller server [Choose CSet: 0.0 ms] [Parallel Time: 43.0 ms, GC Workers: 4] my-pinot-server-0 server [Ref Proc: 1.6 ms] servermy-pinot-server-0 server [Ref Enq: 0.0 ms] 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29759__20210121T1106Z my-pinot-server-1 my-pinot-server-0 serverserver [Redirty Cards: 0.1 ms] my-pinot-controller-1 my-pinot-server-0 controller 2021/01/21 17:22:51.949 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29760__20210121T1106Z my-pinot-server-1 server 2021/01/21 17:22:51.950 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29761__20210121T1106Z my-pinot-server-1 [GC Worker Start (ms): Min: 1350418.9, Avg: 1350418.9, Max: 1350418.9, Diff: 0.0] my-pinot-controller-1 server controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.1, Sum: 4.1] my-pinot-controller-1 controller [Update RS (ms): Min: 23.6, Avg: 23.6, Max: 23.6, Diff: 0.0, Sum: 94.5] my-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 106.8, Max: 109, Diff: 6, Sum: 427] server2021/01/21 17:22:51.950 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29762__20210121T1106Z my-pinot-server-1 server 2021/01/21 17:22:51.951 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29763__20210121T1107Z my-pinot-server-1 [Humongous Register: 0.2 ms] server 2021/01/21 17:22:51.951 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29764__20210121T1107Z my-pinot-server-1 my-pinot-controller-1 server controller [Scan RS (ms): Min: 0.2, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.8] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29765__20210121T1107Z controller my-pinot-server-0 my-pinot-server-1 server server [Humongous Reclaim: 0.3 ms] [Object Copy (ms): Min: 18.0, Avg: 18.1, Max: 18.2, Diff: 0.1, Sum: 72.4] my-pinot-server-0 my-pinot-controller-1 controller server [Free CSet: 0.8 ms] my-pinot-server-0 [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29766__20210121T1107Z my-pinot-server-1 server server2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29767__20210121T1107Z my-pinot-server-1 controller server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29768__20210121T1107Z my-pinot-server-1 [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [GC Worker Total (ms): Min: 43.0, Avg: 43.0, Max: 43.0, Diff: 0.0, Sum: 171.9] server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29769__20210121T1107Z my-pinot-server-1 [Eden: 900.0M(900.0M)->0.0B(167.0M) Survivors: 32768.0K->37888.0K Heap: 3705.3M(4096.0M)->2630.9M(4096.0M)] my-pinot-server-0 server my-pinot-controller-1 controller server [GC Worker End (ms): Min: 1350461.9, Avg: 1350461.9, Max: 1350461.9, Diff: 0.0] my-pinot-controller-1 controller 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29770__20210121T1108Z my-pinot-server-1 [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controller [Code Root Purge: 0.0 ms] my-pinot-controller-1 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29771__20210121T1108Z my-pinot-server-1 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29772__20210121T1108Z [Times: user=0.19 sys=0.00, real=0.03 secs] my-pinot-server-1my-pinot-server-0 server 2021/01/21 17:22:51.952 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29773__20210121T1108Z servermy-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29774__20210121T1108Z my-pinot-server-1 controllerserver 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29775__20210121T1108Z [Clear CT: 0.1 ms] my-pinot-controller-1 controller 2021-01-21T17:25:06.966+0000: 168.170: Total time for which application threads were stopped: 0.0366911 seconds, Stopping threads took: 0.0002408 seconds my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29776__20210121T1108Z my-pinot-server-1 server [Other: 0.6 ms] my-pinot-controller-1 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29777__20210121T1109Z my-pinot-server-1 controller [Choose CSet: 0.0 ms] my-pinot-controller-1 server controller 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29778__20210121T1109Z my-pinot-server-1 2021-01-21T17:25:07.002+0000: 168.207: Application time: 0.0364979 seconds my-pinot-server-0 server 2021-01-21T17:25:07.004+0000: 168.208: [GC pause (G1 Evacuation Pause) (mixed), 0.0506602 secs] my-pinot-server-0 server [Parallel Time: 47.3 ms, GC Workers: 6] [Ref Proc: 0.1 ms] my-pinot-server-0 server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29779__20210121T1109Z server [GC Worker Start (ms): Min: 168208.9, Avg: 168208.9, Max: 168208.9, Diff: 0.1] my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.4, Max: 1.6, Diff: 0.3, Sum: 8.3] my-pinot-server-0 my-pinot-controller-1 controller [Ref Enq: 0.0 ms] my-pinot-controller-1 controller [Redirty Cards: 0.1 ms] my-pinot-controller-1 controllermy-pinot-server-1 [Humongous Register: 0.0 ms] my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] my-pinot-controller-1server 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29780__20210121T1109Z my-pinot-server-1 server server [Update RS (ms): Min: 3.5, Avg: 12.9, Max: 31.6, Diff: 28.1, Sum: 77.6] my-pinot-server-0 2021/01/21 17:22:51.953 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29781__20210121T1109Z my-pinot-server-1 controller server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29782__20210121T1109Z my-pinot-server-1 serverserver [Processed Buffers: Min: 6, Avg: 14.3, Max: 31, Diff: 25, Sum: 86] 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29783__20210121T1109Z my-pinot-server-0 server [Scan RS (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: 0.8, Sum: 2.8] my-pinot-server-1my-pinot-server-0 [Free CSet: 0.1 ms] my-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(45056.0K) Survivors: 7168.0K->7168.0K Heap: 470.8M(1024.0M)->444.5M(1024.0M)] my-pinot-controller-1 controller [Times: user=0.17 sys=0.00, real=0.05 secs] my-pinot-controller-1server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 controller server [Object Copy (ms): Min: 14.1, Avg: 32.3, Max: 41.4, Diff: 27.4, Sum: 193.9] my-pinot-server-0 2021-01-21T17:25:04.069+0000: 1350.463: Total time for which application threads were stopped: 0.0443644 seconds, Stopping threads took: 0.0001010 seconds my-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29784__20210121T1109Z my-pinot-server-1server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29785__20210121T1110Z my-pinot-server-1controller server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29786__20210121T1110Z my-pinot-server-1 servermy-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29787__20210121T1110Z my-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29788__20210121T1110Z my-pinot-server-1 2021-01-21T17:25:04.093+0000: 1350.486: Application time: 0.0234818 seconds my-pinot-controller-1 [GC Worker Total (ms): Min: 47.1, Avg: 47.1, Max: 47.1, Diff: 0.1, Sum: 282.8] my-pinot-server-0 server [GC Worker End (ms): Min: 168256.0, Avg: 168256.0, Max: 168256.1, Diff: 0.0] my-pinot-server-0 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29789__20210121T1110Z my-pinot-server-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29790__20210121T1110Z controller my-pinot-server-1 server 2021-01-21T17:25:04.093+0000: 1350.487: Total time for which application threads were stopped: 0.0004505 seconds, Stopping threads took: 0.0000400 seconds my-pinot-controller-1 server [Code Root Fixup: 0.0 ms] my-pinot-server-0 server [Code Root Purge: 0.0 ms] my-pinot-server-0 server [Clear CT: 0.5 ms] my-pinot-server-0 server [Other: 2.9 ms] my-pinot-server-0 server [Choose CSet: 0.2 ms] my-pinot-server-0 server [Ref Proc: 1.1 ms] my-pinot-server-02021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29791__20210121T1110Z my-pinot-server-1 controller 2021-01-21T17:25:04.093+0000: 1350.487: Application time: 0.0000469 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.094+0000: 1350.487: [GC pause (GCLocker Initiated GC) (young), 0.0285995 secs] my-pinot-controller-1 server controller [Parallel Time: 27.9 ms, GC Workers: 4] my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350487.1, Avg: 1350487.1, Max: 1350487.1, Diff: 0.0] 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29792__20210121T1111Z my-pinot-server-1 server [Ref Enq: 0.0 ms] my-pinot-server-0 my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.1, Diff: 0.2, Sum: 4.1] server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29793__20210121T1111Z my-pinot-server-1 server my-pinot-controller-1 server controller [Update RS (ms): Min: 21.6, Avg: 21.7, Max: 21.9, Diff: 0.2, Sum: 86.8] 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29794__20210121T1111Z my-pinot-controller-1 my-pinot-server-1 controller [Processed Buffers: Min: 85, Avg: 88.0, Max: 90, Diff: 5, Sum: 352] my-pinot-controller-1 [Redirty Cards: 0.1 ms] controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29795__20210121T1111Z my-pinot-controller-1 my-pinot-server-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 server 2021/01/21 17:22:51.954 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29796__20210121T1111Z my-pinot-server-1 server 2021/01/21 17:22:51.955 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29797__20210121T1111Z my-pinot-server-1 server my-pinot-server-0 controllerserver [Humongous Register: 0.1 ms] 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29798__20210121T1111Z my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29799__20210121T1111Z my-pinot-server-1 my-pinot-server-0 server [Object Copy (ms): Min: 4.9, Avg: 5.1, Max: 5.2, Diff: 0.3, Sum: 20.4] [Humongous Reclaim: 0.1 ms] my-pinot-controller-1 my-pinot-server-0 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29800__20210121T1111Z my-pinot-server-1 server server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29801__20210121T1111Z my-pinot-server-1 server my-pinot-controller-1 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29802__20210121T1112Z controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-server-1 [Free CSet: 0.9 ms] my-pinot-server-0 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29803__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29804__20210121T1112Z my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29805__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.956 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29806__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29807__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29808__20210121T1112Z my-pinot-server-1 server server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29809__20210121T1112Z controller [GC Worker Total (ms): Min: 27.8, Avg: 27.9, Max: 27.9, Diff: 0.0, Sum: 111.4] my-pinot-controller-1 [Eden: 167.0M(167.0M)->0.0B(200.0M) Survivors: 37888.0K->4096.0K Heap: 2797.9M(4096.0M)->2222.6M(4096.0M)] controller [GC Worker End (ms): Min: 1350515.0, Avg: 1350515.0, Max: 1350515.0, Diff: 0.0] my-pinot-controller-1 my-pinot-server-0 server [Times: user=0.14 sys=0.00, real=0.05 secs] my-pinot-server-0 controller [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controller [Code Root Purge: 0.0 ms] my-pinot-controller-1 my-pinot-server-1 controller [Clear CT: 0.1 ms] my-pinot-controller-1 controller [Other: 0.6 ms] my-pinot-controller-1 controller [Choose CSet: 0.0 ms] server server 2021-01-21T17:25:07.054+0000: 168.259: Total time for which application threads were stopped: 0.0522104 seconds, Stopping threads took: 0.0008706 seconds my-pinot-server-0 server 2021-01-21T17:25:07.304+0000: 168.509: Application time: 0.2499433 seconds my-pinot-server-0 my-pinot-controller-1 server 2021-01-21T17:25:07.305+0000: 168.510: [GC pause (G1 Evacuation Pause) (mixed), 0.0317508 secs] controller [Ref Proc: 0.1 ms] my-pinot-controller-1 controller [Ref Enq: 0.0 ms] my-pinot-server-0 server my-pinot-controller-1 controller [Redirty Cards: 0.1 ms] [Parallel Time: 29.1 ms, GC Workers: 6] my-pinot-controller-1 controller 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29810__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29811__20210121T1112Z [Humongous Register: 0.0 ms] my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] my-pinot-controller-1 controller my-pinot-server-0 server [GC Worker Start (ms): Min: 168510.7, Avg: 168510.7, Max: 168510.7, Diff: 0.1] my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.2, Avg: 1.3, Max: 1.4, Diff: 0.2, Sum: 7.9] my-pinot-server-1 [Free CSet: 0.1 ms] my-pinot-controller-1 controller [Eden: 45056.0K(45056.0K)->0.0B(91136.0K) Survivors: 7168.0K->2048.0K Heap: 493.8M(1024.0M)->450.0M(1024.0M)] my-pinot-server-0 my-pinot-controller-1 server controller [Times: user=0.11 sys=0.00, real=0.03 secs] my-pinot-controller-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29812__20210121T1112Z my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29813__20210121T1112Z controller 2021-01-21T17:25:04.122+0000: 1350.516: Total time for which application threads were stopped: 0.0289920 seconds, Stopping threads took: 0.0000117 seconds my-pinot-server-1 server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29814__20210121T1112Z my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29815__20210121T1113Z my-pinot-server-1 [Update RS (ms): Min: 22.7, Avg: 22.7, Max: 22.8, Diff: 0.1, Sum: 136.4] my-pinot-server-0 server controller server 2021/01/21 17:22:51.957 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29816__20210121T1113Z [Processed Buffers: Min: 90, Avg: 107.3, Max: 116, Diff: 26, Sum: 644] my-pinot-server-0 server [Scan RS (ms): Min: 1.2, Avg: 1.2, Max: 1.2, Diff: 0.1, Sum: 7.3] my-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29817__20210121T1113Z my-pinot-server-1 my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Object Copy (ms): Min: 3.5, Avg: 3.7, Max: 3.8, Diff: 0.2, Sum: 22.1] server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29818__20210121T1113Z my-pinot-server-1 server 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29819__20210121T1113Z my-pinot-server-1 2021-01-21T17:25:04.125+0000: 1350.518: Application time: 0.0027123 seconds my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller server 2021-01-21T17:25:04.125+0000: 1350.519: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0110860 secs] my-pinot-controller-1 my-pinot-server-0 2021/01/21 17:22:51.958 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29820__20210121T1113Z server [Termination Attempts: Min: 1, Avg: 1.3, Max: 2, Diff: 1, Sum: 8] my-pinot-server-0 server my-pinot-server-1 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] controller [Parallel Time: 10.5 ms, GC Workers: 4] my-pinot-controller-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29821__20210121T1114Z my-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29822__20210121T1114Z controller [GC Worker Start (ms): Min: 1350519.1, Avg: 1350519.1, Max: 1350519.1, Diff: 0.0] my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.2, Max: 1.4, Diff: 0.2, Sum: 5.0] my-pinot-controller-1 my-pinot-server-1 my-pinot-server-0 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29823__20210121T1114Z my-pinot-server-1 server 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29824__20210121T1114Z servermy-pinot-server-1controller [Update RS (ms): Min: 8.3, Avg: 8.3, Max: 8.3, Diff: 0.0, Sum: 33.2] my-pinot-controller-1 controller [GC Worker Total (ms): Min: 28.9, Avg: 29.0, Max: 29.0, Diff: 0.0, Sum: 173.8] my-pinot-server-0 [Processed Buffers: Min: 37, Avg: 38.8, Max: 41, Diff: 4, Sum: 155] my-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server [GC Worker End (ms): Min: 168539.7, Avg: 168539.7, Max: 168539.7, Diff: 0.0] my-pinot-server-0 my-pinot-controller-1 controller [Object Copy (ms): Min: 0.7, Avg: 0.8, Max: 0.9, Diff: 0.2, Sum: 3.3] my-pinot-controller-1 servercontroller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 5] my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29825__20210121T1114Z server [Code Root Fixup: 0.0 ms] my-pinot-server-0 server [Code Root Purge: 0.0 ms] my-pinot-server-0 [GC Worker Total (ms): Min: 10.4, Avg: 10.4, Max: 10.4, Diff: 0.0, Sum: 41.5] my-pinot-controller-1 my-pinot-server-1 server controller [GC Worker End (ms): Min: 1350529.5, Avg: 1350529.5, Max: 1350529.5, Diff: 0.0] server [Clear CT: 0.4 ms] 2021/01/21 17:22:51.959 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29826__20210121T1114Z my-pinot-server-0 my-pinot-server-1 my-pinot-controller-1 controller server [Code Root Fixup: 0.0 ms] [Other: 2.3 ms] my-pinot-controller-1 controller [Code Root Purge: 0.0 ms] my-pinot-controller-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29827__20210121T1114Z my-pinot-server-1 server my-pinot-server-0 server [Choose CSet: 0.2 ms] my-pinot-server-0 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29828__20210121T1114Z controller my-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29829__20210121T1115Z my-pinot-server-1 server 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29830__20210121T1115Z my-pinot-server-1 server [Ref Proc: 0.3 ms] [Clear CT: 0.1 ms] my-pinot-server-0 server my-pinot-controller-1 controller [Other: 0.5 ms] my-pinot-controller-1 controller 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29831__20210121T1115Z my-pinot-server-1 serverserver [Ref Enq: 0.0 ms] my-pinot-server-0 2021/01/21 17:22:51.960 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29832__20210121T1115Z server my-pinot-server-1 server 2021/01/21 17:22:51.961 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29833__20210121T1115Z my-pinot-server-1 server [Redirty Cards: 0.1 ms] my-pinot-server-0 [Choose CSet: 0.0 ms] 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29834__20210121T1115Z my-pinot-server-1 server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29835__20210121T1115Z my-pinot-server-1 my-pinot-controller-1 controller server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29836__20210121T1115Z my-pinot-server-1 server [Humongous Register: 0.1 ms] server 2021/01/21 17:22:51.962 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29837__20210121T1115Z my-pinot-server-0 my-pinot-server-1 serverserver 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29838__20210121T1116Z my-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29839__20210121T1116Z my-pinot-server-1 [Humongous Reclaim: 0.1 ms] server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29840__20210121T1116Z my-pinot-server-1 server 2021/01/21 17:22:51.963 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29841__20210121T1116Z my-pinot-server-0 [Ref Proc: 0.1 ms] server [Free CSet: 1.1 ms] my-pinot-server-0 server [Eden: 200.0M(200.0M)->0.0B(195.0M) Survivors: 4096.0K->9216.0K Heap: 2471.9M(4096.0M)->1820.5M(4096.0M)] my-pinot-server-0 server my-pinot-server-1 [Times: user=0.18 sys=0.01, real=0.03 secs] server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29842__20210121T1116Z my-pinot-server-1 server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29843__20210121T1116Z my-pinot-controller-1 my-pinot-server-0 my-pinot-server-1server 2021-01-21T17:25:07.337+0000: 168.542: Total time for which application threads were stopped: 0.0329319 seconds, Stopping threads took: 0.0005259 seconds my-pinot-server-0 server controller [Ref Enq: 0.0 ms] my-pinot-controller-1server 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29844__20210121T1116Z my-pinot-server-1 server controller 2021-01-21T17:25:07.632+0000: 168.836: Application time: 0.2943829 seconds my-pinot-server-0 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29845__20210121T1117Z my-pinot-server-1 server server 2021-01-21T17:25:07.632+0000: 168.837: [GC pause (G1 Evacuation Pause) (mixed), 0.0321493 secs] [Redirty Cards: 0.1 ms] my-pinot-server-0 server [Parallel Time: 28.6 ms, GC Workers: 6] my-pinot-server-0 server [GC Worker Start (ms): Min: 168838.0, Avg: 168838.0, Max: 168838.0, Diff: 0.1] my-pinot-controller-1 controller [Humongous Register: 0.0 ms] my-pinot-controller-1 controller [Humongous Reclaim: 0.0 ms] my-pinot-server-0 my-pinot-controller-1 server [Ext Root Scanning (ms): Min: 1.3, Avg: 1.3, Max: 1.5, Diff: 0.2, Sum: 8.1] my-pinot-server-0 server [Update RS (ms): Min: 9.7, Avg: 9.8, Max: 9.8, Diff: 0.1, Sum: 58.6] my-pinot-server-0 controllerserver [Free CSet: 0.0 ms] [Processed Buffers: Min: 67, Avg: 79.5, Max: 89, Diff: 22, Sum: 477] 2021/01/21 17:22:51.964 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29846__20210121T1117Z my-pinot-server-1 my-pinot-server-0 my-pinot-controller-1 server [Scan RS (ms): Min: 1.8, Avg: 1.8, Max: 1.9, Diff: 0.1, Sum: 10.9] server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29847__20210121T1117Z controller [Eden: 5120.0K(91136.0K)->0.0B(152.0M) Survivors: 2048.0K->3072.0K Heap: 461.0M(1024.0M)->453.5M(1024.0M)] my-pinot-server-0 my-pinot-controller-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Object Copy (ms): Min: 15.4, Avg: 15.5, Max: 15.6, Diff: 0.2, Sum: 92.9] my-pinot-server-0 servermy-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29848__20210121T1117Z my-pinot-server-1 server 2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29849__20210121T1117Z my-pinot-server-1 controllerserver [Times: user=0.04 sys=0.00, real=0.01 secs] my-pinot-controller-1 controller2021/01/21 17:22:51.965 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29850__20210121T1117Z 2021-01-21T17:25:04.137+0000: 1350.530: [GC concurrent-root-region-scan-start] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 my-pinot-server-0 controller 2021-01-21T17:25:04.137+0000: 1350.530: Total time for which application threads were stopped: 0.0116309 seconds, Stopping threads took: 0.0001002 seconds server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] my-pinot-server-0 my-pinot-server-1my-pinot-controller-1 controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-root-region-scan-end, 0.0031610 secs] server 2021/01/21 17:22:51.966 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29851__20210121T1117Z my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.966 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29852__20210121T1118Z my-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29853__20210121T1118Z server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-server-1controller 2021-01-21T17:25:04.140+0000: 1350.533: [GC concurrent-mark-start] my-pinot-server-0 server [GC Worker Total (ms): Min: 28.4, Avg: 28.4, Max: 28.5, Diff: 0.1, Sum: 170.6] my-pinot-server-0 server my-pinot-controller-1 controller server 2021-01-21T17:25:04.256+0000: 1350.649: Application time: 0.1189493 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.256+0000: 1350.649: [GC pause (G1 Evacuation Pause) (young), 0.0236359 secs] my-pinot-controller-1 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29854__20210121T1118Z controller [Parallel Time: 23.0 ms, GC Workers: 4] my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1350649.5, Avg: 1350649.5, Max: 1350649.5, Diff: 0.1] [GC Worker End (ms): Min: 168866.4, Avg: 168866.4, Max: 168866.4, Diff: 0.0] my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29855__20210121T1118Z [Ext Root Scanning (ms): Min: 1.0, Avg: 1.0, Max: 1.2, Diff: 0.2, Sum: 4.2] my-pinot-controller-1 controller [Update RS (ms): Min: 6.5, Avg: 6.6, Max: 6.6, Diff: 0.1, Sum: 26.2] my-pinot-controller-1 controller [Processed Buffers: Min: 65, Avg: 70.0, Max: 74, Diff: 9, Sum: 280] my-pinot-controller-1 controller [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Object Copy (ms): Min: 15.2, Avg: 15.2, Max: 15.3, Diff: 0.2, Sum: 61.0] my-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-controller-1 controllermy-pinot-server-1 server 2021/01/21 17:22:51.967 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29856__20210121T1118Z my-pinot-server-0 my-pinot-server-1server [Code Root Fixup: 0.0 ms] my-pinot-server-0 [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-controller-1 controller [GC Worker Total (ms): Min: 22.9, Avg: 22.9, Max: 22.9, Diff: 0.0, Sum: 91.6] server server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29857__20210121T1118Z my-pinot-server-1 [Code Root Purge: 0.0 ms] my-pinot-server-0 server [Clear CT: 0.4 ms] my-pinot-server-0 server [Other: 3.2 ms] my-pinot-server-0 server [Choose CSet: 0.4 ms] my-pinot-server-0 server [Ref Proc: 0.8 ms] my-pinot-server-0 server [Ref Enq: 0.0 ms] my-pinot-server-0 server [Redirty Cards: 0.1 ms] my-pinot-server-0 server [Humongous Register: 0.1 ms] my-pinot-server-0 server [Humongous Reclaim: 0.1 ms] my-pinot-server-0 server [Free CSet: 1.2 ms] my-pinot-server-0 server [Eden: 195.0M(195.0M)->0.0B(192.0M) Survivors: 9216.0K->12288.0K Heap: 2064.8M(4096.0M)->1443.5M(4096.0M)] my-pinot-server-0 server [Times: user=0.17 sys=0.00, real=0.03 secs] my-pinot-server-0 server 2021-01-21T17:25:07.665+0000: 168.870: Total time for which application threads were stopped: 0.0330694 seconds, Stopping threads took: 0.0002922 seconds my-pinot-server-0 server 2021-01-21T17:25:07.829+0000: 169.034: Application time: 0.1644559 seconds my-pinot-server-0 server 2021-01-21T17:25:07.838+0000: 169.043: [GC pause (G1 Evacuation Pause) (mixed), 0.0260221 secs] my-pinot-server-0 server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29858__20210121T1118Z my-pinot-server-1my-pinot-controller-1 server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29859__20210121T1119Z my-pinot-server-1 controller [GC Worker End (ms): Min: 1350672.4, Avg: 1350672.4, Max: 1350672.4, Diff: 0.0] my-pinot-controller-1 controller [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controller server 2021/01/21 17:22:51.969 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29860__20210121T1119Z my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29861__20210121T1119Z [Code Root Purge: 0.0 ms] my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29862__20210121T1119Z my-pinot-server-1 my-pinot-controller-1 controller [Clear CT: 0.1 ms] my-pinot-controller-1server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29863__20210121T1119Z my-pinot-server-1 server server [Parallel Time: 23.6 ms, GC Workers: 6] my-pinot-server-0 server [GC Worker Start (ms): Min: 169043.2, Avg: 169043.2, Max: 169043.3, Diff: 0.1] my-pinot-server-0 server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.5, Max: 1.6, Diff: 0.2, Sum: 8.9] my-pinot-server-0 server [Update RS (ms): Min: 13.8, Avg: 13.8, Max: 13.8, Diff: 0.1, Sum: 82.8] my-pinot-server-0 server [Processed Buffers: Min: 50, Avg: 54.3, Max: 58, Diff: 8, Sum: 326] my-pinot-server-0 server [Scan RS (ms): Min: 0.2, Avg: 0.3, Max: 0.3, Diff: 0.1, Sum: 1.6] 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29864__20210121T1119Z my-pinot-server-1my-pinot-server-0 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29865__20210121T1119Z my-pinot-server-1 server 2021/01/21 17:22:51.970 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29866__20210121T1120Z my-pinot-server-1 server controller2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29867__20210121T1120Z my-pinot-server-1 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Object Copy (ms): Min: 7.7, Avg: 7.9, Max: 8.0, Diff: 0.2, Sum: 47.2] my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.2] my-pinot-server-0 server [GC Worker Total (ms): Min: 23.4, Avg: 23.4, Max: 23.5, Diff: 0.1, Sum: 140.7] [Other: 0.6 ms] my-pinot-controller-1 server my-pinot-server-0 server controller [Choose CSet: 0.0 ms] my-pinot-controller-1 controller [Ref Proc: 0.1 ms] [GC Worker End (ms): Min: 169066.7, Avg: 169066.7, Max: 169066.7, Diff: 0.0] my-pinot-server-0 2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29868__20210121T1120Z my-pinot-server-1 server 2021/01/21 17:22:51.971 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29869__20210121T1120Z server my-pinot-server-1 [Code Root Fixup: 0.0 ms] my-pinot-server-0 server [Code Root Purge: 0.0 ms] my-pinot-server-0 server server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29870__20210121T1120Z my-pinot-server-1 my-pinot-controller-1 controller [Ref Enq: 0.0 ms] [Clear CT: 0.2 ms] my-pinot-controller-1 controller my-pinot-server-0 server [Redirty Cards: 0.1 ms] my-pinot-controller-1 [Other: 2.2 ms] my-pinot-server-0 controller server [Choose CSet: 0.0 ms] [Humongous Register: 0.0 ms] my-pinot-server-0 server [Ref Proc: 1.0 ms] my-pinot-server-0 server [Ref Enq: 0.0 ms] my-pinot-server-0 server [Redirty Cards: 0.1 ms] my-pinot-server-0 my-pinot-controller-1 controller server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29871__20210121T1120Z my-pinot-server-1 server [Humongous Register: 0.2 ms] my-pinot-server-0 server [Humongous Reclaim: 0.1 ms] my-pinot-server-0 server [Free CSet: 0.3 ms] server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29872__20210121T1120Z my-pinot-server-1 server 2021/01/21 17:22:51.972 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29873__20210121T1120Z my-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29874__20210121T1120Z my-pinot-server-1 server 2021/01/21 17:22:51.973 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29875__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.974 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29876__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29877__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29878__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29879__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.975 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29880__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29881__20210121T1121Z my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29882__20210121T1121Z my-pinot-server-1 [Humongous Reclaim: 0.1 ms] my-pinot-controller-1 controller [Free CSet: 0.1 ms] my-pinot-controller-1 controller [Eden: 152.0M(152.0M)->0.0B(298.0M) Survivors: 3072.0K->20480.0K Heap: 638.1M(1024.0M)->482.3M(1024.0M)] my-pinot-controller-1 controller [Times: user=0.09 sys=0.00, real=0.03 secs] my-pinot-controller-1 controller 2021-01-21T17:25:04.280+0000: 1350.673: Total time for which application threads were stopped: 0.0241836 seconds, Stopping threads took: 0.0000443 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.955: [GC concurrent-mark-end, 0.4221306 secs] my-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29883__20210121T1122Z my-pinot-server-0 my-pinot-server-1server [Eden: 192.0M(192.0M)->0.0B(2155.0M) Survivors: 12288.0K->23552.0K Heap: 1650.1M(4096.0M)->1449.3M(4096.0M)] my-pinot-server-0 controllerserver [Times: user=0.14 sys=0.00, real=0.03 secs] my-pinot-server-0 server 2021-01-21T17:25:07.864+0000: 169.069: Total time for which application threads were stopped: 0.0349989 seconds, Stopping threads took: 0.0082034 seconds my-pinot-server-0 server 2021-01-21T17:25:09.808+0000: 171.013: Application time: 1.9435757 seconds my-pinot-server-0 serverserver 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29884__20210121T1122Z my-pinot-server-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29885__20210121T1122Z 2021-01-21T17:25:09.813+0000: 171.018: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0428457 secs] my-pinot-server-0 2021-01-21T17:25:04.562+0000: 1350.955: Application time: 0.2822203 seconds server [Parallel Time: 37.6 ms, GC Workers: 6] my-pinot-server-0 my-pinot-controller-1 controller 2021-01-21T17:25:04.562+0000: 1350.956: [GC remark 2021-01-21T17:25:04.562+0000: 1350.956: [Finalize Marking, 0.0002598 secs] 2021-01-21T17:25:04.563+0000: 1350.956: [GC ref-proc, 0.0004448 secs] 2021-01-21T17:25:04.563+0000: 1350.957: [Unloading, 0.0095395 secs], 0.0108947 secs] my-pinot-controller-1 my-pinot-server-1 controller [Times: user=0.04 sys=0.00, real=0.01 secs] my-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Total time for which application threads were stopped: 0.0114314 seconds, Stopping threads took: 0.0000926 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.573+0000: 1350.967: Application time: 0.0000628 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.574+0000: 1350.967: [GC cleanup 774M->655M(1024M), 0.0018691 secs] my-pinot-controller-1 controller [Times: user=0.01 sys=0.00, real=0.00 secs] my-pinot-controller-1 server 2021/01/21 17:22:51.977 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29886__20210121T1122Z controller my-pinot-server-1 server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29887__20210121T1122Z server [GC Worker Start (ms): Min: 171018.3, Avg: 171018.4, Max: 171018.4, Diff: 0.1] my-pinot-server-0 my-pinot-server-1server [Ext Root Scanning (ms): Min: 1.4, Avg: 1.4, Max: 1.6, Diff: 0.2, Sum: 8.6] my-pinot-server-0 server [Update RS (ms): Min: 20.6, Avg: 20.7, Max: 20.8, Diff: 0.1, Sum: 124.3] my-pinot-server-0 server [Processed Buffers: Min: 144, Avg: 149.8, Max: 159, Diff: 15, Sum: 899] my-pinot-server-0 2021-01-21T17:25:04.576+0000: 1350.969: Total time for which application threads were stopped: 0.0023571 seconds, Stopping threads took: 0.0000678 seconds my-pinot-controller-1 server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29888__20210121T1122Z my-pinot-server-1 controllerserver [Scan RS (ms): Min: 0.4, Avg: 0.4, Max: 0.5, Diff: 0.1, Sum: 2.6] server 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-start] my-pinot-controller-1 my-pinot-server-0 server [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-server-0 server [Object Copy (ms): Min: 14.5, Avg: 14.7, Max: 14.8, Diff: 0.2, Sum: 88.2] my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-0 server [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 6] my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4] my-pinot-server-0 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29889__20210121T1122Z my-pinot-server-1 controller 2021-01-21T17:25:04.576+0000: 1350.969: [GC concurrent-cleanup-end, 0.0001739 secs] server 2021/01/21 17:22:51.978 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29890__20210121T1123Z my-pinot-controller-1 server [GC Worker Total (ms): Min: 37.3, Avg: 37.4, Max: 37.4, Diff: 0.1, Sum: 224.2] my-pinot-server-0 server [GC Worker End (ms): Min: 171055.7, Avg: 171055.7, Max: 171055.8, Diff: 0.1] my-pinot-server-0 server [Code Root Fixup: 0.0 ms] my-pinot-server-0 server controller 2021-01-21T17:25:04.654+0000: 1351.048: Application time: 0.0785958 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.655+0000: 1351.048: [GC pause (G1 Evacuation Pause) (young), 0.0666046 secs] my-pinot-server-1 [Code Root Purge: 0.0 ms] server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29891__20210121T1123Z my-pinot-server-1 my-pinot-controller-1 controller [Parallel Time: 65.5 ms, GC Workers: 4] my-pinot-controller-1 my-pinot-server-0 server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29892__20210121T1123Z server [Clear CT: 0.5 ms] my-pinot-server-0 server [Other: 4.8 ms] my-pinot-server-0 server [Choose CSet: 0.0 ms] my-pinot-server-0 server [Ref Proc: 2.2 ms] my-pinot-server-0 server [Ref Enq: 0.0 ms] my-pinot-server-0 server [Redirty Cards: 0.1 ms] my-pinot-server-0 server [Humongous Register: 0.1 ms] my-pinot-server-0 server [Humongous Reclaim: 0.6 ms] controller my-pinot-server-0 [GC Worker Start (ms): Min: 1351048.4, Avg: 1351048.4, Max: 1351048.4, Diff: 0.0] server [Free CSet: 1.0 ms] my-pinot-controller-1 my-pinot-server-1 my-pinot-server-0 server [Eden: 1386.0M(2155.0M)->0.0B(2134.0M) Survivors: 23552.0K->48128.0K Heap: 3188.4M(4096.0M)->1474.4M(4096.0M)] my-pinot-server-0 server [Times: user=0.23 sys=0.00, real=0.04 secs] my-pinot-server-0 server controller [Ext Root Scanning (ms): Min: 1.1, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.6] my-pinot-controller-1 controller server 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29893__20210121T1123Z 2021/01/21 17:25:10.641 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=mylog__0__13120__20210119T1725Z, expected=ONLINE, found=OFFLINE, creationTime=1611249776061, modifiedTime=1611249910406, version=1030, waitingFor=CurrentStateMatch, resource=mylog_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;; my-pinot-server-0 [Update RS (ms): Min: 16.0, Avg: 16.0, Max: 16.0, Diff: 0.1, Sum: 63.9] server 2021-01-21T17:25:09.856+0000: 2021-01-21T17:25:09.856+0000: 171.061: 171.061: [GC concurrent-root-region-scan-start] my-pinot-server-0 server my-pinot-controller-1 controller [Processed Buffers: Min: 103, Avg: 111.0, Max: 117, Diff: 14, Sum: 444] my-pinot-server-1 server Total time for which application threads were stopped: 0.0480709 seconds, Stopping threads took: 0.0044806 seconds my-pinot-server-0 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-root-region-scan-end, 0.0198202 secs] 2021/01/21 17:22:51.979 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29894__20210121T1123Z my-pinot-server-1 my-pinot-controller-1 controller [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.1, Diff: 0.0, Sum: 0.3] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1server 2021/01/21 17:22:51.980 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29895__20210121T1123Z my-pinot-server-1 server 2021/01/21 17:22:51.980 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29896__20210121T1123Z my-pinot-server-1 controller [Object Copy (ms): Min: 48.1, Avg: 48.2, Max: 48.3, Diff: 0.2, Sum: 192.8] server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29897__20210121T1124Z my-pinot-server-1 my-pinot-server-0servermy-pinot-controller-1 controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-controller-1 controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-controller-1 controller [GC Worker Total (ms): Min: 65.4, Avg: 65.4, Max: 65.4, Diff: 0.0, Sum: 261.7] my-pinot-controller-1 controller [GC Worker End (ms): Min: 1351113.8, Avg: 1351113.8, Max: 1351113.8, Diff: 0.0] my-pinot-controller-1 controller [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controller [Code Root Purge: 0.0 ms] my-pinot-controller-1 controller [Clear CT: 0.1 ms] my-pinot-controller-1 server 2021-01-21T17:25:09.876+0000: 171.080: [GC concurrent-mark-start] my-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: [GC concurrent-mark-end, 1.3475822 secs] 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29898__20210121T1124Z my-pinot-server-1 my-pinot-server-0 server 2021-01-21T17:25:11.223+0000: 172.428: Application time: 1.3674839 seconds server my-pinot-server-0 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29899__20210121T1124Z my-pinot-server-1 controller [Other: 1.0 ms] my-pinot-controller-1 controllerserver 2021-01-21T17:25:11.228+0000: 172.433: [GC remark 2021-01-21T17:25:11.228+0000: 172.433: [Finalize Marking, 0.0003440 secs] 2021-01-21T17:25:11.229+0000: 172.434: [GC ref-proc, 0.0850389 secs] 2021-01-21T17:25:11.314+0000: 172.519: [Unloading, 0.0079844 secs], 0.0957425 secs] my-pinot-server-0 server [Times: user=0.11 sys=0.03, real=0.10 secs] my-pinot-server-0 server 2021-01-21T17:25:11.324+0000: 172.529: Total time for which application threads were stopped: 0.1009431 seconds, Stopping threads took: 0.0045995 seconds server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29900__20210121T1124Z [Choose CSet: 0.0 ms] my-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29901__20210121T1124Z my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29902__20210121T1124Z my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29903__20210121T1125Z my-pinot-server-1 controller2021-01-21T17:25:11.324+0000: 172.529: Application time: 0.0000758 seconds my-pinot-server-0 server server 2021-01-21T17:25:11.325+0000: 172.530: [GC cleanup 2780M->2780M(4096M), 0.0035921 secs] my-pinot-server-0 [Ref Proc: 0.2 ms] my-pinot-controller-1 controller server [Ref Enq: 0.0 ms] my-pinot-controller-1 controller [Redirty Cards: 0.3 ms] [Times: user=0.02 sys=0.00, real=0.00 secs] my-pinot-controller-1 2021/01/21 17:22:51.981 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29904__20210121T1125Z my-pinot-server-1 servermy-pinot-server-0 server 2021-01-21T17:25:11.328+0000: 172.533: Total time for which application threads were stopped: 0.0041608 seconds, Stopping threads took: 0.0000167 seconds my-pinot-server-0 server 2021/01/21 17:22:51.982 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29905__20210121T1125Z my-pinot-server-1 2021-01-21T17:25:11.481+0000: 172.686: Application time: 0.1529441 seconds servercontrollermy-pinot-server-0 server 2021-01-21T17:25:11.482+0000: 172.687: Total time for which application threads were stopped: 0.0005239 seconds, Stopping threads took: 0.0000613 seconds my-pinot-server-0 [Humongous Register: 0.0 ms] servermy-pinot-controller-1 2021/01/21 17:22:51.982 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29906__20210121T1125Z my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29907__20210121T1125Z my-pinot-server-1 controller server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29908__20210121T1125Z my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29909__20210121T1125Z [Humongous Reclaim: 0.1 ms] my-pinot-controller-1my-pinot-server-1 server 2021/01/21 17:22:51.983 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29910__20210121T1125Z my-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29911__20210121T1126Z controller [Free CSet: 0.2 ms] my-pinot-controller-1 controller [Eden: 298.0M(298.0M)->0.0B(11264.0K) Survivors: 20480.0K->40960.0K Heap: 700.6M(1024.0M)->429.5M(1024.0M)] my-pinot-controller-1 controller [Times: user=0.26 sys=0.00, real=0.07 secs] my-pinot-controller-1 controller my-pinot-server-1 server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29912__20210121T1126Z my-pinot-server-1 server 2021-01-21T17:25:11.489+0000: 172.694: Application time: 0.0071873 seconds my-pinot-server-0 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29913__20210121T1126Z my-pinot-server-1 server 2021-01-21T17:25:11.490+0000: 172.694: Total time for which application threads were stopped: 0.0004430 seconds, Stopping threads took: 0.0000272 seconds my-pinot-server-0 server 2021-01-21T17:25:11.551+0000: 172.755: Application time: 0.0610437 seconds 2021-01-21T17:25:04.722+0000: 1351.115: Total time for which application threads were stopped: 0.0671829 seconds, Stopping threads took: 0.0000684 seconds my-pinot-controller-1 my-pinot-server-0controller 2021-01-21T17:25:04.737+0000: 1351.130: Application time: 0.0150208 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.737+0000: 1351.130: [GC pause (G1 Evacuation Pause) (mixed), 0.0775762 secs] server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29914__20210121T1126Z server 2021-01-21T17:25:11.552+0000: 172.757: Total time for which application threads were stopped: 0.0013581 seconds, Stopping threads took: 0.0000241 seconds my-pinot-server-0 server 2021-01-21T17:25:11.710+0000: 172.915: Application time: 0.1581764 seconds my-pinot-server-0 my-pinot-server-1 server my-pinot-controller-1 controller [Parallel Time: 76.1 ms, GC Workers: 4] my-pinot-controller-1 controller [GC Worker Start (ms): Min: 1351130.8, Avg: 1351130.8, Max: 1351130.8, Diff: 0.1] my-pinot-controller-1 controller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.2] my-pinot-controller-1 controller [Update RS (ms): Min: 31.3, Avg: 31.4, Max: 31.4, Diff: 0.1, Sum: 125.4] my-pinot-controller-1 controller [Processed Buffers: Min: 124, Avg: 125.0, Max: 126, Diff: 2, Sum: 500] my-pinot-controller-1 server controller 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29915__20210121T1126Z my-pinot-server-1 2021-01-21T17:25:11.718+0000: 172.923: Total time for which application threads were stopped: 0.0080070 seconds, Stopping threads took: 0.0074375 seconds my-pinot-server-0 server 2021-01-21T17:25:11.724+0000: 172.928: Application time: 0.0054288 seconds my-pinot-server-0server 2021/01/21 17:22:51.984 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29916__20210121T1126Z my-pinot-server-1 server 2021/01/21 17:22:51.986 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29917__20210121T1126Z my-pinot-server-1 [Scan RS (ms): Min: 13.4, Avg: 13.5, Max: 13.7, Diff: 0.3, Sum: 54.1] my-pinot-controller-1 controllerserver [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5] 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29918__20210121T1127Z my-pinot-controller-1 my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29919__20210121T1127Z controller [Object Copy (ms): Min: 29.8, Avg: 29.9, Max: 30.1, Diff: 0.3, Sum: 119.8] my-pinot-controller-1 server controller [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29920__20210121T1127Z my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29921__20210121T1127Z 2021-01-21T17:25:11.728+0000: 172.933: Total time for which application threads were stopped: 0.0044722 seconds, Stopping threads took: 0.0031338 seconds my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29922__20210121T1127Z my-pinot-server-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29923__20210121T1127Z my-pinot-controller-1 my-pinot-server-0 server controller [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4] my-pinot-controller-1 controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] my-pinot-controller-1 2021-01-21T17:25:11.732+0000: 172.937: Application time: 0.0044179 seconds my-pinot-server-0 controllerserver my-pinot-server-1 [GC Worker Total (ms): Min: 76.0, Avg: 76.0, Max: 76.0, Diff: 0.0, Sum: 304.1] my-pinot-controller-1 server 2021/01/21 17:22:51.987 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29924__20210121T1127Z my-pinot-server-1 controller [GC Worker End (ms): Min: 1351206.8, Avg: 1351206.8, Max: 1351206.8, Diff: 0.0] server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29925__20210121T1127Z my-pinot-server-1 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0004491 seconds, Stopping threads took: 0.0000215 seconds my-pinot-server-0 server my-pinot-controller-1 controller 2021-01-21T17:25:11.733+0000: 172.938: Application time: 0.0000363 seconds my-pinot-server-0 [Code Root Fixup: 0.0 ms] my-pinot-controller-1 controllerserver 2021-01-21T17:25:11.733+0000: 172.938: Total time for which application threads were stopped: 0.0003160 seconds, Stopping threads took: 0.0000079 seconds my-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Application time: 0.0124480 seconds [Code Root Purge: 0.0 ms] my-pinot-controller-1 server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29926__20210121T1128Z my-pinot-server-1 controller [Clear CT: 0.1 ms] server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29927__20210121T1128Z my-pinot-server-1 server my-pinot-server-0 server 2021-01-21T17:25:11.746+0000: 172.951: Total time for which application threads were stopped: 0.0003742 seconds, Stopping threads took: 0.0000195 seconds my-pinot-server-0 server 2021-01-21T17:25:11.759+0000: 172.964: Application time: 0.0131737 seconds my-pinot-server-02021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29928__20210121T1128Z my-pinot-server-1 server 2021/01/21 17:22:51.988 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29929__20210121T1128Z my-pinot-server-1 my-pinot-controller-1server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29930__20210121T1128Z my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29931__20210121T1128Z my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29932__20210121T1128Z my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29933__20210121T1129Z my-pinot-server-1 server 2021/01/21 17:22:51.989 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29934__20210121T1129Z server my-pinot-server-1 controller [Other: 1.3 ms] my-pinot-controller-1 controller [Choose CSet: 0.2 ms] my-pinot-controller-1 controller [Ref Proc: 0.1 ms] my-pinot-controller-1 controller [Ref Enq: 0.0 ms] my-pinot-controller-1 2021-01-21T17:25:11.760+0000: 172.965: Total time for which application threads were stopped: 0.0007156 seconds, Stopping threads took: 0.0002863 seconds controller [Redirty Cards: 0.4 ms] my-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Application time: 0.0087255 seconds my-pinot-server-0 server 2021-01-21T17:25:11.769+0000: 172.974: Total time for which application threads were stopped: 0.0007774 seconds, Stopping threads took: 0.0002464 seconds my-pinot-controller-1 controller [Humongous Register: 0.0 ms] my-pinot-controller-1 my-pinot-server-0 server 2021-01-21T17:25:11.771+0000: 172.976: Application time: 0.0015455 seconds controller my-pinot-server-0 server 2021-01-21T17:25:11.773+0000: 172.977: Total time for which application threads were stopped: 0.0015423 seconds, Stopping threads took: 0.0002360 seconds my-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: Application time: 0.5611696 seconds my-pinot-server-0 server 2021-01-21T17:25:12.334+0000: 173.539: [GC pause (G1 Evacuation Pause) (young), 0.2155960 secs] my-pinot-server-0 [Humongous Reclaim: 0.0 ms] servermy-pinot-controller-1 controller [Free CSet: 0.4 ms] my-pinot-controller-1 controller [Eden: 11264.0K(11264.0K)->0.0B(48128.0K) Survivors: 40960.0K->4096.0K Heap: 455.0M(1024.0M)->354.2M(1024.0M)] my-pinot-controller-1 [Parallel Time: 208.2 ms, GC Workers: 6] my-pinot-server-0 server server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29935__20210121T1129Z [GC Worker Start (ms): Min: 173539.5, Avg: 173539.5, Max: 173539.5, Diff: 0.1] my-pinot-server-1 controller server my-pinot-server-0 [Times: user=0.30 sys=0.00, real=0.07 secs] server [Ext Root Scanning (ms): Min: 1.1, Avg: 1.3, Max: 1.6, Diff: 0.4, Sum: 7.7] 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29936__20210121T1129Z my-pinot-server-1 my-pinot-controller-1 controller 2021-01-21T17:25:04.815+0000: 1351.208: Total time for which application threads were stopped: 0.0781603 seconds, Stopping threads took: 0.0000425 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.841+0000: 1351.235: Application time: 0.0263536 seconds my-pinot-controller-1 controller 2021-01-21T17:25:04.842+0000: 1351.235: [GC pause (G1 Evacuation Pause) (mixed), 0.0820991 secs] server my-pinot-controller-1 controller 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29937__20210121T1129Z my-pinot-server-1 [Parallel Time: 81.1 ms, GC Workers: 4] my-pinot-server-0 my-pinot-controller-1 server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29938__20210121T1129Z controller [GC Worker Start (ms): Min: 1351235.2, Avg: 1351235.2, Max: 1351235.2, Diff: 0.0] my-pinot-controller-1 servercontroller [Ext Root Scanning (ms): Min: 1.0, Avg: 1.1, Max: 1.2, Diff: 0.2, Sum: 4.3] my-pinot-controller-1 [Update RS (ms): Min: 15.5, Avg: 15.5, Max: 15.6, Diff: 0.1, Sum: 93.3] my-pinot-server-0 server [Processed Buffers: Min: 108, Avg: 112.2, Max: 115, Diff: 7, Sum: 673] my-pinot-server-0 server [Scan RS (ms): Min: 166.2, Avg: 166.3, Max: 166.5, Diff: 0.2, Sum: 998.0] my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29939__20210121T1129Z my-pinot-server-1 controller server [Update RS (ms): Min: 35.6, Avg: 35.7, Max: 35.9, Diff: 0.4, Sum: 142.9] my-pinot-controller-1 controller [Processed Buffers: Min: 150, Avg: 155.0, Max: 159, Diff: 9, Sum: 620] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2] my-pinot-controller-1 controller [Scan RS (ms): Min: 5.9, Avg: 14.1, Max: 22.5, Diff: 16.6, Sum: 56.4] my-pinot-controller-1 controller [Code Root Scanning (ms): Min: 0.0, Avg: 4.2, Max: 16.4, Diff: 16.4, Sum: 16.8] server 2021/01/21 17:22:51.990 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29940__20210121T1129Z my-pinot-server-1 server 2021/01/21 17:22:51.991 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29941__20210121T1129Z my-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29942__20210121T1129Z my-pinot-controller-1 my-pinot-server-1 controller my-pinot-server-0 [Object Copy (ms): Min: 21.7, Avg: 25.9, Max: 38.2, Diff: 16.6, Sum: 103.5] server server my-pinot-controller-1 controller 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29943__20210121T1130Z my-pinot-server-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29944__20210121T1130Z my-pinot-server-1 [Object Copy (ms): Min: 24.4, Avg: 24.8, Max: 25.0, Diff: 0.6, Sum: 148.7] my-pinot-server-0 server server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] my-pinot-server-02021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29945__20210121T1130Z my-pinot-server-1 server [Termination Attempts: Min: 1, Avg: 1.2, Max: 2, Diff: 1, Sum: 7] my-pinot-server-0 server [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4] my-pinot-server-0 server [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server [GC Worker Total (ms): Min: 208.0, Avg: 208.0, Max: 208.1, Diff: 0.1, Sum: 1248.2] 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29946__20210121T1130Z my-pinot-server-1 my-pinot-controller-1 server 2021/01/21 17:22:51.992 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29947__20210121T1130Z my-pinot-server-1 my-pinot-server-0 server [GC Worker End (ms): Min: 173747.5, Avg: 173747.5, Max: 173747.6, Diff: 0.1] server my-pinot-server-0 server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29948__20210121T1130Z my-pinot-server-1 controller [Termination Attempts: Min: 1, Avg: 5.8, Max: 10, Diff: 9, Sum: 23] my-pinot-controller-1 [Code Root Fixup: 0.0 ms] my-pinot-server-0 server controller [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29949__20210121T1130Z my-pinot-server-1 server 2021/01/21 17:22:51.994 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29950__20210121T1130Z my-pinot-server-1 server 2021/01/21 17:22:51.995 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29951__20210121T1130Z my-pinot-server-1 server my-pinot-controller-1 controller [GC Worker Total (ms): Min: 81.0, Avg: 81.0, Max: 81.0, Diff: 0.0, Sum: 324.0] [Code Root Purge: 0.0 ms] 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29952__20210121T1130Z my-pinot-server-1 my-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29953__20210121T1130Z my-pinot-controller-1 controller [GC Worker End (ms): Min: 1351316.2, Avg: 1351316.2, Max: 1351316.2, Diff: 0.0] my-pinot-controller-1 [Clear CT: 0.9 ms] controller my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29954__20210121T1130Z my-pinot-server-0 my-pinot-server-1 server [Other: 6.5 ms] [Code Root Fixup: 0.0 ms] my-pinot-controller-1 my-pinot-server-0 server server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29955__20210121T1130Z [Choose CSet: 0.0 ms] my-pinot-server-1 server my-pinot-server-0 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29956__20210121T1131Z server [Ref Proc: 3.8 ms] my-pinot-server-0 server [Ref Enq: 0.0 ms] my-pinot-server-0 server [Redirty Cards: 0.2 ms] my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29957__20210121T1131Z my-pinot-server-0 my-pinot-server-1 server 2021/01/21 17:22:51.996 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29958__20210121T1131Z my-pinot-server-1 controller [Code Root Purge: 0.0 ms] my-pinot-controller-1 controller [Clear CT: 0.1 ms] server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29959__20210121T1131Z server my-pinot-server-1 server [Humongous Register: 0.2 ms] my-pinot-server-0 my-pinot-controller-1 controller 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29960__20210121T1131Z my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29961__20210121T1131Z my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29962__20210121T1132Z my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29963__20210121T1132Z my-pinot-server-1 server 2021/01/21 17:22:51.997 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29964__20210121T1132Z [Other: 0.8 ms] server [Humongous Reclaim: 0.7 ms] my-pinot-server-0 my-pinot-controller-1 server [Free CSet: 1.4 ms] controller my-pinot-server-0 server my-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29965__20210121T1132Z [Choose CSet: 0.1 ms] my-pinot-server-1 server 2021/01/21 17:22:51.998 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29966__20210121T1132Z my-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29967__20210121T1132Z my-pinot-server-1 server 2021/01/21 17:22:51.999 ERROR [mylog_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread] Cannot delete file /var/pinot/server/data/index/mylog_REALTIME/consumers/mylog__0__29968__20210121T1132Z ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
