SudhaGarimella edited a comment on issue #5600: URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-648783983
@haibow : [seg-create-issue.txt](https://github.com/apache/incubator-pinot/files/4825293/seg-create-issue.txt) Sorry did not realize that it was truncated. Please check this one. Please check attached file. Below is extracted from attached file java.io.IOException: Unable to delete directory /var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__0__40__20200623T1858Z-1592999308227/tmp-1592999308462--859419267052440122. at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2756) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:230) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:199) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:141) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:720) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:671) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:588) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252] 2020/06/24 11:49:38.926 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Consumed 19866 events from (rate:331.08344/s), currentOffset=196880089, numRowsConsumedSoFar=99773, numRowsIndexedSoFar=99773 2020/06/24 11:49:39.019 INFO [LLRealtimeSegmentDataManager_jeepusermap__2__54__20200623T1843Z] [jeepusermap__2__54__20200623T1843Z] Consumed 20005 events from (rate:333.39444/s), currentOffset=197074187, numRowsConsumedSoFar=100182, numRowsIndexedSoFar=100182 2020/06/24 11:53:06.296 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Stopping consumption due to row limit nRows=169204 numRowsIndexed=169204, numRowsConsumed=169204 2020/06/24 11:53:06.305 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown 2020/06/24 11:53:06.306 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 2020/06/24 11:53:09.311 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown 2020/06/24 11:53:09.311 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 2020/06/24 11:53:12.315 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown 2020/06/24 11:53:12.315 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 2020/06/24 11:53:15.320 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown 2020/06/24 11:53:15.320 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 Similar logs show that retry is happening. Since all consuming threads on all servers threw the same issue, no segment got created in controller also and all 3 consuming threads are down. And from azure portal I am able to delete the directory mentioned in above exception without any issues ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
