tibrewalpratik17 opened a new pull request, #13257: URL: https://github.com/apache/pinot/pull/13257
label: `upsert` `bugfix` Saw this in our Uber cluster today where ingestion got stopped due to some exception in snapshot flow. The following was the error we encountered (pinot-version-build: 1.0.0): ``` java.lang.IllegalArgumentException: Path: /data/upinot/stream-pinot-server/dataDir/<tableName>/<tableName>__3__31__20240427T0642Z is not a directory at com.google.common.base.Preconditions.checkArgument(Preconditions.java:210) at org.apache.pinot.segment.spi.store.SegmentDirectoryPaths.findSegmentDirectory(SegmentDirectoryPaths.java:50) at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.getValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:172) at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.hasValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:152) at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.doTakeSnapshot(BasePartitionUpsertMetadataManager.java:667) at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.takeSnapshot(BasePartitionUpsertMetadataManager.java:637) at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:679) at java.base/java.lang.Thread.run(Thread.java:829) ``` Ideally the segment was actually deleted but somehow was still a part of `trackedSegments`. This patch prevents the consumption thread from erroring out if the snapshot flow has any encountered issue. cc @Jackie-Jiang @klsince -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
