pvary commented on issue #8802: URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1765938139
> > Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks). > > I'm sorry, I didn't quite understand this point. Could you please explain it in more detail? We fetch all the tasks for a given plan here (and covert it to an IcebergSplit using `fromCombinedScanTask`): https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java#L73-L82 And we use a [SerializableComparator](https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/split/SplitComparators.java#L33) to order the splits before assigning them: https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/assigner/DefaultSplitAssigner.java#L44-L46 I hope this helps! Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
