sandmannn opened a new pull request, #17176: URL: https://github.com/apache/iceberg/pull/17176
Addresses concern (3) from https://github.com/apache/iceberg/issues/17001 As called out in https://github.com/apache/iceberg/issues/17001. The async planner has differnet behavior in `fillQueue(Snapshot readFrom)` and `fillQueue(fromOffset, toOffset)` with regard to the start snapshot. Specifically, `fillQueue(fromOffset, toOffset)` includes the first snapshot even it is a replacement commit. This PR aligns the behavior to consistnetly drop the replacement commit by calling the same filtering logic that is already used in `fillQueue(Snapshot readFrom)`. Added test illustrates the change in the behavior for the async planner: wihtout the change in the `fillQueue`, the assertion will objserve the rows not only from the second batch, but also from the first batch added before the timestamp passed into the stream initialization. As this change can be fairly isolated from other issues in https://github.com/apache/iceberg/issues/17001, creating a dedicated PR just for this one. ### Follow ups This change addresses the behavior in spark 3.5; can downport to spark 4.0 and 3.5 as a next change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
