hililiwei commented on PR #6253: URL: https://github.com/apache/iceberg/pull/6253#issuecomment-1330000575
> @hililiwei time-based partition commit seems quite complicated. trying to understand its value. With watermark info to mark the data completeness, downstream can decide which partition (hourly or daily) has the complete data and it is ok to trigger the processing of the completed hour or day. Hi @stevenzwu 1. If time-based commit is supported, as long as the partition is visible, its data is ready. downstream applications can directly use incremental reads instead of round robin. 2. In some scenarios, when the partition data is not completely written, we hope that downstream applications cannot see it. Watermarks alone are not friendly enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
