anuragmantri commented on PR #55: URL: https://github.com/apache/datafusion-iceberg/pull/55#issuecomment-6286179065
Thanks @gabotechs, I appreciate your candid review. I'm writing this reply myself rather than through an agent. I'm new to Rust and leaned heavily on an AI assistant for this PR, and I should have curated its output much more carefully. I apologize for the extra review effort. I agree with the next steps you suggested: - I'll wait for the community's feedback on apache/iceberg#16750 and bring the agreed approach back here. I'll also think about a separate `IcebergSortedDataSource` instead of branches in the existing read path. - On the Java side I could write sorted data files with Spark directly, which made the integration tests simpler. I'll look at how other engines test this, such as Comet's sorted-merge work and DataFusion's own tests over sorted Parquet files, and aim for a few readable integration tests. - I like the idea of splitting out sqllogictests that capture today's plans and results. I'm happy to work through the maintainability concerns and the approach with you all, and I'd welcome input as I rework it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
