geruh commented on issue #2775: URL: https://github.com/apache/iceberg-python/issues/2775#issuecomment-3560851559
Hey @kevinjqliu, thanks! I built out a POC I'll open as a draft here shortly. > Working on the "core sync planning" first makes sense to me; as i dont think the REST of the apis are implemented yet in the iceberg-rest-fixture. Yeah the logic is there but nothing to enable the async logic. > From an user-facing api standpoint, today we use table.scan().to_arrow(). Perhaps we can expose this remote scan planning functionality like table.scan(remote=true).to_arrow() or table.remote_scan().to_arrow() I've been giving this some thought, and I found a PR on the Java side where they're using a catalog level property to enable REST scan planning: https://github.com/apache/iceberg/pull/13400/files#diff-86450612dbe323d6d06cbc3846aa1913f042eaedadc0ca027c36bfbe08d3a46cR263. I think it would make sense to align with this approach. Especially if a catalog server supports scan planning, it can provide an config override to force scan planning. wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
