dwsmith1983 commented on PR #6753: URL: https://github.com/apache/datafusion-comet/pull/6753#issuecomment-6125250316
> Could you merge `main` and fix the conflicts? Merged in 545735cd5 and resolved the way you describe: `extractScanObjectStoreOptions` keeps both ABFS checks and returns the union across schemes when they pass, and both serdes call it in place of the union. In the CSV serde it runs on the scan's files before the split rather than after the 3.4 guard, so an ABFS decline reports its own reason. The split only regroups the same files, so the options come out the same. > Could it add a `gs://` file and check that the scan gets both schemes' options? `non-abfs scans forward every scheme's options` now reads `s3a`, `s3`, `gs` and two alias buckets, checks the `gs` key and each bucket's translated alias settings, and checks that the first file's options alone would miss them. > Could it point at the ABFS fallback described in the Azure section? The multi-store section now says ABFS across containers or accounts, or mixed with other schemes, falls back, with a link to the Azure section. `CometMultiStoreScanSuite` also has ABFS cases now: one container stays native and forwards only that account's resolution, while two containers, two accounts, or ABFS next to HDFS fall back. `NativeConfigSuite` adds `abfss` with `wasb` in one scan. One more change in this push: `NativeConfig.bucketForUri` now returns the host the native S3 store looks up, without a port or user info. Before, alias settings for `blob://bucket:9000/...` were scoped as `bucket:9000` and dropped. The shared-alias fallback now compares buckets, so `blob://bucket:9000` next to `s3a://bucket` falls back like `blob://bucket` next to `s3a://bucket`. The Iceberg scan and write paths share that helper, and the description covers what changes there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
