dwsmith1983 commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6122235744
> The `CometNativeScan` one needs a decision on your side, since `buildNativeScanCommon` and `populateScanConfFlags` still take `firstFileUri` while `convert` now uses `extractScanObjectStoreOptions` over every file. Merged main in e2d6deb3b. `buildNativeScanCommon` and `populateScanConfFlags` now take the resolved object store options instead of `firstFileUri`, so each caller decides. `CometNativeScan.convert` does what main does: `extractScanObjectStoreOptions` over every selected file, and a fallback with the reason on both the scan and the wrapped scan. The Delta scan runs the same check over its data files. Delta's existing gate already declines data files and deletion vectors whose authority carries userinfo, which covers every ABFS path, so the only Azure authority that can reach the Delta option union is the table root. The union's comment now says so, since #6059 made the ABFS keys global. The Delta suites pass on Spark 3.5 with strict warnings, the MinIO ones included, and so do the object store and ABFS suites on 3.5 and 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
