andygrove commented on PR #5872: URL: https://github.com/apache/datafusion-comet/pull/5872#issuecomment-5876609462
This is a light fully automated review since there are so many PRs open. I think the addressing change needs an entry in the upgrade guide as well as the note in `datasources.md`. `docs/source/user-guide/latest/migration-guide.md` now says each release's section also lists changes that need no legacy key but can still change what an existing deployment does, "such as a fix that makes Comet apply a setting as documented", and the 1.1.0 section has a "Settings That Now Take Effect as Documented" subsection for exactly that kind of fix. This PR is one of them. With `fs.s3a.endpoint=http://minio.internal:9000` and `fs.s3a.path.style.access` unset or `false`, native reads move from `http://minio.internal:9000/bucket/key` to `http://bucket.minio.internal:9000/key`. The upgrade guide is the page that tells people to read every section between their old and new release, and right now the only mention is the paragraph at `docs/source/user-guide/latest/datasources.md:260`. Could you add an entry there for the release this lands in, naming `fs.s3a.path.style.acce ss=true` as the setting that keeps the old addressing? On the new row at `docs/source/user-guide/latest/datasources.md:234`, `org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider` first shipped in Hadoop 3.4.2. Spark 3.4 and 3.5 bundle Hadoop 3.3.4 and Spark 4.0 bundles 3.4.1, and the matching `hadoop-aws` jars do not contain the class. On those versions S3A cannot create the provider, so the query fails on the JVM side before the native scan runs. Since the next row says the SDK spellings ignore the profile keys, users on those Spark versions have no way to use `fs.s3a.auth.profile.name` or `fs.s3a.auth.profile.file` at all. Would it make sense to say in the row that this provider needs Hadoop 3.4.2 or later, which Spark 4.1 bundles? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
