laxmanmamididd opened a new issue, #7389: URL: https://github.com/apache/datafusion-comet/issues/7389
### Is your feature request related to a problem or challenge? #7363 adds `IcebergVendedS3CredentialProvider` in the spark-4.x source set only, because it links against AWS SDK v2. Iceberg 1.8.1, which the Spark 3.5 build uses, already has the same refresh logic in `AwsClientProperties`, and Comet loads Iceberg by reflection. On Spark 3.x, a long native Iceberg scan against a REST catalog with short-lived credentials still fails with `ExpiredToken`. ### Describe the solution you'd like Ship the provider in Spark 3.x builds too. Iceberg's provider is SDK v2 there as well, so the class would need its own body in the spark-3.x source set, or a loader-neutral way to read the SDK v2 credentials. ### Additional context The docs added in #7363 state that Spark 3.x builds don't include the provider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
