sfc-gh-mrojas opened a new issue, #12469: URL: https://github.com/apache/iceberg/issues/12469
### Apache Iceberg version 1.8.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When using polaris catalog to write to Iceberg from Spark, it stopped working after 1.7.1. It does not work for 1.8.0 and 1.8.1 This setup fails ``` import findspark findspark.init() from pyspark.sql import SparkSession from pyspark.sql.functions import col, when EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.8.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.8.1.jar" spark = SparkSession.builder.appName('iceberg_lab')\ .config('spark.driver.extraClassPath', EXTRA_CLASS_PATH) \ .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \ .config('spark.sql.defaultCatalog', 'polaris') \ .config('spark.sql.catalog.polaris', 'org.apache.iceberg.spark.SparkCatalog') \ .config('spark.sql.catalog.polaris.type', 'rest') \ .config('spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation', 'true') \ .config('spark.sql.catalog.polaris.uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog') \ .config('spark.sql.catalog.polaris.oauth2-server-uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens') \ .config('spark.sql.catalog.polaris.credential',f'{CLIENT_ID}:{CLIENT_SECRET}') \ .config('spark.sql.catalog.polaris.warehouse','AZURE_CATALOG') \ .config('spark.sql.catalog.polaris.scope',f'PRINCIPAL_ROLE:ALL') .getOrCreate() ``` Using old jars it works EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.6.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.7.1.jar" It retuns an error related to authentation: ``` com.azure.identity.CredentialUnavailableException: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/java/identity/workloadidentitycredential/troubleshoot Managed Identity authentication is not available. SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache. IntelliJ Authentication not available. Please log in with Azure Tools for IntelliJ plugin in the IDE. Fore more details refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/intellijcredential/troubleshoot AzureCliCredential authentication unavailable. Azure CLI not installed.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/azclicredential/troubleshoot Az.Account module with version >= 2.2.0 is not installed. It needs to be installed to use Azure PowerShell Credential. AzureDeveloperCliCredential authentication unavailable. Azure Developer CLI not installed.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/azdevclicredential/troubleshootTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot ``` ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org