gh-yzou commented on code in PR #1503: URL: https://github.com/apache/polaris/pull/1503#discussion_r2071978195
########## plugins/spark/README.md: ########## @@ -72,24 +77,26 @@ bin/spark-shell \ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \ --conf spark.sql.catalog.polaris.warehouse=<catalog-name> \ ---conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=true \ +--conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \ --conf spark.sql.catalog.polaris=org.apache.polaris.spark.SparkCatalog \ --conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \ --conf spark.sql.catalog.polaris.credential="root:secret" \ --conf spark.sql.catalog.polaris.scope='PRINCIPAL_ROLE:ALL' \ --conf spark.sql.catalog.polaris.token-refresh-enabled=true \ ---conf spark.sql.catalog.polaris.type=rest \ --conf spark.sql.sources.useV1SourceList='' ``` # Limitations The Polaris Spark client supports catalog management for both Iceberg and Delta tables, it routes all Iceberg table requests to the Iceberg REST endpoints, and routes all Delta table requests to the Generic Table REST endpoints. -Following describes the current limitations of the Polaris Spark client: +The Spark Client requires at least delta 3.2.1 to work with Delta tables, which requires at least Apache Spark 3.5.3. +Following describes the current functionality limitations of the Polaris Spark client: 1) Create table as select (CTAS) is not supported for Delta tables. As a result, the `saveAsTable` method of `Dataframe` is also not supported, since it relies on the CTAS support. 2) Create a Delta table without explicit location is not supported. 3) Rename a Delta table is not supported. 4) ALTER TABLE ... SET LOCATION/SET FILEFORMAT/ADD PARTITION is not supported for DELTA table. -5) For other non-iceberg tables like csv, there is no specific guarantee provided today. +5) For other non-Iceberg tables like csv, there is no specific guarantee provided today. +6) TABLE_WRITE_DATA privilege is not supported for Delta Table. +7) Credential Vending is not supported for Delta Table. Review Comment: Actually, those are our Polaris service limitations, not really client limitations, it might be better to introduce another generic table support page and put that limitation there. I removed it from the polaris spark client for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
