matepek opened a new issue, #10003: URL: https://github.com/apache/iceberg/issues/10003
### Apache Iceberg version 1.5.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 # The issue It's great that 1.5 supports views. We have been waiting for this for long. But it bleed from some issues. ## Repro My expectations are based on the [ducumentation](https://iceberg.apache.org/docs/1.5.0/spark-configuration/#catalog-configuration): ``` org.apache.iceberg.spark.SparkSessionCatalog adds support for Iceberg tables to Spark's built-in catalog, and delegates to the built-in catalog for non-Iceberg tables ``` Note that `spark_catalog` is a `org.apache.iceberg.spark.SparkSessionCatalog ### Test Set 1: NOTE: `spark.sql.defaultCatalog = spark_catalog` ``` CREATE OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS SELECT 1 AS id, TIMESTAMP('2000-01-01 01') AS ts, 'a' AS col1 ``` ``` show tables in test_schema ``` Expected: the table Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.` ``` show tables in spark_catalog.test_schema ``` Expected: the table Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.` ``` show tables in iceberg_catalog.test_schema ``` Expected: the table Actual: ❌ `org.apache.iceberg.exceptions.NoSuchNamespaceException: Namespace does not exist: test_schema` ``` select * from test_schema.iceberg_table ``` Expected: the table content Actual: ✅ the table content ``` select * from spark_catalog.test_schema.iceberg_table ``` Expected: the table content Actual: ✅ the table content ``` select * from iceberg_catalog.test_schema.iceberg_table ``` Expected: the table content Actual: ❌ `org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_NOT_FOUND]` ### Test Set 2: I've tried setting the default catalog: `spark.sql.defaultCatalog = iceberg_catalog` ``` CREATE OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS SELECT 1 AS id, TIMESTAMP('2000-01-01 01') AS ts, 'a' AS col1 ``` The result: ``` show tables in test_schema ``` Expected: the table Actual: ✅ the table ``` show tables in spark_catalog.test_schema ``` Expected: the table Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.` ``` show tables in iceberg_catalog.test_schema ``` Expected: the table Actual: ✅ the table ``` select * from test_schema.iceberg_table ``` Expected: the table content Actual: ✅ the table content ``` select * from spark_catalog.test_schema.iceberg_table ``` Expected: the table content Actual: ❌ `org.apache.spark.sql.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view 'spark_catalog'.'test_schema'.'iceberg_table' cannot be found.` ``` select * from iceberg_catalog.test_schema.iceberg_table ``` Expected: the table content Actual: ✅ the table content # The Setup: iceberg : 1.5.0 Java Version : 11.0.22 (Eclipse Adoptium) Scala Version : version 2.12.18 (offical image with little tweaks) | Config | value | | --- | --- | |spark.sql.catalog.iceberg_catalog | org.apache.iceberg.spark.SparkCatalog| |spark.sql.catalog.iceberg_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO| |spark.sql.catalog.iceberg_catalog.jdbc.password | *********(redacted)| |spark.sql.catalog.iceberg_catalog.jdbc.schema-version | V1| |spark.sql.catalog.iceberg_catalog.jdbc.useSSL | false| |spark.sql.catalog.iceberg_catalog.jdbc.user | *********(redacted)| |spark.sql.catalog.iceberg_catalog.jdbc.verifyServerCertificate | false| |spark.sql.catalog.iceberg_catalog.table-default.write.metadata.delete-after-commit.enabled | true| |spark.sql.catalog.iceberg_catalog.table-default.write.metadata.previous-versions-max | 50| |spark.sql.catalog.iceberg_catalog.type | jdbc| |spark.sql.catalog.iceberg_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog| |spark.sql.catalog.iceberg_catalog.warehouse | gs://spark_warehouse_staging| |spark.sql.catalog.spark_catalog | org.apache.iceberg.spark.SparkSessionCatalog| |spark.sql.catalog.spark_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO| |spark.sql.catalog.spark_catalog.jdbc.password | *********(redacted)| |spark.sql.catalog.spark_catalog.jdbc.schema-version | V1| |spark.sql.catalog.spark_catalog.jdbc.useSSL | false| |spark.sql.catalog.spark_catalog.jdbc.user | *********(redacted)| |spark.sql.catalog.spark_catalog.jdbc.verifyServerCertificate | false| |spark.sql.catalog.spark_catalog.type | jdbc| |spark.sql.catalog.spark_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog| |spark.sql.catalog.spark_catalog.warehouse | gs://spark_warehouse_staging| |spark.sql.catalogImplementation | hive| |spark.sql.defaultCatalog | spark_catalog| |spark.sql.extensions | org.apache.sedona.sql.SedonaSqlExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions| -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
