raviston opened a new pull request, #18989:
URL: https://github.com/apache/iceberg/pull/18989

   Closes #17288
   
   ## Problem
   
   With `spark_catalog` set to `SparkSessionCatalog` over a Hive catalog, 
reading an Iceberg view that has column aliases fails:
   
   ```sql
   CREATE VIEW v (c1, c2) AS VALUES (1, 2);
   SELECT * FROM v;
   -- [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] The SQL query of view 
`spark_catalog`.`default`.`v` has an incompatible
   -- schema change and column c1 cannot be resolved. Expected 1 columns named 
c1 but got [].
   ```
   
   `WITH SCHEMA EVOLUTION` from the issue isn't required. The column aliases 
are what trigger it.
   
   The Hive catalog stores Iceberg views in the Hive Metastore as 
`VIRTUAL_VIEW`s.
   1. `SparkSessionCatalog.loadTable` fails to load the view as an Iceberg 
table and falls back to the session catalog.
   2. The session catalog returns the metastore entry as a Hive view 
(`V1Table`).
   3. Spark's `ResolveRelations` then resolves it as a v1 view before the 
`ResolveViews` rule of the Iceberg extensions gets to it.
   4. `SessionCatalog.fromCatalogTable` looks up each view column by name in 
the output of the view's query. Without the `view.query.out.col.*` properties 
that Spark writes for its own views, it looks for the view's column names 
(`c1`). The query only outputs `col1` and `col2`, so the lookup fails.
   
   Iceberg views are also missed by the rest of the Iceberg view handling in 
this setup: `DESCRIBE`, `SHOW CREATE TABLE`, the schema binding mode, temp 
object checks, and others.
   
   ## Fix
   
   In `SparkSessionCatalog.loadTable`, the session catalog's result is not 
returned when it is a view that the Iceberg view catalog knows (`viewExists`). 
The original `NoSuchTableException` is rethrown instead, so `ResolveViews` 
resolves the view as an Iceberg view, the same as for the other catalogs. The 
extra `viewExists` call is only made when the session catalog returns a view. 
Hive views that aren't Iceberg views are still resolved by Spark, because 
`HiveCatalog.viewExists` is false for them.
   
   ## Testing
   
   - New `TestSessionCatalogHiveViews` covers `SparkSessionCatalog` with 
`type=hive`, which `TestViews` doesn't cover. It has three tests:
     - column aliases over `VALUES`
     - the exact SQL from the issue, with `WITH SCHEMA EVOLUTION`
     - aliases with a column comment over a table
     
     All three fail without the fix with the error above.
   - Locally, I also ran `TestViews` with only the `SparkSessionCatalog` + Hive 
configuration as its parameter. That run isn't part of this PR.
     - Without the fix, 26 of its 86 tests fail.
     - With the fix, 25 of those pass and no new failures appear.
     - The remaining one, `renameViewToDifferentTargetCatalog`, fails with or 
without the fix. `HiveCatalog.renameView` rejects the three-part target 
identifier, which is a separate issue.
   - On Spark 4.2, `TestSparkSessionCatalog`, `TestViews`, 
`TestViewCatalogCache`, `TestReferencedByViewChain`, `TestTempViewRefresh` and 
`TestSessionCatalogHiveViews` pass.
   
   Spark 3.5, 4.0 and 4.1 have the same code. I'll backport in a follow-up once 
this is reviewed.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to