csringhofer commented on PR #17112:
URL: https://github.com/apache/iceberg/pull/17112#issuecomment-6019213714
I did a more thorough interop test with Spark/Tirino/Hive/Impala using
HiveCatalog
Versions: Spark 4.1.3, Trino: 483, Hive 4.2++ (newer than latest release),
Impala asf-master + geometry patch
For geometry:
Hive doesn't know geometry at all
Spark knows geometry, but can't create tables with geometry columns (this PR
would fix it)
Trino created tables: both Spark and Impala can read
Impala created table: both Spark and Trino can read
Also checked variant, where unknown type is used:
Hive created tables: every engine can read
Spark created tables: Trino can't read, Impala/Hive can
Trino created tables: every engine can read
Impala can't create tables (not yet implemented)
One caveat is the very new Hive I used in the metastore - using "unknown" is
unlikely to work with older Hive Metastore, including the last released version
4.2.1. In the wild users often have very old HMS versions AFAIK. Besides the
Trino issue, this is my main motivation to use binary, which is in Hive since a
very long time.
Another tricky part with unknown are nested types: I couldn't create a table
with nested variant in Spark:
create table iceberg.default.spark_variant_nested (s
struct<a:int,v:variant>) using iceberg tblproperties ('format-version' = '3');
Error running query: java.lang.RuntimeException: Internal error parsing
position 0 of 'unknown'
So it seems ok with 'unknown' at top level, but not when nested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
