brysd opened a new issue, #10174: URL: https://github.com/apache/iceberg/issues/10174
### Apache Iceberg version 1.4.1 ### Query engine Spark ### Please describe the bug 🐞 Related to #6290 we build upon the spark DESCRIBE statement to retrieve the partition columns and enable dynamic modification of partition definitions (spec). We were on spark 3.3 and iceberg 1.3.1. However we currently are testing the migration to spark 3.4 and iceberg 1.4.1. It seems however that the describe statement of a table doesn't provide any information anymore on the partition spec if there are no partition columns defined. Before we had this as output of a DESCRIBE statement if there are no partition fields: |col_name |data_type|comment| |------------------|---------|-------| |id |string | | |value1 |string | | |value2 |string | | |value3 |string | | |cre |timestamp| | |lw |timestamp| | |x_src_cdc_ts |timestamp| | |x_beg_ts |timestamp| | |x_end_ts |timestamp| | |x_current_flag |int | | |x_cre_ts |timestamp| | |x_upd_ts |timestamp| | |x_ingest_method |string | | |x_landing_filename|string | | |x_cdc_ts |timestamp| | | | | | |# Partitioning | | | |Not partitioned | | | With spark 3.4 and iceberg 1.4.1 we get this: |col_name |data_type|comment| |------------------|---------|-------| |id |string | | |value1 |string | | |value2 |string | | |value3 |string | | |cre |timestamp| | |lw |timestamp| | |x_src_cdc_ts |timestamp| | |x_beg_ts |timestamp| | |x_end_ts |timestamp| | |x_current_flag |int | | |x_cre_ts |timestamp| | |x_upd_ts |timestamp| | |x_ingest_method |string | | |x_landing_filename|string | | |x_cdc_ts |timestamp| | If there are partition fields this is what we had before (spark 3.3, iceberg 1.3.1): |col_name |data_type|comment| |------------------|---------|-------| |id |string | | |value1 |string | | |value2 |string | | |value3 |string | | |cre |timestamp| | |lw |timestamp| | |x_src_cdc_ts |timestamp| | |x_beg_ts |timestamp| | |x_end_ts |timestamp| | |x_current_flag |int | | |x_cre_ts |timestamp| | |x_upd_ts |timestamp| | |x_ingest_method |string | | |x_landing_filename|string | | |x_cdc_ts |timestamp| | | | | | |# Partitioning | | | |Part 0 |id | | With spark 3.4 / iceberg 1.4.1 we get this: |col_name |data_type|comment| |-----------------------|---------|-------| |id |string |null | |value1 |string |null | |value2 |string |null | |value3 |string |null | |cre |timestamp|null | |lw |timestamp|null | |x_src_cdc_ts |timestamp|null | |x_beg_ts |timestamp|null | |x_end_ts |timestamp|null | |x_current_flag |int |null | |x_cre_ts |timestamp|null | |x_upd_ts |timestamp|null | |x_ingest_method |string |null | |x_landing_filename |string |null | |x_cdc_ts |timestamp|null | |# Partition Information| | | |# col_name |data_type|comment| |id |string |null | Does this mean that when there are no partition columns defined the # Partitioning 'header' or Partition Information is not available anymore? Has anything changed with respect to the iceberg implementation of describe or is this a potential spark 3.4 issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
