dyno opened a new issue, #8927: URL: https://github.com/apache/iceberg/issues/8927
https://github.com/apache/iceberg/blob/333227fbd13821365cec1bdbfcb9314a239bea0f/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L244-L255 recently we encounter a few cases that write to iceberg table aborted and then table is not usable complaining with error message like ``` Caused by: org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: s3://some/path/to/table/metadata/13637-45c53fb2-5124-4891-ace3-c63ed91e1d26.metadata.json ``` the symptem seems to be that the hive commit is persistent in metastore, but the spark write abort then deleted the location file. result in the table is not useable. ``` E1024 22:07:51.303 pool-38-thread-273 o.a.s.s.e.d.v.OverwriteByExpressionExec:77] Data source write support IcebergBatchWrite(table=<redacted>, format=PARQUET) is aborting. W1024 22:07:51.303 pool-38-thread-273 o.a.i.s.s.SparkWrite:226] Skipping cleanup of written files E1024 22:07:51.303 pool-38-thread-273 o.a.s.s.e.d.v.OverwriteByExpressionExec:77] Data source write support IcebergBatchWrite(table=<redacted>, format=PARQUET) aborted. ``` and through s3 access log we can confirm metadata location file is deleted along the way. and we have to fix the table by restore the metadata location to previous one in hive. ```sql -- get previous_metadata_location. show tblproperties xxx; alter table xxx set tblproperties('metadata_location' = '{previous_metadata_location}'); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
