Chaho12 opened a new issue, #10101:
URL: https://github.com/apache/iceberg/issues/10101
### Apache Iceberg version
1.4.3
### Query engine
Hive
### Please describe the bug 🐞
As we all know through the commit process, iceberg registers the most
up-to-date metadata file path to the table properties in the hive metastore.
However, in some cases, I notice that even if iceberg registered the path of
the metadata file in the hive metastore and received a normal response, but it
was `not registered in the hive metastore`, causing the issue.
Since from the perspective of flink iceberg, it is correct to proceed to the
next checkpoint because hms responded normally so this unregistered metadata
gets lost.
Does anyone have any ideas on fixing this?
```
2024-03-19 03:31:02,535 INFO
org.apache.iceberg.flink.sink.IcebergFilesCommitter [] - Start to
flush snapshot state to state backend, table: hive.custom_schema.public_hms,
checkpointId: 36569
2024-03-19 03:31:02,712 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new compressor [.zstd]
2024-03-19 03:31:03,127 INFO
org.apache.iceberg.flink.sink.IcebergFilesCommitter [] - Committing
append for checkpoint 36569 to table hive.custom_schema.public_hms branch main
with summary: CommitSummary{dataFilesCount=10, dataFilesRecordCount=374,
dataFilesByteCount=72923, deleteFilesCount=0, deleteFilesRecordCount=0,
deleteFilesByteCount=0}
2024-03-19 03:31:03,506 INFO org.apache.hadoop.io.compress.CodecPool
[] - Got brand-new compressor [.zstd]
2024-03-19 03:31:03,966 INFO org.apache.iceberg.hive.HiveTableOperations
[] - Committed to table hive.custom_schema.public_hms with the
new metadata location
hdfs://.../user/user_hive/warehouse/custom_schema.db/public_hms/metadata/36767-0e5f399a-867c-49de-90e1-0806482ea2fa.metadata.json
2024-03-19 03:31:04,042 INFO
org.apache.iceberg.BaseMetastoreTableOperations [] - Successfully
committed to table hive.custom_schema.public_hms in 611 ms
2024-03-19 03:31:04,042 INFO org.apache.iceberg.SnapshotProducer
[] - Committed snapshot 517916272951067890 (MergeAppend)
2024-03-19 03:31:04,196 WARN org.apache.iceberg.SnapshotProducer
[] - Failed to load committed snapshot, skipping manifest clean-up
2024-03-19 03:31:04,329 WARN org.apache.iceberg.MergingSnapshotProducer
[] - Failed to load committed snapshot: omitting sequence number
from notifications
2024-03-19 03:31:04,329 INFO
org.apache.iceberg.metrics.LoggingMetricsReporter [] - Received
metrics report: CommitReport{tableName=hive.custom_schema.public_hms,
snapshotId=s, sequenceNumber=-1, operation=append,
commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS,
totalDuration=PT1.069239305S, count=1}, attempts=CounterResult{unit=COUNT,
value=1}, addedDataFiles=CounterResult{unit=COUNT, value=10},
removedDataFiles=null, totalDataFiles=null, addedDeleteFiles=null,
addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null,
removedDeleteFiles=null, removedEqualityDeleteFiles=null,
removedPositionalDeleteFiles=null, totalDeleteFiles=null,
addedRecords=CounterResult{unit=COUNT, value=374}, removedRecords=null,
totalRecords=null, addedFilesSizeInBytes=CounterResult{unit=BYTES,
value=72923}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=null,
addedPositionalDeletes=null, removedPositionalDeletes=null,
totalPositionalDeletes=null, add
edEqualityDeletes=null, removedEqualityDeletes=null,
totalEqualityDeletes=null}, metadata={engine-version=1.15.2, engine-name=flink,
iceberg-version=Apache Iceberg 1.2.0 (commit
e340ad5be04e902398c576f431810c3dfa4fe717)}}
```
<hadoop.version>3.1.0</hadoop.version>
<hive.version>3.1.2</hive.version>
<metastore.version>3.1.3</metastore.version>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
