maekchi commented on issue #10765:
URL: https://github.com/apache/iceberg/issues/10765#issuecomment-2254833226
@pvary
> Which version of Flink do you using btw?
I use flink version 1.15.4
> What Catalog are you using? Is there any cache, or something which might
return wrong data for the table?
We use hive catalog, and set up 'engine.hive.lock-enabled=true'
and no cache and no wrong data. (we are filtering every wrong data)
-----------------------
By the way, there seems to be one more strange thing in the log.
After processing the checkpointId for 19516, there should be a log somewhere
for processing the 19517 checkpoint ID, but there is no log at all.
Because our system performs checkpoints every minute, the checkpoint for
19517 should be performed around 04:58:32 after the 19516 checkpoint performed
at 04:57:32.
However, there is no related log. This is because the taskmanager was shut
down around that time.
However, an attempt was made for 19516 at 04:59:22, and if you look at the
log at 05:00:00, the 19516 snapshot was successfully committed. But there is no
log for 19517 anywhere.
There is no record for 19517 even when looking at metadata.json! Is it
normal situation during recovery?
```
{
"sequence-number" : 201719,
"snapshot-id" : 8203882888081487848,
"parent-snapshot-id" : 7556868946872881546,
"timestamp-ms" : 1721764676985,
"summary" : {
"operation" : "append",
"flink.operator-id" : "9135501d46e54bf84710f477c1eb5f38",
"flink.job-id" : "ba65ea243c487f4f0fd52c158e4ed985",
"flink.max-committed-checkpoint-id" : "19516",
"added-data-files" : "1",
"added-records" : "17554",
"added-files-size" : "664840",
"changed-partition-count" : "1",
"total-records" : "3966880804",
"total-files-size" : "241007398466",
"total-data-files" : "774",
"total-delete-files" : "2",
"total-position-deletes" : "18608",
"total-equality-deletes" : "0"
},
"manifest-list" :
"hdfs://~~~~~/metadata/snap-8203882888081487848-1-354fd0bb-38d9-4706-8483-8a4276888dc3.avro",
"schema-id" : 2
}, {
"sequence-number" : 201720,
"snapshot-id" : 3289453546560274810,
"parent-snapshot-id" : 8203882888081487848,
"timestamp-ms" : 1721764798149,
"summary" : {
"operation" : "append",
"flink.operator-id" : "9135501d46e54bf84710f477c1eb5f38",
"flink.job-id" : "ba65ea243c487f4f0fd52c158e4ed985",
"flink.max-committed-checkpoint-id" : "19516",
"added-data-files" : "1",
"added-records" : "17554",
"added-files-size" : "664840",
"changed-partition-count" : "1",
"total-records" : "3966898358",
"total-files-size" : "241008063306",
"total-data-files" : "775",
"total-delete-files" : "2",
"total-position-deletes" : "18608",
"total-equality-deletes" : "0"
},
"manifest-list" :
"hdfs://~~~~~/metadata/snap-3289453546560274810-2-e0983626-a2a5-49f2-988b-dc432f100451.avro",
"schema-id" : 2
}, {
"sequence-number" : 201721,
"snapshot-id" : 3232659717465048464,
"parent-snapshot-id" : 3289453546560274810,
"timestamp-ms" : 1721764843143,
"summary" : {
"operation" : "append",
"flink.operator-id" : "9135501d46e54bf84710f477c1eb5f38",
"flink.job-id" : "ba65ea243c487f4f0fd52c158e4ed985",
"flink.max-committed-checkpoint-id" : "19518",
"added-data-files" : "1",
"added-records" : "56759",
"added-files-size" : "2237712",
"changed-partition-count" : "1",
"total-records" : "3966955117",
"total-files-size" : "241010301018",
"total-data-files" : "776",
"total-delete-files" : "2",
"total-position-deletes" : "18608",
"total-equality-deletes" : "0"
},
"manifest-list" :
"hdfs://~~~~~/metadata/snap-3232659717465048464-1-8c5a3ab7-9303-45e5-910c-41d47be08142.avro",
"schema-id" : 2
},
```
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
