maekchi commented on issue #10765:
URL: https://github.com/apache/iceberg/issues/10765#issuecomment-2249927915
@pvary
I also think this is different than #10526.
I am only appending data.
and perform with `exactly once` mode in Flink.
Because there were other Flink apps, there were too many logs, so I am
attaching additional logs by filtering them by table name among all logs.
```
Jul 24, 2024 @ 04:57:32.562 Committing append for checkpoint 19516 to table
hive.table branch main with summary: CommitSummary{dataFilesCount=1,
dataFilesRecordCount=17554, dataFilesByteCount=664840, deleteFilesCount=0,
deleteFilesRecordCount=0, deleteFilesByteCount=0} 90 task_manager
Jul 24, 2024 @ 04:57:32.562 Start to flush snapshot state to state backend,
table: hive.table, checkpointId: 19516 90 task_manager
Jul 24, 2024 @ 04:59:00.153 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (a035c92a2f05258b27a3986798fcde66)
switched from CANCELING to CANCELED. 25,369 job_manager
Jul 24, 2024 @ 04:59:00.153 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (eb7fcaa1da87cd760c8e489b36528ac7)
switched from CREATED to SCHEDULED. 25,371 job_manager
Jul 24, 2024 @ 04:59:00.153 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (a035c92a2f05258b27a3986798fcde66)
switched from RUNNING to CANCELING. 25,369 job_manager
Jul 24, 2024 @ 04:59:19.156 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (eb7fcaa1da87cd760c8e489b36528ac7)
switched from SCHEDULED to DEPLOYING. 25,360 job_manager
Jul 24, 2024 @ 04:59:19.156 Deploying freezer-IcebergFilesCommitter ->
Sink: freezer-IcebergSink hive.table (1/1) (attempt #1) with attempt id
eb7fcaa1da87cd760c8e489b36528ac7 and vertex id
9135501d46e54bf84710f477c1eb5f38_0 to 172.24.71.24:6122-16ff02 @ 172.24.71.24
(dataPort=45067) with allocation id fa75b71da4f9b96484bc5435b5f55b00 25,360
job_manager
Jul 24, 2024 @ 04:59:20.156 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (eb7fcaa1da87cd760c8e489b36528ac7)
switched from DEPLOYING to INITIALIZING. 25,366 job_manager
Jul 24, 2024 @ 04:59:20.780 Loading JAR files for task
freezer-IcebergFilesCommitter -> Sink: freezer-IcebergSink hive.table (1/1)#1
(eb7fcaa1da87cd760c8e489b36528ac7) [DEPLOYING]. 3,208 task_manager
Jul 24, 2024 @ 04:59:20.780 Received task freezer-IcebergFilesCommitter ->
Sink: freezer-IcebergSink hive.table (1/1)#1
(eb7fcaa1da87cd760c8e489b36528ac7), deploy into slot with allocation id
fa75b71da4f9b96484bc5435b5f55b00. 3,196 task_manager
Jul 24, 2024 @ 04:59:20.780 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1)#1 (eb7fcaa1da87cd760c8e489b36528ac7)
switched from CREATED to DEPLOYING. 3,208 task_manager
Jul 24, 2024 @ 04:59:20.781 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1)#1 (eb7fcaa1da87cd760c8e489b36528ac7)
switched from DEPLOYING to INITIALIZING. 3,208 task_manager
Jul 24, 2024 @ 04:59:22.783 Table loaded by catalog: hive.table 3,208
task_manager
Jul 24, 2024 @ 04:59:22.783 Committing append for checkpoint 19516 to table
hive.table branch main with summary: CommitSummary{dataFilesCount=1,
dataFilesRecordCount=17554, dataFilesByteCount=664840, deleteFilesCount=0,
deleteFilesRecordCount=0, deleteFilesByteCount=0} 3,208 task_manager
Jul 24, 2024 @ 04:59:45.158 Failed to trigger checkpoint for job
ba65ea243c487f4f0fd52c158e4ed985 since Checkpoint triggering task
freezer-IcebergFilesCommitter -> Sink: freezer-IcebergSink hive.table (1/1) of
job ba65ea243c487f4f0fd52c158e4ed985 is not being executed at the moment.
Aborting checkpoint. Failure reason: Not all required tasks are currently
running.. 173 job_manager
Jul 24, 2024 @ 04:59:59.159 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1) (eb7fcaa1da87cd760c8e489b36528ac7)
switched from INITIALIZING to RUNNING. 25,397 job_manager
Jul 24, 2024 @ 05:00:00.788 freezer-IcebergFilesCommitter -> Sink:
freezer-IcebergSink hive.table (1/1)#1 (eb7fcaa1da87cd760c8e489b36528ac7)
switched from INITIALIZING to RUNNING. 3,208 task_manager
Jul 24, 2024 @ 05:00:00.788 Successfully committed to table hive.table in
130 ms 3,208 task_manager
Jul 24, 2024 @ 05:00:00.788 Committed to table hive.table with the new
metadata location
hdfs://~~~~~/metadata/202296-54e38fd0-c3c5-4431-b072-962402eedb7b.metadata.json
3,208 task_manager
Jul 24, 2024 @ 05:00:00.788 Committed append to table: hive.table, branch:
main, checkpointId 19516 in 36033 ms 3,208 task_manager
Jul 24, 2024 @ 05:00:00.788 Received metrics report:
CommitReport{tableName=hive.table, snapshotId=3289453546560274810,
sequenceNumber=201720, operation=append,
commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS,
totalDuration=PT36.012583152S, count=1}, attempts=CounterResult{unit=COUNT,
value=2}, addedDataFiles=CounterResult{unit=COUNT, value=1},
removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=775},
addedDeleteFiles=null, addedEqualityDeleteFiles=null,
addedPositionalDeleteFiles=null, removedDeleteFiles=null,
removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null,
totalDeleteFiles=CounterResult{unit=COUNT, value=2},
addedRecords=CounterResult{unit=COUNT, value=17554}, removedRecords=null,
totalRecords=CounterResult{unit=COUNT, value=3966898358},
addedFilesSizeInBytes=CounterResult{unit=BYTES, value=664840},
removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{unit=BYTES,
value=241008063306}, addedPositio
nalDeletes=null, removedPositionalDeletes=null,
totalPositionalDeletes=CounterResult{unit=COUNT, value=18608},
addedEqualityDeletes=null, removedEqualityDeletes=null,
totalEqualityDeletes=CounterResult{unit=COUNT, value=0}},
metadata={engine-version=1.15.2, engine-name=flink, iceberg-version=Apache
Iceberg 1.4.3 (commit 9a5d24fee239352021a9a73f6a4cad8ecf464f01)}} 3,208
task_manager
Jul 24, 2024 @ 05:00:09.567 Committed to table hive.table with the new
metadata location
hdfs://~~~~~/metadata/202295-d906b795-214d-46a6-b2dd-5353f134a9a6.metadata.json
90 task_manager
Jul 24, 2024 @ 05:00:45.789 Committed append to table: hive.table, branch:
main, checkpointId 19518 in 1112 ms 3,208 task_manager
Jul 24, 2024 @ 05:00:45.789 Table loaded by catalog: hive.table 3,208
task_manager
Jul 24, 2024 @ 05:00:45.789 Start to flush snapshot state to state backend,
table: hive.table, checkpointId: 19518 3,208 task_manager
```
Checking the log, it seems that there is a "commiting" log at 04:57, but
there is no "commited" log for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
