[I] [core] Table metadata points to metadata.json that doesn't exist as it failed to upload [iceberg]

[via GitHub]

abmo-x opened a new issue, #9997:
URL: https://github.com/apache/iceberg/issues/9997

   ### Apache Iceberg version
   
   1.3.1
   
   ### Query engine
   
   Flink
   
   ### Please describe the bug 🐞
   
   Iceberg table points to a metadata.json that failed to upload to s3. Due to 
this the table was not loadable as latest metadata json was not found 
   `The main method caused an error: Failed to open input stream for file: 
s3a://bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json`
   
   
   ### Expected:
   when s3 upload to metadata json failed, iceberg's hive commit should have 
failed as write metadata failed.
   
   ### Actual:
   
   Iceberg logs show that commit was successful
   
   `2024-03-06T23:06:36.160+00:00 INFO 
org.apache.iceberg.hive.HiveTableOperations 
        Committed to table iceberg.default.some_table with the new metadata 
location 
s3a://bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
 `
   
   #### However the upload to s3 failed!
   `
   06/Mar/2024:23:06:01 +0000 REST.PUT.OBJECT   
prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
 "PUT 
/bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
 HTTP/1.1"  **400   IncompleteBody**`
   
   #### Flink pipeline fails around same time as upload 
   
   `2024-03-06T23:06:36.110+00:00  WARN 
org.apache.flink.runtime.taskmanager.Task 
       message:  Source: table[11] -> PythonCalc[12] -> (Calc[13] -> 
IcebergStreamWriter, Calc[15] -> IcebergStreamWriter) (2/4)#0 switched from 
INITIALIZING to FAILED with failure cause:`
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org