geruh commented on code in PR #9292:
URL: https://github.com/apache/iceberg/pull/9292#discussion_r1426221963
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -2367,6 +2368,21 @@ components:
items:
type: string
+ AppendFilesUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ required:
+ - appended-manifests
+ properties:
+ action:
+ type: string
+ enum: [ "append-files" ]
+ appended-manifests:
Review Comment:
Hey Jack, you make a good point. The intentions behind the appended-manifest
was to showcase the need for compression, and it was a middle ground between a
List of DataFiles and a ManifestList. In the case of a large scale Append
operation this list can become very large therefore, we can trade some overhead
for these operations. However, at the end of the day this is a list of string
locations. How the service appends these files is depending on it's ability
to read
[DataFiles](https://iceberg.apache.org/javadoc/master/org/apache/iceberg/DataFiles.html)
into their service from the Request body.
Some other potential approaches we could take are:
* Pagination/batching
* Compression (GZIP, Brotli)
* File Upload of a different type other than avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
