nhobin219 opened a new issue, #4093:
URL: https://github.com/apache/iceberg-python/issues/4093
### Apache Iceberg version
0.12.0 (also on main)
### Please describe the bug 🐞
`PyArrowFileIO` writes every S3 object as a multipart upload, however small
it is. pyarrow's `S3FileSystem.open_output_stream` starts a multipart upload as
soon as the stream opens (apache/arrow#51029). Each `metadata.json`, manifest
list, manifest, `version-hint.text` and small data file therefore costs three
requests: `CreateMultipartUpload`, `UploadPart` and `CompleteMultipartUpload`.
All three are billed as writes, even for a 1-byte object.
pyarrow's `S3FileSystem` has had an option for this, `allow_delayed_open`,
since pyarrow 17. With it set, a stream that closes before reaching a part's
size is sent as a single `PutObject`, and a larger one is still a multipart
upload. `_initialize_s3_fs` builds the filesystem from a fixed set of
properties, though, so there's no way to set the option through FileIO
properties.
The cost is real for a table that commits often. Each commit writes several
small metadata objects, so it makes about three times the write requests it
needs.
### Repro
Run against any S3-compatible endpoint (this was run against a local rustfs):
```python
import pyarrow.fs as fs
from pyiceberg.io.pyarrow import PyArrowFileIO
fs.initialize_s3(fs.S3LogLevel.Debug) # logs each request
io = PyArrowFileIO({
"s3.endpoint": "http://127.0.0.1:9000";,
"s3.access-key-id": "...",
"s3.secret-access-key": "...",
"s3.region": "us-east-1",
})
with io.new_output("s3://bucket/version-hint.text").create(overwrite=True)
as f:
f.write(b"1")
```
The debug log shows three requests for the one byte:
```
POST /bucket/version-hint.text?uploads
PUT /bucket/version-hint.text?partNumber=1&uploadId=...
POST /bucket/version-hint.text?uploadId=...
```
When the same write goes through an `S3FileSystem` built with
`allow_delayed_open=True`, the log shows a single `PUT
/bucket/version-hint.text`.
### Proposal
Pass `allow_delayed_open` in `_initialize_s3_fs`, controlled by a FileIO
property such as `s3.allow-delayed-open`. I'd suggest defaulting it to true,
since it only changes how small objects are uploaded. pyiceberg already
requires a pyarrow version that supports the option.
The workaround today is to subclass `PyArrowFileIO`, override
`_initialize_s3_fs`, and rebuild the filesystem with the option added. That
depends on a private method.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
