David-Banquet commented on issue #4093: URL: https://github.com/apache/iceberg-python/issues/4093#issuecomment-6145569837
I ran this against real S3 (eu-west-1, pyarrow 25.0.1), counting requests from the same SDK debug log: | Write | Today | With `allow_delayed_open=True` | |---|---|---| | 1-byte `version-hint.text` | 3 (multipart) | 1 `PutObject` | | `create_table` + `append` of 3 rows (SQL catalog: 1 data file, 1 manifest, 1 manifest list, 2 `metadata.json`) | 15 | 5 | | 6 MB file | 3 | 1 | | 11 MB file | multipart, 2 parts | same | The cutoff is arrow's 10 MiB part size, so larger files are unchanged. Reads are unaffected: the option is only used by the output stream. On the default, Java's `S3OutputStream` already behaves like this. It buffers locally and only starts a multipart upload above `s3.multipart.threshold` × part size (48 MiB by default), and below that it sends one `PutObject` on `close()`. So write errors surface on close there too. With the option on in pyarrow, an error such as a missing bucket or a denied write moves from `create()` to `close()`. It's the same `OSError`, and the two places that call `create()` outside a `with` block close the stream explicitly, so the caller still sees it. @nhobin219 are you planning to open the PR? If you'd rather not, I have a patch with a test and the docs entry and can open it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
