David-Banquet opened a new pull request, #4109: URL: https://github.com/apache/iceberg-python/pull/4109
Closes #4093 # Rationale for this change pyarrow's `S3FileSystem.open_output_stream` starts a multipart upload as soon as the stream opens, so `PyArrowFileIO` writes every S3 object, however small, with three requests: `CreateMultipartUpload`, `UploadPart` and `CompleteMultipartUpload`. pyarrow 21 added `allow_delayed_open` for this: a stream that closes before reaching the part size (10 MiB) is sent as a single `PutObject`. `_initialize_s3_fs` builds the filesystem from a fixed set of properties, so there was no way to turn it on. This adds `s3.allow-delayed-open`, passed to `S3FileSystem` for `s3://` and `oss://`. It is off by default, and setting it on pyarrow < 21 raises the same `ImportError` as `s3.ssl-ca-cert`. # Are these changes tested? Yes. `test_pyarrow_s3_session_properties_with_allow_delayed_open` checks the option reaches `S3FileSystem`, and `test_pyarrow_s3_allow_delayed_open_requires_supported_pyarrow_version` checks the version guard. Against real S3 (eu-west-1, pyarrow 25.0.1), counting requests from the SDK debug log: | Write | Today | `s3.allow-delayed-open=true` | |---|---|---| | 1-byte `version-hint.text` | 3 (multipart) | 1 `PutObject` | | `create_table` + `append` of 3 rows (SQL catalog: 1 data file, 1 manifest, 1 manifest list, 2 `metadata.json`) | 15 | 5 | | 6 MB file | 3 | 1 | | 11 MB file | multipart, 2 parts | same | # Are there any user-facing changes? Yes, a new FileIO property, documented in `configuration.md`. Nothing changes unless it is set. With it on, an error such as a missing bucket or a denied write is raised when the file is closed instead of when it is opened. It is the same `OSError`, and Java's `S3OutputStream` already behaves this way below its multipart threshold. I left the default off so the existing behavior does not move; happy to flip it if you would rather match Java. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
