mccormickt12 commented on code in PR #2251:
URL: https://github.com/apache/iceberg-python/pull/2251#discussion_r2240407642
##########
mkdocs/docs/configuration.md:
##########
@@ -105,6 +105,33 @@ You can also set the FileIO explicitly:
For the FileIO there are several configuration options available:
+### PyArrow FileSystem Extra Properties
+
+When using `PyArrowFileIO`, any properties with filesystem specific prefixes
that are not explicitly handled by PyIceberg will be passed to the underlying
PyArrow filesystem implementations.
+
+To use these properties, follow the format:
+
+```txt
+{fs_scheme}.{parameter_name}
+```
+
+- {fs_scheme} is the filesystem scheme (e.g., s3, hdfs, gcs).
+- {parameter_name} must match the name expected by the PyArrow filesystem.
+- Property values must use the correct type expected by the underlying
filesystem (e.g., string, integer, boolean).
+
+Below are examples of supported prefixes and how the properties are passed
through:
+
+<!-- markdown-link-check-disable -->
+
+| Property Prefix | FileSystem
| Example |
Description |
+|-----------------|------------------------------------------------------------------------------------------------------|-----------------------------|-----------------------------------------------------|
+| `s3.` |
[S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html)
| `s3.load_frequency=900` | Passed as `load_frequency=900` to
S3FileSystem |
+| `hdfs.` |
[HadoopFileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html)
| `hdfs.replication=3` | Passed as `replication=3` to HadoopFileSystem
|
Review Comment:
oh so these are only for the params to pyarrow. all the params in
`*-site.xml` files are unchanged and don't need this prefix right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
