acake88 opened a new pull request, #3242: URL: https://github.com/apache/iceberg-python/pull/3242
<h1 style="margin: 0px; padding: 0px; box-sizing: border-box; margin-block: 0 var(--markdown-line-gap); font-weight: 600; font-size: 13px; color: rgb(59, 59, 59); font-family: Inter, -apple-system, "system-ui", "Segoe UI (Custom)", Roboto, "Helvetica Neue", "Open Sans (Custom)", system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Rationale for this change</h1><p style="margin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-content-gap); padding-left: 0px; box-sizing: border-box; color: rgb(59, 59, 59); font-family: Inter, -apple-system, "system-ui", "Segoe UI (Custom)", Roboto, "Helvetica Neue", "Open Sans (Custom)", system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><a href="https://www.tencentcloud.com/products/cos"; target="_blank" rel="noopener noreferrer" node="[object Object]" style="margin: 0px; padding: 0px; box-sizing: border-box; color: var(--cb-vscode-textLink-foreground); text-decoration: none;">Tencent Cloud Object Storage (COS)</a><span> </span>is one of the major obje ct storage services in China, widely used in big data workloads with Spark, Flink, and Hive. Hadoop already supports<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cosn://</code><span> </span>scheme natively via<span> </span><a href="https://hadoop.apache.org/docs/stable/hadoop-cos/cloud-storage/index.html"; target="_blank" rel="noopener noreferrer" node="[object Object]" style="margin: 0px; padding: 0px; box-sizing: border-box; color: var(--cb-vscode-textLink-foreground); text-decoration: none;">hadoop-cos</a>, but PyIceberg currently has<span> </span><strong style="margin: 0px; padding: 0px; box-sizing: border-box; font-weight: 600;">no COS support</strong><span> </span>— users cannot access Iceberg tables stored on COS from Python.</p><p style="m argin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-content-gap); padding-left: 0px; box-sizing: border-box; color: rgb(59, 59, 59); font-family: Inter, -apple-system, "system-ui", "Segoe UI (Custom)", Roboto, "Helvetica Neue", "Open Sans (Custom)", system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">This PR adds COS as a storage backend via the fsspec-compatible<span> </span><a href="https://pypi.org/project/cosfs/"; target="_blank" rel="noopener noreferrer" nod e="[object Object]" style="margin: 0px; padding: 0px; box-sizing: border-box; color: var(--cb-vscode-textLink-foreground); text-decoration: none;"><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-textLink-foreground); pointer-events: none; text-decoration: none;">cosfs</code></a><span> </span>library, following the exact same pattern as existing integrations (S3 via<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">s3fs</code>, Azure via<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-mark down-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">adlfs</code>, GCS via<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">gcsfs</code>, HuggingFace via<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">huggingface-hub</code>).</p><p style="margin: 0px; padding: 0px; box-sizing: border-box; color: rgb(59, 59, 59); font-family: Inter, -apple-system, "system-ui", "Segoe UI (Custom)", Roboto, "Helvetica Neue", "Open Sans (Custom)&q uot;, system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="margin: 0px; padding: 0px; box-sizing: border-box; font-weight: 600;">Changes:</strong></p><ul style="margin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-content-gap); padding-left: var(--markdown-list-padding-left); box-sizing: border-box; list-style-type: disc; color: rgb(59, 59, 59); font-family: Inter, -apple-system, "system-ui", "Segoe UI (Custom)", Roboto, "Helvetica Neue", "Open Sans (Cus tom)", system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="margin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-line-gap); padding-left: 0px; box-sizing: border-box;"><strong style="margin: 0px; padding: 0px; box-sizing: border-box; font-weight: 600;"><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #ccccc c); pointer-events: none;">pyiceberg/io/__init__.py</code></strong>: Added 5 COS configuration properties (<code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos.secret-id</code>,<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos.secret-key</code>,<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos.session-token</ code>,<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos.region</code>,<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos.endpoint</code>) and registered<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cosn://</code><span> </span>/<span> </span><code class="" style="margin: 0px; padding: 0px; b ox-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cos://</code><span> </span>schemes in<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">SCHEMA_TO_FILE_IO</code></li><li style="margin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-line-gap); padding-left: 0px; box-sizing: border-box;"><strong style="margin: 0px; padding: 0px; box-sizing: border-box; font-weight: 600;"><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: va r(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">pyiceberg/io/fsspec.py</code></strong>: Added<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">_cosn()</code><span> </span>factory function and registered<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cosn</code><span> </span>/<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-f oreground, #cccccc); pointer-events: none;">cos</code><span> </span>in<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">SCHEME_TO_FS</code></li><li style="margin: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: var(--markdown-line-gap); padding-left: 0px; box-sizing: border-box;"><strong style="margin: 0px; padding: 0px; box-sizing: border-box; font-weight: 600;"><code class="" style="margin: 0px; padding: 0px; box-sizing: border-box; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">pyproject.toml</code></strong>: Added<span> </span><code class="" style="margin: 0px; padding: 0px; box-sizing: border-bo x; padding-inline: 4px; border-radius: 3px; background-color: var(--cb-markdown-code-bg-color, rgba(255, 255, 255, .1)); color: var(--cb-vscode-editor-foreground, #cccccc); pointer-events: none;">cosfs>=2026.3.25</code><span> </span>as an optional dependency</li></ul> <br class="Apple-interchange-newline">Rationale for this change [Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is one of the major object storage services in China, widely used in big data workloads with Spark, Flink, and Hive. Hadoop already supports cosn:// scheme natively via [hadoop-cos](https://hadoop.apache.org/docs/stable/hadoop-cos/cloud-storage/index.html), but PyIceberg currently has no COS support — users cannot access Iceberg tables stored on COS from Python. This PR adds COS as a storage backend via the fsspec-compatible [cosfs](https://pypi.org/project/cosfs/) library, following the exact same pattern as existing integrations (S3 via s3fs, Azure via adlfs, GCS via gcsfs, HuggingFace via huggingface-hub). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
