Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-20 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2158556942 ## pyiceberg/io/__init__.py: ## @@ -82,6 +82,10 @@ ADLS_CLIENT_ID = "adls.client-id" ADLS_CLIENT_SECRET = "adls.client-secret" ADLS_ACCOUNT_HOST = "a

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-20 Thread via GitHub
kevinjqliu merged PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-20 Thread via GitHub
kevinjqliu commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2991981353 Thanks for working on this @NikitaMatskevich and thanks @Fokko for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-20 Thread via GitHub
kevinjqliu commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2991915984 actually i just push the `make lint` change :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-20 Thread via GitHub
kevinjqliu commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2991873104 @NikitaMatskevich looks like the linter errored, could you run `make lint`? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157932408 ## pyiceberg/io/pyarrow.py: ## @@ -197,6 +204,7 @@ MAP_VALUE_NAME = "value" DOC = "doc" UTC_ALIASES = {"UTC", "+00:00", "Etc/UTC", "Z"} +MIN_PYARROW_VERSIO

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157577217 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157468740 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@pytest

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157465863 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@pytest

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157322524 ## pyiceberg/io/pyarrow.py: ## @@ -475,6 +486,42 @@ def _initialize_s3_fs(self, netloc: Optional[str]) -> FileSystem: return S3FileSystem(**

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157328970 ## pyiceberg/io/pyarrow.py: ## @@ -197,6 +204,7 @@ MAP_VALUE_NAME = "value" DOC = "doc" UTC_ALIASES = {"UTC", "+00:00", "Etc/UTC", "Z"} +MIN_PYARROW_

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157328970 ## pyiceberg/io/pyarrow.py: ## @@ -197,6 +204,7 @@ MAP_VALUE_NAME = "value" DOC = "doc" UTC_ALIASES = {"UTC", "+00:00", "Etc/UTC", "Z"} +MIN_PYARROW_

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157321373 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157299424 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@pytest

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157298498 ## tests/io/test_pyarrow.py: ## @@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None: @pytest.mark.gcs -@pytest

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
kevinjqliu commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2157259714 ## pyiceberg/io/pyarrow.py: ## @@ -394,6 +402,9 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste elif scheme in {

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-19 Thread via GitHub
NikitaMatskevich commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2987917962 > I am one of those users, does this support authentication using auth token ? (not sas token) From the [docs](https://arrow.apache.org/docs/python/filesystems.html

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-18 Thread via GitHub
djouallah commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2985979949 I am one of those users, does this support authentication using auth token ? (not sas token) -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-18 Thread via GitHub
NikitaMatskevich commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2154869508 ## pyproject.toml: ## @@ -62,7 +62,7 @@ pyparsing = ">=3.1.0,<4.0.0" zstandard = ">=0.13.0,<1.0.0" tenacity = ">=8.2.3,<10.0.0" pyroaring = ">=1.0.0,

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-18 Thread via GitHub
NikitaMatskevich commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2984168003 Thank you for looking into this! Sorry, indeed, missed some formatting issues. Now it should be fine. > @NikitaMatskevich Thanks for working on this, I know a lot of

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-17 Thread via GitHub
Fokko commented on PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#issuecomment-2981224114 @NikitaMatskevich Thanks for working on this, I know a lot of users are waiting for this. It looks like some tests are failing (you can run the linters locally using `make lint`), co

Re: [PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-17 Thread via GitHub
Fokko commented on code in PR #2111: URL: https://github.com/apache/iceberg-python/pull/2111#discussion_r2152806261 ## pyproject.toml: ## @@ -62,7 +62,7 @@ pyparsing = ">=3.1.0,<4.0.0" zstandard = ">=0.13.0,<1.0.0" tenacity = ">=8.2.3,<10.0.0" pyroaring = ">=1.0.0,<2.0.0" -py

[PR] Support ADLS with Pyarrow file IO [iceberg-python]

2025-06-17 Thread via GitHub
NikitaMatskevich opened a new pull request, #2111: URL: https://github.com/apache/iceberg-python/pull/2111 # Rationale for this change Starting from version 20, PyArrow supports ADLS filesystem. This PR adds Pyarrow Azure support to Pyiceberg. PyArrow is the