Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2584573283 Thanks @smaheshwar-pltr for working on this and @Fokko for the review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu merged PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911453761 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911421207 ## pyiceberg/io/pyarrow.py: ## @@ -2622,13 +2631,15 @@ def _dataframe_to_data_files( property_name=TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911421207 ## pyiceberg/io/pyarrow.py: ## @@ -2622,13 +2631,15 @@ def _dataframe_to_data_files( property_name=TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2584223657 > I think we'd also want to add docs around this feature! Maybe similar to [FileIO](https://py.iceberg.apache.org/configuration/#fileio), we can add a new section about Loc

Re: [PR] Support Location Providers [iceberg-python]

Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911379496 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] Support Location Providers [iceberg-python]

Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911368008 ## mkdocs/docs/api.md: ## @@ -1077,6 +1077,7 @@ with table.update_schema() as update: with table.update_schema() as update: update.add_column(("details", "co

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911271050 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911203978 ## mkdocs/docs/api.md: ## @@ -1077,6 +1077,7 @@ with table.update_schema() as update: with table.update_schema() as update: update.add_column(("det

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911198932 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadat

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911172676 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadata: Ta

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911149088 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911146551 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadat

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1911063518 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1910352044 ## pyiceberg/table/__init__.py: ## @@ -187,6 +187,14 @@ class TableProperties: WRITE_PARTITION_SUMMARY_LIMIT = "write.summary.partition-limit"

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2581158650 > This matches the behavior of the [Java implementation](https://iceberg.apache.org/docs/nightly/custom-catalog/#custom-location-provider-implementation). However, if we're

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909356132 ## tests/integration/test_writes/test_partitioned_writes.py: ## @@ -280,6 +280,43 @@ def test_query_filter_v1_v2_append_null( assert df.where(f"

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909309033 ## tests/integration/test_writes/test_writes.py: ## @@ -285,6 +285,33 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909339914 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadat

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909339914 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadat

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909339914 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadat

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909330087 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909330087 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909315124 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909315124 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909315124 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909309033 ## tests/integration/test_writes/test_writes.py: ## @@ -285,6 +285,33 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909299027 ## tests/table/test_locations.py: ## @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909297301 ## tests/table/test_locations.py: ## @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909284421 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909281754 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909263229 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909263268 ## pyiceberg/io/pyarrow.py: ## @@ -2234,7 +2235,9 @@ def data_file_statistics_from_parquet_metadata( ) -def write_file(io: FileIO, table_metadata: Ta

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909052866 ## pyiceberg/table/__init__.py: ## @@ -1627,6 +1632,67 @@ class AddFileTask: partition_field_value: Record +class LocationProvider(ABC): Review

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1909052866 ## pyiceberg/table/__init__.py: ## @@ -1627,6 +1632,67 @@ class AddFileTask: partition_field_value: Record +class LocationProvider(ABC): Review

Re: [PR] Support Location Providers [iceberg-python]

jiakai-li commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1908109298 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1905630618 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Support Location Providers [iceberg-python]

kevinjqliu commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1901229046 ## pyiceberg/io/pyarrow.py: ## @@ -2622,13 +2631,15 @@ def _dataframe_to_data_files( property_name=TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,

Re: [PR] Support Location Providers [iceberg-python]

Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1899070323 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] Support Location Providers [iceberg-python]

Fokko commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1899068942 ## pyiceberg/table/__init__.py: ## @@ -1627,6 +1632,67 @@ class AddFileTask: partition_field_value: Record +class LocationProvider(ABC): Review Comment:

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1895864213 ## pyiceberg/io/pyarrow.py: ## @@ -2622,13 +2631,15 @@ def _dataframe_to_data_files( property_name=TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2559905325 @Fokko, think this is ready for review now! I've implemented this for write codepaths - `add_files` seems like it should just add the files specified without transfor

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1895864213 ## pyiceberg/io/pyarrow.py: ## @@ -2622,13 +2631,15 @@ def _dataframe_to_data_files( property_name=TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1894364450 ## tests/table/test_locations.py: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1894364450 ## tests/table/test_locations.py: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1894365471 ## tests/table/test_locations.py: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1894365471 ## tests/table/test_locations.py: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1894364450 ## tests/table/test_locations.py: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893981856 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2557102323 @Fokko (maybe @kevinjqliu also?) I've smoke-tested these changes locally with a catalog (hashing is the same as the Java implementation too), would love to get a first pass

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893986378 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893983962 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893983962 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893981856 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893979732 ## pyiceberg/table/__init__.py: ## @@ -192,6 +195,14 @@ class TableProperties: WRITE_PARTITION_SUMMARY_LIMIT = "write.summary.partition-limit"

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893970505 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#issuecomment-2557074217 @Fokko (maybe @kevinjqliu also?) I've smoke-tested these changes locally with a catalog (hashing is the same as the Java implementation too), would love to get a first pass

Re: [PR] Support Location Providers [iceberg-python]

smaheshwar-pltr commented on code in PR #1452: URL: https://github.com/apache/iceberg-python/pull/1452#discussion_r1893970505 ## pyiceberg/table/locations.py: ## @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme