Re: [PR] Implement column projection [iceberg-python]

Fokko commented on PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#issuecomment-2648011772 Thanks for working on this @gabeiglio, and thanks for the review @kevinjqliu 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Implement column projection [iceberg-python]

Fokko merged PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1941979863 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1218,45 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1941731592 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1218,45 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1940542160 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1218,45 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1938141310 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1127,129 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1929627882 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1127,127 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1925757635 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1265,26 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_time

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1922764225 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1265,26 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_tim

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1919197614 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1913225607 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1913166114 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1912344028 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1912328061 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906149085 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906149085 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906142055 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values(

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1906110946 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1903118090 ## pyiceberg/io/pyarrow.py: ## @@ -1286,14 +1310,20 @@ def _task_to_record_batches( continue output_batches = arr

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1903117970 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1127,63 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_ta

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1902357065 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1127,63 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1902352957 ## pyiceberg/io/pyarrow.py: ## @@ -1286,14 +1310,20 @@ def _task_to_record_batches( continue output_batches = ar

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1902247982 ## pyiceberg/io/pyarrow.py: ## @@ -1286,14 +1310,20 @@ def _task_to_record_batches( continue output_batches = arr

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1902240839 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1127,63 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_ta

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1901382763 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1257,20 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_tim

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1901092863 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899080083 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070927 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1257,20 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_timestam

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070896 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1899070416 ## pyiceberg/io/pyarrow.py: ## @@ -1216,6 +1216,25 @@ def _field_id(self, field: pa.Field) -> int: return -1 +def _get_column_projection_values( +

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1895050481 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_timestam

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894504972 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894502208 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894502208 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894425789 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894421818 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894421818 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894415796 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894415796 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894415796 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894410288 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894288136 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_time

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894286043 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_time

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894115453 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_timestam

Re: [PR] Implement column projection [iceberg-python]

Fokko commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1894112102 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_timestam

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1893108397 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1893009264 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_t

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1892693244 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_tim

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1892677874 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_time

Re: [PR] Implement column projection [iceberg-python]

gabeiglio commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1892677874 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1238,29 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_time

Re: [PR] Implement column projection [iceberg-python]

kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1892511992 ## tests/io/test_pyarrow.py: ## @@ -1122,6 +1123,110 @@ def test_projection_concat_files(schema_int: Schema, file_int: str) -> None: assert repr(result_

[PR] Implement column projection [iceberg-python]

gabeiglio opened a new pull request, #1443: URL: https://github.com/apache/iceberg-python/pull/1443 This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection [rules](https://iceberg.apache.org/spec/#column-projection) -- This