Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
sungwy merged PR #955: URL: https://github.com/apache/iceberg-python/pull/955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690550621 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690505567 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690505567 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690408866 ## tests/integration/test_deletes.py: ## @@ -417,3 +452,107 @@ def test_delete_truncate(session_catalog: RestCatalog) -> None: assert len(entries) == 1

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690328141 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690328141 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690301032 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690176614 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1689929311 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,165 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1689900531 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,165 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688973596 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +594,127 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688971266 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,165 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688687715 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +594,127 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688687715 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +594,127 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688686566 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,165 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688686566 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,165 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688678348 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +594,114 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688622860 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +594,114 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1688410579 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,160 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687991934 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,160 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687981851 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,160 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687978956 ## tests/io/test_pyarrow_visitor.py: ## @@ -580,3 +590,57 @@ def test_pyarrow_schema_ensure_large_types(pyarrow_schema_nested_without_ids: pa ), ])

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-23 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687535523 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,160 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-22 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687434820 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,124 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-22 Thread via GitHub
syun64 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1687162697 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,124 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-22 Thread via GitHub
syun64 commented on PR #955: URL: https://github.com/apache/iceberg-python/pull/955#issuecomment-2243845630 Proposed implementation is consistent with Spark Iceberg's behavior. For a given Iceberg table: ``` spark.read.table("demo.tacocat.test_null").show() >> +++