Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-20 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2364203606 Thank you @JE-Chen for the contribution and @sungwy for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-20 Thread via GitHub
kevinjqliu merged PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-20 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2364133054 > @JE-Chen I was able to run integration tests for this PR locally. And given that CI also pass, let's merge this and debug your VM environment issue separately. How does that soun

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-20 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2364130196 @JE-Chen I was able to run integration tests for this PR locally. And given that CI also pass, let's merge this and debug your VM environment issue separately. How does that

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-19 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2362965089 My Windows environment has the same pull issue. ![{ADD7151F-C1E1-48BE-96AB-F1575E271E57}](https://github.com/user-attachments/assets/0a5f6d74-c2ba-4694-be69-bd31f9adeb4d)

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-19 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2362951863 seems like its an issue with your docker installation. https://github.com/apache/iceberg-python/blob/41a3c8ea4e9c3592331755970bfc38f52eaf3fd8/dev/docker-compose-integrati

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-19 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2362928724 I’m trying to remove all docker images and re-run make-integration, and I noticed this. So, I think the test failed because: Can’t pull the python-integration image (access de

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2359019036 Gotcha, thanks! I think the CI runs in ubuntu, so it should mirror your own setup -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358968377 > ``` >Suppressed: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (3227648) > Allocator(toArrowBatchIterator) 0/3227648/

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
kevinjqliu commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358919234 ``` Suppressed: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (3227648) Allocator(toArrowBatchIterator) 0/3227648/3

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358640052 Look like all error cause by: `pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.` There a

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
sungwy commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358538854 Thank you for sending over the reproducible steps @JE-Chen - I do not use PyCharm for my development (I'm a VSCode user) so it might take a while for me to reproduce your setup.

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358520838 * Open Pycharm * Open iceberg project * Create new venv using python 3.11.9 * pip install poetry * make install * make test-integration * Get this message four ti

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
sungwy commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358429830 > Log of make test-integration on my VM has so many errors. Any ideas? (It succeeds on CI Test). What kind of errors are you running into @JE-Chen ? Could you provide us an e

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-18 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2358170897 `make test` log: ``` warnings summary ==

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-17 Thread via GitHub
JE-Chen commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2356354267 Hi @sungwy I have already fixed the problem that ruff found. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Use ArrowScan.to_table to replace project_table [iceberg-python]

2024-09-17 Thread via GitHub
sungwy commented on PR #1180: URL: https://github.com/apache/iceberg-python/pull/1180#issuecomment-2356057167 Hi @JE-Chen thank you for putting together this PR! It looks like these were missed in the initial refactoring. Running the CI now. -- This is an automated message from the Apache