Re: [PR] Cache Manifest files [iceberg-python]

2024-09-10 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2340823370 rebased off main! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Cache Manifest files [iceberg-python]

2024-09-09 Thread via GitHub
corleyma commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1751060909 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,13 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache Review Comment: is 128 big enough?

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-12 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2163505455 I created a new PR against my fork, and once the GitHub actions failed, I manually re-tried them. https://github.com/chinmay-bhat/iceberg-python/pull/1/checks?sha=8c2e79a9c62

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-12 Thread via GitHub
HonahX commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2162422611 > Is this a duck db problem? Or do I need to open a new PR (from main branch + my changes) to resolve it? I tried this PR locally and did not observe this issue. My testing platform

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-11 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2161194170 > Being able to configure (and also disable) the cachine would be a very nice touch Hi @Fokko, I'm curious to know why we would want to allow custom sized manifest caches

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
Fokko commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2159288156 @chinmay-bhat Looks like the CI is still a bit sad :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
MehulBatra commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632922261 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,13 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache Review Comment: Agree the default

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
Fokko commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2159087897 Being able to configure (and also disable) the cachine would be a very nice touch Op ma 10 jun 2024 om 09:52 schreef Chinmay Bhat ***@***.***> > ***@***. commented o

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
MehulBatra commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632922261 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,13 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache Review Comment: Agree the default

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2157626790 I'm not sure why `test_duckdb_url_import` continues to fail with the same error. I've rebased onto latest main branch, and the test runs locally. I also don't see this issue in

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
chinmay-bhat commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632758206 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,13 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache Review Comment: As [the default

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
MehulBatra commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632704500 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,13 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache Review Comment: what are your thou

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-10 Thread via GitHub
MehulBatra commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2157521431 @chinmay-bhat @Fokko Sorry for the last-minute feedback, but wouldn't it be better if we explicitly pass (maxsize=128) for the lru_cache annotation, When a new call comes in, t

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2157332920 @Fokko @HonahX Thanks for the review!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
Fokko commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632590946 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,15 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache +def _manifests(io: FileIO, manifest_list:

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
Fokko commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632590946 ## pyiceberg/table/snapshots.py: ## @@ -228,6 +229,15 @@ def __eq__(self, other: Any) -> bool: ) +@lru_cache +def _manifests(io: FileIO, manifest_list:

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
Fokko commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632371435 ## pyiceberg/table/snapshots.py: ## @@ -247,12 +248,19 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
chinmay-bhat commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632346194 ## pyiceberg/table/snapshots.py: ## @@ -247,12 +248,19 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-09 Thread via GitHub
Fokko commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1632224368 ## pyiceberg/table/snapshots.py: ## @@ -247,12 +248,19 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
chinmay-bhat commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625482510 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2146758073 Hmm, it's my first time to see this error. I've merged a PR that bumps `duckdb` to 1.0.0: #793. Hope that can fix the issue -- This is an automated message from the Apache Git Servi

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-03 Thread via GitHub
chinmay-bhat commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2144956518 One test is failing CI (test_duckdb_url_import). ``` FAILED tests/integration/test_writes/test_writes.py::test_duckdb_url_import - duckdb.duckdb.IOException: IO Error: