Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-22 Thread via GitHub
elphastori commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2901144875 @kevinjqliu Thanks for the context, I confirmed that the issue also occurs while paginating `namespaces` and `views`: - http://localhost:8181/v1/namespaces?pageSize=1 - htt

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102688184 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Flink 2.0: Remove the JUnit4 dependency [iceberg]

2025-05-22 Thread via GitHub
JeonDaehong commented on PR #13021: URL: https://github.com/apache/iceberg/pull/13021#issuecomment-2901402140 Thank you ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102680685 ## pyiceberg/table/__init__.py: ## @@ -1507,16 +1566,17 @@ def _parse_row_filter(expr: Union[str, BooleanExpression]) -> BooleanExpression: return

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102683614 ## pyiceberg/table/__init__.py: ## @@ -1507,16 +1566,17 @@ def _parse_row_filter(expr: Union[str, BooleanExpression]) -> BooleanExpression: return

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102700547 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Docs: Fix Flink upsert doc on equality fields requirement [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on code in PR #13127: URL: https://github.com/apache/iceberg/pull/13127#discussion_r2102701620 ## docs/docs/flink-writes.md: ## @@ -75,7 +75,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ``` !!! info -OVERW

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102715753 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102720237 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Run dependency license check to release script [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on code in PR #1367: URL: https://github.com/apache/iceberg-rust/pull/1367#discussion_r2102720796 ## scripts/release.sh: ## @@ -32,6 +32,12 @@ rc_version="${ICEBERG_VERSION_RC:-rc.1}" # Corresponding git repository branch git_branch=release-${release_versi

Re: [PR] Docs: Fix Flink upsert doc on equality fields requirement [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on code in PR #13127: URL: https://github.com/apache/iceberg/pull/13127#discussion_r2102701620 ## docs/docs/flink-writes.md: ## @@ -75,7 +75,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ``` !!! info -OVERW

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102697102 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102727321 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,181 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101331249 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1884,246 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101337805 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1884,246 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102674112 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/RewriteDataFilesConfig.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Sof

Re: [PR] AWS: pass storage credentials to s3 client [iceberg]

2025-05-22 Thread via GitHub
nastra closed pull request #13118: AWS: pass storage credentials to s3 client URL: https://github.com/apache/iceberg/pull/13118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102604780 ## dev/provision.py: ## @@ -413,3 +413,58 @@ ) spark.sql(f"ALTER TABLE {catalog_name}.default.test_empty_scan_ordered_str WRITE ORDERED BY id"

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101335471 ## pyiceberg/table/__init__.py: ## @@ -1092,6 +1096,61 @@ def scan( limit=limit, ) +# TODO: Consider more concise name +d

[PR] Docs: add Tinybird to the list of vendors and blog posts [iceberg]

2025-05-22 Thread via GitHub
futurepastori opened a new pull request, #13128: URL: https://github.com/apache/iceberg/pull/13128 This PR includes Tinybird in the list of vendors who support Iceberg, as well as it includes a link to a recent blog post on how to integrate Iceberg with Tinybird. -- This is an automated

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102619518 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/JdbcLockFactoryBuilder.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102615704 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/BaseLockFactoryBuilder.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102619101 ## dev/provision.py: ## @@ -413,3 +413,58 @@ ) spark.sql(f"ALTER TABLE {catalog_name}.default.test_empty_scan_ordered_str WRITE ORDERED BY id"

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101327716 ## pyiceberg/manifest.py: ## @@ -717,6 +717,14 @@ def fetch_manifest_entry(self, io: FileIO, discard_deleted: bool = True) -> List if n

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102631306 ## pyiceberg/table/__init__.py: ## @@ -1092,6 +1096,61 @@ def scan( limit=limit, ) +# TODO: Consider more concise name +d

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102634086 ## pyiceberg/table/__init__.py: ## @@ -1092,6 +1096,61 @@ def scan( limit=limit, ) +# TODO: Consider more concise name +d

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102637549 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/BaseLockFactoryBuilder.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102634086 ## pyiceberg/table/__init__.py: ## @@ -1092,6 +1096,61 @@ def scan( limit=limit, ) +# TODO: Consider more concise name +d

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102624828 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1888,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102645826 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/RewriteDataFilesConfig.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101335471 ## pyiceberg/table/__init__.py: ## @@ -1092,6 +1096,61 @@ def scan( limit=limit, ) +# TODO: Consider more concise name +d

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102772540 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1887,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102769661 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1887,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102776541 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1887,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102778115 ## pyiceberg/table/__init__.py: ## @@ -1834,116 +2186,79 @@ def plan_files(self) -> Iterable[FileScanTask]: for data_entry in data_entries

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102776961 ## pyiceberg/table/__init__.py: ## @@ -1810,6 +2159,9 @@ def plan_files(self) -> Iterable[FileScanTask]: ], ) ): +

Re: [I] Iceberg Table and View Unification Proposal [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on issue #13126: URL: https://github.com/apache/iceberg/issues/13126#issuecomment-2901552128 Sorry if i'm incorrect but this feels like an AI proposal. If you are a real human and would like to proceed please just respond with some details about what the clas

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102779860 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1887,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Core: Add basic classes for writing table format-version 4 [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13123: URL: https://github.com/apache/iceberg/pull/13123#discussion_r2102786877 ## core/src/test/java/org/apache/iceberg/DeleteFileIndexTestBase.java: ## @@ -50,7 +50,7 @@ public abstract class DeleteFileIndexTestBase< @Parameters(nam

Re: [PR] Core: Add basic classes for writing table format-version 4 [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13123: URL: https://github.com/apache/iceberg/pull/13123#discussion_r2102788382 ## core/src/test/java/org/apache/iceberg/TestCommitReporting.java: ## @@ -36,7 +36,7 @@ public class TestCommitReporting extends TestBase { @Parameters(na

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102786896 ## tests/integration/test_reads.py: ## @@ -1003,3 +1003,205 @@ def test_scan_with_datetime(catalog: Catalog) -> None: df = table.scan(row_filter=L

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102786896 ## tests/integration/test_reads.py: ## @@ -1003,3 +1003,205 @@ def test_scan_with_datetime(catalog: Catalog) -> None: df = table.scan(row_filter=L

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101322950 ## pyiceberg/table/snapshots.py: ## @@ -440,11 +440,40 @@ def ancestors_of(current_snapshot: Optional[Snapshot], table_metadata: TableMeta def ancestor

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102784767 ## tests/integration/test_reads.py: ## @@ -1003,3 +1003,205 @@ def test_scan_with_datetime(catalog: Catalog) -> None: df = table.scan(row_filter=L

Re: [PR] Core: Add basic classes for writing table format-version 4 [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13123: URL: https://github.com/apache/iceberg/pull/13123#discussion_r2102790880 ## core/src/test/resources/TableMetadataUnsupportedVersion.json: ## @@ -1,5 +1,5 @@ { - "format-version": 4, + "format-version": 5, Review Comment: nit:

Re: [I] Implement schema update [iceberg-go]

2025-05-22 Thread via GitHub
Shreyas220 commented on issue #430: URL: https://github.com/apache/iceberg-go/issues/430#issuecomment-2901277897 Hey folks 👋 , Apologies wasn't able to make much progress after the initial commits, should pick that up soon (today or tomorrow ) Things according left are - `com

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102604780 ## dev/provision.py: ## @@ -413,3 +413,58 @@ ) spark.sql(f"ALTER TABLE {catalog_name}.default.test_empty_scan_ordered_str WRITE ORDERED BY id"

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
pvary commented on PR #12979: URL: https://github.com/apache/iceberg/pull/12979#issuecomment-2901284713 Sorry for the late realization, but I think the `TableMaintenanceConfig` and the `LockFactoryConfig` values doesn't need the `rewrite` in their name. If we add more maintenance tasks thes

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102729622 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1888,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102729622 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1888,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2101329296 ## pyiceberg/table/__init__.py: ## @@ -1536,10 +1595,177 @@ def __init__( self.row_filter = _parse_row_filter(row_filter) self.selected

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102733224 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1888,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Make `dependencies.py generate` fail on cargo-deny error [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on code in PR #1366: URL: https://github.com/apache/iceberg-rust/pull/1366#discussion_r2102745732 ## scripts/dependencies.py: ## @@ -22,34 +22,41 @@ DIRS = [ "crates/iceberg", - -"crates/catalog/glue", "crates/catalog/hms", -"crates/catalog/m

[PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu opened a new pull request, #1370: URL: https://github.com/apache/iceberg-rust/pull/1370 ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Instead of hardcoding a few repos to check in `dependencies.py`, lets wal

Re: [PR] Make `dependencies.py generate` fail on cargo-deny error [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on code in PR #1366: URL: https://github.com/apache/iceberg-rust/pull/1366#discussion_r2102757484 ## scripts/dependencies.py: ## @@ -22,34 +22,41 @@ DIRS = [ "crates/iceberg", - -"crates/catalog/glue", "crates/catalog/hms", -"crates/catalog/m

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102756854 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1888,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Flink 2.0: Remove the JUnit4 dependency [iceberg]

2025-05-22 Thread via GitHub
nastra commented on PR #13021: URL: https://github.com/apache/iceberg/pull/13021#issuecomment-2901480859 @pvary could you please take a look as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102771559 ## pyiceberg/table/__init__.py: ## @@ -1688,102 +1887,252 @@ def _match_deletes_to_data_file(data_entry: ManifestEntry, positional_delete_ent r

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on PR #12598: URL: https://github.com/apache/iceberg/pull/12598#issuecomment-2901725915 @emkornfield Did we ever finish this vote? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #12598: URL: https://github.com/apache/iceberg/pull/12598#discussion_r2102878522 ## format/spec.md: ## @@ -1761,6 +1763,10 @@ The reference Java implementation uses a type 4 uuid and XORs the 4 most signifi Java writes `-1` for "no curr

[PR] Partition statistics metadata reading [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr opened a new pull request, #2033: URL: https://github.com/apache/iceberg-python/pull/2033 # Rationale for this change # Are these changes tested? # Are there any user-facing changes? -- This is an automated message from the Apac

[I] Support reading table metadata with partition statistics files [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr opened a new issue, #2034: URL: https://github.com/apache/iceberg-python/issues/2034 ### Feature Request / Improvement https://github.com/apache/iceberg/pull/12450 looks like it'll be merged soon, in which case it's plausible that table metadata will begin to appear w

Re: [PR] Partition statistics metadata reading [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2033: URL: https://github.com/apache/iceberg-python/pull/2033#discussion_r2102909289 ## pyiceberg/table/statistics.py: ## @@ -29,15 +29,24 @@ class BlobMetadata(IcebergBaseModel): properties: Optional[Dict[str, str]] = None -clas

Re: [PR] Partition statistics metadata reading [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2033: URL: https://github.com/apache/iceberg-python/pull/2033#discussion_r2102910004 ## pyiceberg/table/statistics.py: ## @@ -29,15 +29,24 @@ class BlobMetadata(IcebergBaseModel): properties: Optional[Dict[str, str]] = None -clas

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2102918617 ## pyiceberg/table/inspect.py: ## @@ -678,6 +689,28 @@ def all_manifests(self) -> "pa.Table": ) return pa.concat_tables(manifests_by_sn

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-22 Thread via GitHub
kevinjqliu commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2901811346 Nice i think those should cover all of them. I double checked with the rest spec * [list namespace](https://github.com/apache/iceberg/blob/91dff9886e6e6c494f6a970129f4c08487c

Re: [PR] Docs: Fix Flink upsert doc on equality fields requirement [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on code in PR #13127: URL: https://github.com/apache/iceberg/pull/13127#discussion_r2102701620 ## docs/docs/flink-writes.md: ## @@ -75,7 +75,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ``` !!! info -OVERW

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2102924653 ## pyiceberg/table/inspect.py: ## @@ -678,6 +689,28 @@ def all_manifests(self) -> "pa.Table": ) return pa.concat_tables(manifests_by_sn

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102993338 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -89,9 +129,6 @@ public static

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102997034 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -81,6 +87,40 @@ record ->

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2102998034 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -196,6 +196,18 @@ public static

Re: [PR] Expose Avro reader to PyIceberg [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1328: URL: https://github.com/apache/iceberg-rust/pull/1328#issuecomment-2901908181 I like #2 as well. The refactor should be less effort than scaffolding between python class and rust struct -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103007012 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRowLevelOperationsWithLineage.java: ## @@ -81,6 +87,40 @@ record ->

Re: [PR] [WIP] Spark4.0: RC -> official release [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2901943953 So soon! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Make `dependencies.py` check all subdirectories for cargo toml files [iceberg-rust]

2025-05-22 Thread via GitHub
kevinjqliu commented on PR #1370: URL: https://github.com/apache/iceberg-rust/pull/1370#issuecomment-2901960347 Note, the license check currently fails with: ``` Checking dependencies of /Users/kevinliu/repos/iceberg-rust/crates/integrations/cli error[rejected]: failed to satisfy li

Re: [PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c commented on PR #9335: URL: https://github.com/apache/iceberg/pull/9335#issuecomment-2901969371 @manuzhang Yes, I'll come back to it soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] AWS: Support multiple storage credential prefixes [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar merged PR #12799: URL: https://github.com/apache/iceberg/pull/12799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103006056 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103044192 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103044192 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103051361 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] Flink: port range distribution to v2 iceberg sink [iceberg]

2025-05-22 Thread via GitHub
rodmeneses commented on code in PR #12071: URL: https://github.com/apache/iceberg/pull/12071#discussion_r2103206341 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -645,72 +711,135 @@ private DataStream distributeDataStream(DataStream in

[PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c opened a new pull request, #9335: URL: https://github.com/apache/iceberg/pull/9335 ### Note to reviewers - Closes https://github.com/apache/iceberg/issues/8856 - Instead of returning `ManifestFile` in `BaseAllMetadataTableScan::reachableManifests`, we return a `Pair` from

Re: [PR] Core: Add reference snapshot ID/timestamps to AllEntriesTable and AllManifestsTable [iceberg]

2025-05-22 Thread via GitHub
hsiang-c commented on PR #9335: URL: https://github.com/apache/iceberg/pull/9335#issuecomment-2902285554 Thanks @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] feat (glue): Add Set Table property support for Glue [iceberg-go]

2025-05-22 Thread via GitHub
maninc closed pull request #378: feat (glue): Add Set Table property support for Glue URL: https://github.com/apache/iceberg-go/pull/378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-22 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2102674112 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/RewriteDataFilesConfig.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on code in PR #2031: URL: https://github.com/apache/iceberg-python/pull/2031#discussion_r2102680685 ## pyiceberg/table/__init__.py: ## @@ -1507,16 +1566,17 @@ def _parse_row_filter(expr: Union[str, BooleanExpression]) -> BooleanExpression: return

Re: [PR] Core: Add basic classes for writing table format-version 4 [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13123: URL: https://github.com/apache/iceberg/pull/13123#discussion_r2102787906 ## core/src/test/java/org/apache/iceberg/TestDeleteFiles.java: ## @@ -83,11 +83,9 @@ public class TestDeleteFiles extends TestBase { @Parameters(name = "f

Re: [PR] Incremental Append Scan [iceberg-python]

2025-05-22 Thread via GitHub
smaheshwar-pltr commented on PR #533: URL: https://github.com/apache/iceberg-python/pull/533#issuecomment-2901589298 Hi folks, I've put up an attempt at revival https://github.com/apache/iceberg-python/pull/2031 given inactivity here. Many thanks @hililiwei, and many apologies if you / anyo

Re: [PR] Spark 3.5: Verify base snapshot hasn't changed before commit in RemoveDanglingDeletesSparkAction [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on PR #13120: URL: https://github.com/apache/iceberg/pull/13120#issuecomment-2901589786 @dramaticlly @szehon-ho PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Docs: add column descriptions for entries metadata table [iceberg]

2025-05-22 Thread via GitHub
manuzhang commented on code in PR #13104: URL: https://github.com/apache/iceberg/pull/13104#discussion_r2102813536 ## docs/docs/spark-queries.md: ## @@ -301,6 +301,16 @@ SELECT * FROM prod.db.table.entries; | -- | -- | -- | -- | -- | -- | | 2 | 57897183625154 | 0 | 0 | {"cont

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103068654 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103080451 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [PR] AWS: pass storage credentials to s3 client [iceberg]

2025-05-22 Thread via GitHub
nastra commented on PR #13118: URL: https://github.com/apache/iceberg/pull/13118#issuecomment-2902047069 https://github.com/apache/iceberg/pull/12799 has been merged before I had the chance to merge your PR. Can you rebase this please so that we can get the tests in -- This is an automat

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
RussellSpitzer commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103091055 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Mani

Re: [PR] Docs: add column descriptions for entries metadata table [iceberg]

2025-05-22 Thread via GitHub
elphastori commented on code in PR #13104: URL: https://github.com/apache/iceberg/pull/13104#discussion_r2103074320 ## docs/docs/spark-queries.md: ## @@ -301,6 +301,16 @@ SELECT * FROM prod.db.table.entries; | -- | -- | -- | -- | -- | -- | | 2 | 57897183625154 | 0 | 0 | {"con

Re: [PR] [WIP] Spark4.0: RC -> official release [iceberg]

2025-05-22 Thread via GitHub
szehon-ho commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2902031997 Spark 4.0 release vote finally passed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-22 Thread via GitHub
elphastori opened a new pull request, #13129: URL: https://github.com/apache/iceberg/pull/13129 Allow listing tables, views and namespaces in rest catalog without `pageToken` query parameter when `pageSize` query parameter is included. Closes #13119 Examples: - http://localhost:81

Re: [PR] Spark: Fix row lineage inheritance for distributed planning [iceberg]

2025-05-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #13061: URL: https://github.com/apache/iceberg/pull/13061#discussion_r2103108339 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java: ## @@ -46,6 +47,7 @@ public static ManifestFileBean fromManifest(Ma

Re: [I] Cannot create a table from a pyarrow schema [iceberg-python]

2025-05-22 Thread via GitHub
DavidEscott commented on issue #2030: URL: https://github.com/apache/iceberg-python/issues/2030#issuecomment-2902134199 Not in my use-case, because I also want to specify the `PartitionSpec` for `create_table`. From what I can tell the signature of `catalog.create_table` doesn't work at pr

  1   2   3   >