Re: [PR] Spark: when doing rewrite_data_files, check for partitioning schema compatibility [iceberg]

2025-04-04 Thread via GitHub
pvary commented on code in PR #12651: URL: https://github.com/apache/iceberg/pull/12651#discussion_r2026872895 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -265,6 +265,22 @@ public boolean equals(Object other) { return Arrays.equals(fields, that.fields)

Re: [PR] Doc: Add doc for flink exec config [iceberg]

2025-04-04 Thread via GitHub
pvary commented on code in PR #12691: URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026603208 ## docs/docs/flink-configuration.md: ## @@ -198,4 +198,42 @@ they are. This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For {@link

Re: [PR] feat: sort order [iceberg-cpp]

2025-04-04 Thread via GitHub
zhjwpku commented on code in PR #59: URL: https://github.com/apache/iceberg-cpp/pull/59#discussion_r2026148130 ## src/iceberg/sort_field.cc: ## @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See th

Re: [I] Cannot scan empty table [iceberg-rust]

2025-04-04 Thread via GitHub
Xuanwo closed issue #1145: Cannot scan empty table URL: https://github.com/apache/iceberg-rust/issues/1145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Cannot scan empty table [iceberg-rust]

2025-04-04 Thread via GitHub
Xuanwo closed issue #1145: Cannot scan empty table URL: https://github.com/apache/iceberg-rust/issues/1145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] fix: support empty scans [iceberg-rust]

2025-04-04 Thread via GitHub
Xuanwo merged PR #1166: URL: https://github.com/apache/iceberg-rust/pull/1166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Config for deciding whether to use Iceberg Time type [iceberg]

2025-04-04 Thread via GitHub
kumarpritam863 commented on PR #11174: URL: https://github.com/apache/iceberg/pull/11174#issuecomment-2780148892 @braislchao sorry for this late reply but this was closed due to being stale so could not reply. I have already opened a PR with the SMT logic and will be adding that once it is

Re: [PR] API: Define RepairManifests action interface [iceberg]

2025-04-04 Thread via GitHub
github-actions[bot] commented on PR #10784: URL: https://github.com/apache/iceberg/pull/10784#issuecomment-2774026359 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2768313133 > Looks like the magic words don't work. Can you submit an empty or trivial commit to retrigger it? Another valid workaround for this is to just close/re-open the PR and CI will b

Re: [PR] AWS: Fix Catalog URI within VendedCredentialsProvider [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12612: URL: https://github.com/apache/iceberg/pull/12612#issuecomment-2768309015 > I'm traveling for work this week. There's one more test that needs to be fixed: `refreshCredentialsEndpointWithOverridingOAuthToken()`. I'll take a look later tomorrow. No worri

Re: [PR] Core: Use InputFile.location() Instead of Direct Object Reference in Error Messages [iceberg]

2025-04-04 Thread via GitHub
ajantha-bhat commented on code in PR #12716: URL: https://github.com/apache/iceberg/pull/12716#discussion_r2029317787 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -175,7 +175,8 @@ public static ViewMetadata read(InputFile file) { codec ==

[PR] Added support for evolving the partition of the table [iceberg]

2025-04-04 Thread via GitHub
kumarpritam863 opened a new pull request, #12723: URL: https://github.com/apache/iceberg/pull/12723 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Build: Bump com.azure:azure-sdk-bom from 1.2.31 to 1.2.33 [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12689: URL: https://github.com/apache/iceberg/pull/12689#issuecomment-2768401280 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: Add support for Avro's timestamp-millis LogicalType in DataReader [iceberg]

2025-04-04 Thread via GitHub
github-actions[bot] closed pull request #12397: Core: Add support for Avro's timestamp-millis LogicalType in DataReader URL: https://github.com/apache/iceberg/pull/12397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Core: use ReachableFileCleanup when table has discontinuous snapshots [iceberg]

2025-04-04 Thread via GitHub
github-actions[bot] commented on PR #12261: URL: https://github.com/apache/iceberg/pull/12261#issuecomment-2779940034 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Flink: Emit watermarks for the new timestamp_ns and timestamptz_ns types [iceberg]

2025-04-04 Thread via GitHub
github-actions[bot] commented on issue #9137: URL: https://github.com/apache/iceberg/issues/9137#issuecomment-2779939948 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-04 Thread via GitHub
wypoon commented on PR #12681: URL: https://github.com/apache/iceberg/pull/12681#issuecomment-2779908426 @pvary I have rebased on main. I see that this picked up https://github.com/apache/iceberg/commit/c661a71091e496393c743ddd879d9e1a0f2747b2 / https://github.com/apache/iceberg/pull/126

Re: [PR] Parquet: Implement Variant metrics [iceberg]

2025-04-04 Thread via GitHub
rdblue commented on PR #12496: URL: https://github.com/apache/iceberg/pull/12496#issuecomment-2743888207 Thanks for the reviews, @aihuaxu and @amogh-jahagirdar! I'll leave this open a bit longer because I think @danielcweeks also wanted to take a look. I'll follow up with him. -- This is

Re: [PR] feat: check whether table ops conflict when committing [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1816: URL: https://github.com/apache/iceberg-python/pull/1816#discussion_r2005732527 ## pyiceberg/table/update/snapshot.py: ## @@ -239,7 +241,21 @@ def _summary(self, snapshot_properties: Dict[str, str] = EMPTY_DICT) -> Summary: trunc

Re: [I] Spark returns incorrect results when reading Parquet bloom filters created by Trino [iceberg]

2025-04-04 Thread via GitHub
hsiang-c commented on issue #12458: URL: https://github.com/apache/iceberg/issues/12458#issuecomment-2779889507 @ebyhr May I ask the Trino version you're using? I have: Trino: 474 Spark: 3.5.3 I got 1 from Iceberg 1.7.0, 1.8.0 and 1.8.1 -- This is an automated

Re: [PR] AWS: Add AWS integ tests to check task and enable tests based on required environment variables [iceberg]

2025-04-04 Thread via GitHub
nastra commented on code in PR #12671: URL: https://github.com/apache/iceberg/pull/12671#discussion_r2026383513 ## aws/src/integration/java/org/apache/iceberg/aws/EnableAwsTest.java: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] CORE: Allow HTTPClient to parse headers from properties. [iceberg]

2025-04-04 Thread via GitHub
nastra commented on code in PR #12595: URL: https://github.com/apache/iceberg/pull/12595#discussion_r2021094167 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -124,6 +125,8 @@ public void createCatalog() throws Exception { HTTPHeaders.of(Map.o

Re: [I] Validation Exception: Found new conflicting delete files that can apply to records matching true [iceberg]

2025-04-04 Thread via GitHub
Agney19 commented on issue #11795: URL: https://github.com/apache/iceberg/issues/11795#issuecomment-2772018211 @Jasonlaradsouza I'm facing the same issue. Have you found the solution? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] WIP: Interface based DataFile reader and writer API [iceberg]

2025-04-04 Thread via GitHub
snazy commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r2003301101 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkObjectModels.java: ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[PR] AWS: Refactor DynamoDB and Glue properties into separated properties classes [iceberg]

2025-04-04 Thread via GitHub
lliangyu-lin opened a new pull request, #12722: URL: https://github.com/apache/iceberg/pull/12722 ### Description * Add deprecation notices on DynamoDB and Glue properties and methods in `AwsProperties.java` * Created `DynamoDbProperties.java` and `GlueProperties.java` and unit tests

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-04 Thread via GitHub
deniskuzZ commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2028440651 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -135,20 +142,114 @@ public static PartitionStatisticsFile computeAndWriteStatsFile(

Re: [PR] Build: Bump guava from 33.4.0-jre to 33.4.6-jre [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12686: URL: https://github.com/apache/iceberg/pull/12686#issuecomment-2768820274 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] AWS: Support StaticCredentialsProvider in DefaultAwsClientFactory [iceberg]

2025-04-04 Thread via GitHub
morozov commented on PR #12695: URL: https://github.com/apache/iceberg/pull/12695#issuecomment-2773810216 > @xiaoxuandev Do you have a specific use case for this? In general, I think this is bad practice since you ideally wouldn't be configuring credentials explicitly. Is there a reason the

Re: [PR] spec: Variant lower/upper bounds [iceberg]

2025-04-04 Thread via GitHub
aihuaxu commented on PR #12658: URL: https://github.com/apache/iceberg/pull/12658#issuecomment-2779790932 > @aihuaxu and @rdblue is there a reason we need to explicitly restrict the lower/upper bounds to shredded fields? I would think that the stats pruning would be useful for any field tha

Re: [PR] AWS: Update the aws-bundle with latest dependencies [iceberg]

2025-04-04 Thread via GitHub
SanjayMarreddi commented on code in PR #12553: URL: https://github.com/apache/iceberg/pull/12553#discussion_r2021575246 ## aws-bundle/build.gradle: ## @@ -25,8 +25,11 @@ project(":iceberg-aws-bundle") { dependencies { implementation platform(libs.awssdk.bom) +imple

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2022289856 ## format/spec.md: ## @@ -1414,12 +1414,16 @@ Each partition field in `fields` is stored as a JSON object with the following p | V1 | V2 | V3 | Fi

Re: [PR] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

2025-04-04 Thread via GitHub
wypoon commented on PR #12449: URL: https://github.com/apache/iceberg/pull/12449#issuecomment-2752763998 @rdblue can you review this? I have some stale PRs implementing Parquet page skipping and at that time I came across the fact that `ParquetFilters.ConvertFilterToParquet` does not con

Re: [PR] Spark: when doing rewrite_data_files, check for partitioning schema compatibility [iceberg]

2025-04-04 Thread via GitHub
danielcweeks commented on code in PR #12651: URL: https://github.com/apache/iceberg/pull/12651#discussion_r2025570136 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -265,6 +265,22 @@ public boolean equals(Object other) { return Arrays.equals(fields, that.

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-04 Thread via GitHub
nastra commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2027126876 ## core/src/main/java/org/apache/iceberg/MetadataUpdate.java: ## @@ -328,6 +328,11 @@ public void applyTo(TableMetadata.Builder metadataBuilder) { } } + /*

Re: [PR] Core: Add MetricsReporter for SnapshotManager [iceberg]

2025-04-04 Thread via GitHub
nastra commented on code in PR #12665: URL: https://github.com/apache/iceberg/pull/12665#discussion_r2022576907 ## core/src/main/java/org/apache/iceberg/BaseTable.java: ## @@ -232,7 +232,7 @@ public ExpireSnapshots expireSnapshots() { @Override public ManageSnapshots man

[PR] allow string transforms in add_field() [iceberg-python]

2025-04-04 Thread via GitHub
Federico-Vichachi opened a new pull request, #1883: URL: https://github.com/apache/iceberg-python/pull/1883 Closes #1011 # Rationale for this change This change allows the `add_field` method to accept `str` values as transform inputs, in addition to `Transform` instances. This impr

Re: [PR] Spark 3.5: Support case sensitive in replace where statement [iceberg]

2025-04-04 Thread via GitHub
dolcino-li commented on code in PR #12706: URL: https://github.com/apache/iceberg/pull/12706#discussion_r2026583674 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -351,6 +353,7 @@ private OverwriteByFilter(Expression overwriteExpr) {

Re: [PR] Core: Use InputFile.location() Instead of Direct Object Reference in Error Messages [iceberg]

2025-04-04 Thread via GitHub
Jordano-Dremio commented on code in PR #12716: URL: https://github.com/apache/iceberg/pull/12716#discussion_r2029404043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -175,7 +175,8 @@ public static ViewMetadata read(InputFile file) { codec

[PR] refactor: Split `manifest` module into multiple modules [iceberg-rust]

2025-04-04 Thread via GitHub
jonathanc-n opened a new pull request, #1119: URL: https://github.com/apache/iceberg-rust/pull/1119 ## Which issue does this PR close? - Closes #1083. ## What changes are included in this PR? Split manifest modules -- This is an automated message from the Apache Gi

Re: [PR] Doc: Add doc for flink exec config [iceberg]

2025-04-04 Thread via GitHub
pvary commented on code in PR #12691: URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026603875 ## docs/docs/flink-configuration.md: ## @@ -198,4 +198,42 @@ they are. This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For {@link

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-04 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023478884 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] + az

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029297526 ## mkdocs/docs/expression-dsl.md: ## @@ -0,0 +1,244 @@ +# Expression DSL Review Comment: ```suggestion # Expression DSL ``` ## mkdocs

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-04 Thread via GitHub
wypoon commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023308361 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception {

Re: [PR] Parquet: Add variant array reader in Parquet [iceberg]

2025-04-04 Thread via GitHub
rdblue commented on code in PR #12512: URL: https://github.com/apache/iceberg/pull/12512#discussion_r2006374595 ## parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java: ## @@ -900,6 +1176,31 @@ private static GenericRecord record(GroupType type, Map fields)

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-04-04 Thread via GitHub
xiaoxuandev commented on PR #12671: URL: https://github.com/apache/iceberg/pull/12671#issuecomment-2771411171 > @xiaoxuandev @nastra There are some concerns on moving the tests to moto since we need to add additional configs to make it work correctly. So I revert those changes in this PR an

Re: [PR] Core: Enable row lineage for all v3 tables [iceberg]

2025-04-04 Thread via GitHub
RussellSpitzer commented on code in PR #12593: URL: https://github.com/apache/iceberg/pull/12593#discussion_r2010570195 ## core/src/test/java/org/apache/iceberg/TestRowLineageMetadata.java: ## @@ -1,334 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one -

Re: [I] Replace parquet metadata thrift version with in memory version. [iceberg-rust]

2025-04-04 Thread via GitHub
liurenjie1024 closed issue #1004: Replace parquet metadata thrift version with in memory version. URL: https://github.com/apache/iceberg-rust/issues/1004 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029322605 ## mkdocs/docs/row-filter-syntax.md: ## @@ -0,0 +1,171 @@ + + +# Row Filter Syntax + +In addtion to the primary [Expression DSL](expression-dsl.md), PyIceberg pro

Re: [I] Iceberg Kafka Connector experiences a constant hanging lag for low-volume topics [iceberg]

2025-04-04 Thread via GitHub
pjadav commented on issue #11818: URL: https://github.com/apache/iceberg/issues/11818#issuecomment-2779611046 Hey @fenil25 We are also facing this issue. We can see data is landing but connector is not able to write updated metadata to make data visible. Although its not perfect solutio

Re: [I] Does main branch reference reset requiring a clean up of snapshot logs [iceberg]

2025-04-04 Thread via GitHub
github-actions[bot] closed issue #11109: Does main branch reference reset requiring a clean up of snapshot logs URL: https://github.com/apache/iceberg/issues/11109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Core, Parquet, ORC: Fix missing data when writing unknown [iceberg]

2025-04-04 Thread via GitHub
rdblue commented on PR #12581: URL: https://github.com/apache/iceberg/pull/12581#issuecomment-2741736285 Thanks for the reviews, @pvary and @danielcweeks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing [iceberg-rust]

2025-04-04 Thread via GitHub
sdd commented on code in PR #982: URL: https://github.com/apache/iceberg-rust/pull/982#discussion_r2025403948 ## crates/iceberg/src/arrow/delete_file_manager.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] decimal physicial type mapping [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1839: URL: https://github.com/apache/iceberg-python/pull/1839#discussion_r2020991374 ## pyiceberg/io/pyarrow.py: ## @@ -2350,8 +2351,19 @@ def data_file_statistics_from_parquet_metadata( stats_col.iceberg_type, statisti

Re: [PR] Hive: Support custom HMSClient [iceberg]

2025-04-04 Thread via GitHub
danielcweeks commented on code in PR #12712: URL: https://github.com/apache/iceberg/pull/12712#discussion_r2027306627 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -80,6 +80,8 @@ public class HiveCatalog extends BaseMetastoreViewCatalog publi

Re: [I] Iceberg Kafka Connect Sink has AVRO 1.12.0, but Kafka Connect AVRO Converter 7.9.0 wants 1.11.4 and therefore Avro Converter does not work [iceberg]

2025-04-04 Thread via GitHub
lk-1984 commented on issue #12571: URL: https://github.com/apache/iceberg/issues/12571#issuecomment-2742481491 This happens as there are two different versions of Avro in two of the JARs.. I removed the 1.12.0 one, and it works. -- This is an automated message from the Apache Git Service.

Re: [PR] feat: check whether table ops conflict when committing [iceberg-python]

2025-04-04 Thread via GitHub
YinZheng-Sun closed pull request #1816: feat: check whether table ops conflict when committing URL: https://github.com/apache/iceberg-python/pull/1816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] refactor(table): shift parquet-specific details to internal package [iceberg-go]

2025-04-04 Thread via GitHub
kevinjqliu commented on code in PR #353: URL: https://github.com/apache/iceberg-go/pull/353#discussion_r2019196486 ## table/snapshot_producers.go: ## @@ -349,38 +347,3 @@ func (sp *snapshotProducer) commit() ([]Update, []Requirement, error) { AssertRefSn

[PR] Spark 3.4: Test metadata tables with format-version=v3 / add ExtensionsTestBase [iceberg]

2025-04-04 Thread via GitHub
nastra opened a new pull request, #12600: URL: https://github.com/apache/iceberg/pull/12600 this backports #12135 to Spark 3.4 and also includes changes around switching from JUnit4 to JUnit5 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] feat: add file_io and local fs impl [iceberg-cpp]

2025-04-04 Thread via GitHub
lidavidm commented on PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#issuecomment-2750178633 > Then the only remaining I/O we need to address is the table metadata file which stores a single json string. This is the only reason to keep the `InputFile` and `OutputFile` interfaces

[PR] build(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 [iceberg-go]

2025-04-04 Thread via GitHub
dependabot[bot] opened a new pull request, #374: URL: https://github.com/apache/iceberg-go/pull/374 Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2. Release notes Sourced from https://github.com/golang-jwt/jwt/releases";>github.com/golang

Re: [PR] Core: HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs [iceberg]

2025-04-04 Thread via GitHub
steveloughran commented on PR #10233: URL: https://github.com/apache/iceberg/pull/10233#issuecomment-2767307590 yeah, classpaths are mess in places, primarily due to downstream things * spark pulling in 3.3.6. These need to be excluded and then the new versions pulled in. * hive-metast

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-04 Thread via GitHub
amogh-jahagirdar merged PR #12670: URL: https://github.com/apache/iceberg/pull/12670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] fix: chore cargo lock and fix two warning for python bindings [iceberg-rust]

2025-04-04 Thread via GitHub
yihong0618 opened a new pull request, #1121: URL: https://github.com/apache/iceberg-rust/pull/1121 ## Which issue does this PR close? This patch chore the cargo.lock and fix the warning for python binding -- This is an automated message from the Apache Git Service. To respon

Re: [I] Generation of Snapshot Summaries [iceberg-rust]

2025-04-04 Thread via GitHub
liurenjie1024 closed issue #724: Generation of Snapshot Summaries URL: https://github.com/apache/iceberg-rust/issues/724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Exception on Subsequent Writes When Using Nessie or REST Catalog in Iceberg [iceberg]

2025-04-04 Thread via GitHub
mohsenify commented on issue #12652: URL: https://github.com/apache/iceberg/issues/12652#issuecomment-2764581395 @nastra Of course, here’s the writer code I forgot to send earlier: ``` private DataFile writeRecordsToFile(String filePath, List records, Table table) { t

Re: [I] catalog table-default and table-override properties are not supported in CREATE_OR_REPLACE operation in IRC [iceberg]

2025-04-04 Thread via GitHub
nastra commented on issue #12506: URL: https://github.com/apache/iceberg/issues/12506#issuecomment-2736040633 I'll go ahead and close this issue. Please re-open in case we missed something -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Updated vendors documentation to add SingleStore [iceberg]

2025-04-04 Thread via GitHub
danielcweeks commented on code in PR #12708: URL: https://github.com/apache/iceberg/pull/12708#discussion_r2025520638 ## site/docs/vendors.md: ## @@ -90,6 +90,12 @@ PuppyGraph is a cloud-native graph analytics engine that enables users to query [RisingWave](https://risingwav

Re: [PR] Spark 3.5: Support case sensitive in replace where statement [iceberg]

2025-04-04 Thread via GitHub
nastra commented on code in PR #12706: URL: https://github.com/apache/iceberg/pull/12706#discussion_r2024930208 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestReplaceWhere.java: ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2027096293 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -415,9 +416,9 @@ public void testAddSnapshotFromJson() throws IOException {

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-04 Thread via GitHub
zratkai commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2020810406 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1060,19 +1060,19 @@ public void testNotExposeTableProperties() { parameters.

Re: [I] Cannot scan empty table [iceberg-rust]

2025-04-04 Thread via GitHub
danking commented on issue #1145: URL: https://github.com/apache/iceberg-rust/issues/1145#issuecomment-2779062466 I created a PR that fixes this issue for me: https://github.com/apache/iceberg-rust/pull/1166 -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] allow dashes in glue database and table names [iceberg]

2025-04-04 Thread via GitHub
jrouly commented on PR #12677: URL: https://github.com/apache/iceberg/pull/12677#issuecomment-2767625269 @xiaoxuandev the issue, in my mind at least, is less that glue validation is on or off by default and more that it is validating incorrectly. Legal glue table and database names may incl

Re: [PR] Data: Refactor PartitionStatsHandler [iceberg]

2025-04-04 Thread via GitHub
pvary commented on PR #12550: URL: https://github.com/apache/iceberg/pull/12550#issuecomment-2736042315 Let's wait a bit for @aokolnychyi, and if he has not time to review, then we can merge this. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Core: lazy init workerPool [iceberg]

2025-04-04 Thread via GitHub
abstractdog commented on code in PR #12427: URL: https://github.com/apache/iceberg/pull/12427#discussion_r2026704750 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1772,6 +1775,24 @@ public void testNoSchemasOrSpecsToRemove() {

Re: [I] Move docker-specific tests to integrationTest configuration [iceberg]

2025-04-04 Thread via GitHub
kevinjqliu commented on issue #12236: URL: https://github.com/apache/iceberg/issues/12236#issuecomment-2779570365 Thanks all! I was able to verify locally that `./gradlew test --continue` without docker did not fail any docker related tests -- This is an automated message from the Apache

Re: [PR] Support quoted column identifiers for scan `row_filter` string argument [iceberg-python]

2025-04-04 Thread via GitHub
Fokko merged PR #1863: URL: https://github.com/apache/iceberg-python/pull/1863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029321835 ## mkdocs/docs/expression-dsl.md: ## @@ -0,0 +1,261 @@ + + +# Expression DSL + +The PyIceberg library provides a powerful expression DSL (Domain Specific Languag

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029321432 ## mkdocs/docs/expression-dsl.md: ## @@ -0,0 +1,261 @@ + + +# Expression DSL + +The PyIceberg library provides a powerful expression DSL (Domain Specific Languag

Re: [I] Add V3 types `timestamp_ns` and `timestamptz_ns` [iceberg-python]

2025-04-04 Thread via GitHub
Fokko closed issue #1552: Add V3 types `timestamp_ns` and `timestamptz_ns` URL: https://github.com/apache/iceberg-python/issues/1552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Spec: Allow the use of `source-id` in V3 [iceberg]

2025-04-04 Thread via GitHub
rdblue commented on code in PR #12644: URL: https://github.com/apache/iceberg/pull/12644#discussion_r2027416683 ## format/spec.md: ## @@ -1605,13 +1611,8 @@ All readers are required to read tables with unknown partition transforms, ignor Writing v3 metadata: * Partition Fie

Re: [PR] Add pure virtual classes for Catalog, Table, etc. [iceberg-cpp]

2025-04-04 Thread via GitHub
wgtmac commented on code in PR #47: URL: https://github.com/apache/iceberg-cpp/pull/47#discussion_r2007170941 ## src/iceberg/table_identifier.h: ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. Se

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029319804 ## mkdocs/docs/expression-dsl.md: ## @@ -0,0 +1,261 @@ + + +# Expression DSL + +The PyIceberg library provides a powerful expression DSL (Domain Specific Languag

Re: [PR] Documented `row_filter` expressions [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on code in PR #1862: URL: https://github.com/apache/iceberg-python/pull/1862#discussion_r2029319471 ## mkdocs/docs/expression-dsl.md: ## @@ -0,0 +1,261 @@ + + +# Expression DSL + +The PyIceberg library provides a powerful expression DSL (Domain Specific Languag

Re: [PR] Core: ability to inject an AuthManager in RESTCatalog [iceberg]

2025-04-04 Thread via GitHub
varpa89 commented on PR #12655: URL: https://github.com/apache/iceberg/pull/12655#issuecomment-2765646665 > It's not a custom vs built-in problem, it's a shared vs non-shared problem. > > Consider this hypothetical example: let's suppose we have a custom manager that authenticates usi

Re: [PR] Build: Bump com.azure:azure-sdk-bom from 1.2.31 to 1.2.33 [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12689: URL: https://github.com/apache/iceberg/pull/12689#issuecomment-2772054182 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-04 Thread via GitHub
pvary commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2007579325 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -299,7 +299,10 @@ private void setHmsTableParameters( BaseMetastoreTab

Re: [PR] Docs: Site updates for 1.7.2 [iceberg]

2025-04-04 Thread via GitHub
ajantha-bhat commented on code in PR #12576: URL: https://github.com/apache/iceberg/pull/12576#discussion_r2003558804 ## site/docs/releases.md: ## @@ -88,6 +88,20 @@ The 1.8.1 release contains bug fixes and fixes to LICENSE/NOTICE files. For full ## Past releases +### 1.7.

Re: [PR] Support Snowflake-Managed Iceberg Tables via SnowflakeCatalog [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on PR #1834: URL: https://github.com/apache/iceberg-python/pull/1834#issuecomment-2779545731 @srilman From what I understand from Snowflake, it is also transitioning to the Rest Catalog protocol. I much rather use that since it is properly tested. As you mentioned, there are

Re: [PR] chore: update denpendencies, fix ci [iceberg-rust]

2025-04-04 Thread via GitHub
Xuanwo commented on code in PR #1106: URL: https://github.com/apache/iceberg-rust/pull/1106#discussion_r2002760992 ## rust-toolchain.toml: ## @@ -16,5 +16,5 @@ # under the License. [toolchain] -channel = "nightly-2024-06-10" +channel = "nightly-2024-10-17" Review Comment:

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-04 Thread via GitHub
zhjwpku commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2022215514 ## test/CMakeLists.txt: ## @@ -42,7 +42,13 @@ if(ICEBERG_BUILD_BUNDLE) add_executable(arrow_test) target_sources(arrow_test PRIVATE arrow_test.cc) - target_lin

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-04-04 Thread via GitHub
Kontinuation commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r2004561757 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +565,134 @@ public int hashCode() { } } + public static class GeometryType exte

Re: [PR] Add support for `Transaction.update_statistics()` [iceberg-python]

2025-04-04 Thread via GitHub
Fokko merged PR #1831: URL: https://github.com/apache/iceberg-python/pull/1831 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Add support for `Transaction.update_statistics()` [iceberg-python]

2025-04-04 Thread via GitHub
Fokko commented on PR #1831: URL: https://github.com/apache/iceberg-python/pull/1831#issuecomment-2779543118 Thanks @srilman 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-04 Thread via GitHub
jonathanc-n commented on code in PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023638265 ## crates/iceberg/src/transform/bucket.rs: ## @@ -167,6 +167,16 @@ impl TransformFunction for Bucket { .downcast_ref::() .unw

Re: [PR] Add pure virtual classes for Catalog, Table, etc. [iceberg-cpp]

2025-04-04 Thread via GitHub
lidavidm commented on code in PR #47: URL: https://github.com/apache/iceberg-cpp/pull/47#discussion_r2007100719 ## src/iceberg/table_identifier.h: ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements.

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-04-04 Thread via GitHub
gaborkaszab commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r2005892800 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -230,12 +220,12 @@ protected void doCommit(TableMetadata base, TableMetada

Re: [PR] Core: Make TestBase abstract [iceberg]

2025-04-04 Thread via GitHub
nastra commented on PR #12678: URL: https://github.com/apache/iceberg/pull/12678#issuecomment-2765622362 @sullis what's the benefit we're gaining by making it abstract? So far I don't see any use cases that would require making it abstract -- This is an automated message from the Apache G

[PR] docs: update go impl status [iceberg]

2025-04-04 Thread via GitHub
zeroshade opened a new pull request, #12578: URL: https://github.com/apache/iceberg/pull/12578 Updating the implementation status for Go implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-04-04 Thread via GitHub
szehon-ho commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r2003918915 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +565,134 @@ public int hashCode() { } } + public static class GeometryType extends

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-04 Thread via GitHub
gaborkaszab commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2027152279 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -135,20 +143,113 @@ public static PartitionStatisticsFile computeAndWriteStatsFil

  1   2   >