Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

nastra commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1814315018 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java: ## @@ -64,7 +72,8 @@ public Map configuration() { private CatalogContext initial

Re: [PR] Flink: Add RowConverter for Iceberg Source [iceberg]

abharath9 commented on code in PR #11301: URL: https://github.com/apache/iceberg/pull/11301#discussion_r1814251209 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/reader/RowConverter.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Data: Add partition stats writer and reader [iceberg]

aokolnychyi commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2434273228 I think we should try to use "internal" writers. @rdblue added "internal" readers recently. Any guidance on how to add a writer, @rdblue? We can start with Avro for now. We w

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on PR #11021: URL: https://github.com/apache/iceberg/pull/11021#issuecomment-2434187344 > AccessPoints being not supported Yes, today users configure features like cross bucket replication also through the access point configuration. The access point naming is no

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 merged PR #11021: URL: https://github.com/apache/iceberg/pull/11021 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Fix when reading struct-type data without an id in iceberg-parquet [iceberg]

joyCurry30 commented on PR #11378: URL: https://github.com/apache/iceberg/pull/11378#issuecomment-2434180831 @nastra Can you help review this PR? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] More accurate estimate on parquet row groups size [iceberg]

jinyangli34 commented on code in PR #11258: URL: https://github.com/apache/iceberg/pull/11258#discussion_r1813328284 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java: ## @@ -66,6 +66,9 @@ class ParquetWriter implements FileAppender, Closeable { private b

Re: [PR] Aliyun: Remove spring-boot dependency [iceberg]

manuzhang commented on code in PR #11291: URL: https://github.com/apache/iceberg/pull/11291#discussion_r1814144426 ## aliyun/src/test/java/org/apache/iceberg/aliyun/TestUtility.java: ## @@ -65,7 +65,7 @@ public static AliyunOSSExtension initialize() { } else { LOG.in

Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]

1316147945 commented on issue #9488: URL: https://github.com/apache/iceberg/issues/9488#issuecomment-2434044363 COALESCE(column,0) maybe useful, and SET `spark.sql.iceberg.check-nullability`=`false` + COALESCE(column,0) can solve problem -- This is an automated message from the Apache

Re: [PR] Spec: Adds Row Lineage [iceberg]

flyrain commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1814055846 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Spark: Hive-View Integration Tests [iceberg]

github-actions[bot] commented on PR #10088: URL: https://github.com/apache/iceberg/pull/10088#issuecomment-2433857720 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Add capability to support an external index for additional file filtering [iceberg]

github-actions[bot] commented on issue #10083: URL: https://github.com/apache/iceberg/issues/10083#issuecomment-2433857679 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] SparkSessionCatalog.listTables only returns with the hive tables but not iceberg [iceberg]

github-actions[bot] commented on issue #10065: URL: https://github.com/apache/iceberg/issues/10065#issuecomment-2433857645 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

github-actions[bot] commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-2433857259 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1

github-actions[bot] commented on issue #10056: URL: https://github.com/apache/iceberg/issues/10056#issuecomment-2433857608 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Logs: More precise error message for Iceberg Partition Parsing [iceberg]

github-actions[bot] commented on PR #10047: URL: https://github.com/apache/iceberg/pull/10047#issuecomment-2433857553 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

github-actions[bot] closed pull request #9710: Add filter pushdown API for contains and endsWith URL: https://github.com/apache/iceberg/pull/9710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] iceberg does not allow selection of AWS profiles other than "default" [iceberg]

github-actions[bot] commented on issue #10085: URL: https://github.com/apache/iceberg/issues/10085#issuecomment-2433857700 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Aliyun: Add loading oss file into memory option to OSS client properties [iceberg]

github-actions[bot] commented on PR #10062: URL: https://github.com/apache/iceberg/pull/10062#issuecomment-2433857628 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Document `advisory-partition-size` configuration for Spark's Adaptive Query Execution [iceberg]

github-actions[bot] commented on issue #10051: URL: https://github.com/apache/iceberg/issues/10051#issuecomment-2433857588 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813980941 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, Manife

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813980941 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, Manife

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813980941 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, Manife

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

amogh-jahagirdar commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813980941 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, Manife

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813521623 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -297,7 +297,14 @@ private List deleteBatch(String bucket, Collection keysToDelete) @Overr

Re: [PR] ci: Fix CI for bindings python [iceberg-rust]

kevinjqliu commented on PR #678: URL: https://github.com/apache/iceberg-rust/pull/678#issuecomment-2433021197 FYI, CI failing on main https://github.com/apache/iceberg-rust/commits/main/ Is it related to this PR? error message, ``` error: use of deprecated method `open

Re: [PR] Flink Support for TIMESTAMP_NANOS [iceberg]

rodmeneses commented on code in PR #11348: URL: https://github.com/apache/iceberg/pull/11348#discussion_r1813237820 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/TypeToFlinkType.java: ## @@ -113,6 +113,15 @@ public LogicalType primitive(Type.PrimitiveType primitive

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813944812 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, ManifestFile man

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813956145 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -341,43 +367,44 @@ private ManifestFile filterManifest(Schema tableSchema, ManifestFile man

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813954048 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, ManifestFile man

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813944812 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, ManifestFile man

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813931853 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -323,11 +345,15 @@ private ManifestFile filterManifest(Schema tableSchema, ManifestFile man

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813919704 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -185,6 +200,13 @@ List filterManifests(Schema tableSchema, List manife return Immuta

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813917449 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -185,6 +200,13 @@ List filterManifests(Schema tableSchema, List manife return Immuta

Re: [PR] Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten [iceberg]

rdblue commented on code in PR #11131: URL: https://github.com/apache/iceberg/pull/11131#discussion_r1813909190 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -162,11 +178,13 @@ void delete(F file) { void delete(CharSequence path) { Preconditi

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813814216 ## format/spec.md: ## @@ -454,35 +457,40 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

[I] Iceberg Extensions [iceberg-go]

dwilson1988 opened a new issue, #183: URL: https://github.com/apache/iceberg-go/issues/183 ### Feature Request / Improvement There are some various table formats that extend Iceberg by allowing additional metadata to be added to various components, for example, [Havasu](https://githu

Re: [PR] feat: more builders and writing manifests [iceberg-go]

zeroshade commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813481928 ## manifest.go: ## @@ -831,14 +946,53 @@ func (m *manifestEntryV1) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV1) DataFile()

Re: [I] Row Lineage for V3 [iceberg]

rdblue closed issue #11129: Row Lineage for V3 URL: https://github.com/apache/iceberg/issues/11129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issue

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813136835 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1813670115 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue merged PR #11130: URL: https://github.com/apache/iceberg/pull/11130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on PR #11130: URL: https://github.com/apache/iceberg/pull/11130#issuecomment-2433543495 Merged! Thanks for the awesome work on this, @RussellSpitzer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1813684620 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1813673364 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1813660233 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Spec: Adds Row Lineage [iceberg]

rdblue commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1813684620 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is: -|

Re: [PR] Reset Spark Conf for each test in TestCompressionSettings [iceberg]

RussellSpitzer commented on PR #11333: URL: https://github.com/apache/iceberg/pull/11333#issuecomment-2433510526 Thank you @huaxingao ! It's always great to fix those hidden bad tests! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813656198 ## format/spec.md: ## @@ -841,19 +855,45 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813648755 ## format/spec.md: ## @@ -454,35 +457,40 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo `data_file` is a struct with the follo

Re: [PR] Spark 3.4: Action to remove dangling deletes [iceberg]

amogh-jahagirdar merged PR #11377: URL: https://github.com/apache/iceberg/pull/11377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r181364 ## format/spec.md: ## @@ -619,19 +627,25 @@ Data files that match the query filter must be read by the scan. Note that for any snapshot, all file paths marked with "

Re: [PR] Core: Add portable Roaring bitmap for row positions [iceberg]

aokolnychyi commented on code in PR #11372: URL: https://github.com/apache/iceberg/pull/11372#discussion_r1813326978 ## core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java: ## @@ -0,0 +1,309 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Build: Enable errorprone PatternMatchingInstanceof [iceberg]

ebyhr commented on PR #11374: URL: https://github.com/apache/iceberg/pull/11374#issuecomment-2433488950 @nastra This is mainly for enforcing styles. I will close if we don't use errorprone for such purposes. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813357582 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

haizhou-zhao commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1813548551 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java: ## @@ -64,7 +72,8 @@ public Map configuration() { private CatalogContext i

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813606048 ## manifest.go: ## @@ -831,14 +946,53 @@ func (m *manifestEntryV1) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV1) DataFile(

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813507988 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813592483 ## format/spec.md: ## @@ -841,19 +855,45 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813567209 ## format/spec.md: ## @@ -841,19 +855,45 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] Reset Spark Conf for each test in TestCompressionSettings [iceberg]

huaxingao commented on PR #11333: URL: https://github.com/apache/iceberg/pull/11333#issuecomment-2433467807 Thanks a lot @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813442346 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

rdblue commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1813586205 ## format/spec.md: ## @@ -841,19 +855,45 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level del

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813528711 ## manifest.go: ## @@ -831,14 +946,53 @@ func (m *manifestEntryV1) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV1) DataFile(

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

soumilshah1995 commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2433444077 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813521154 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813511239 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -297,7 +297,14 @@ private List deleteBatch(String bucket, Collection keysToDelete) @Overr

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

singhpk234 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813496517 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -297,7 +297,14 @@ private List deleteBatch(String bucket, Collection keysToDelete) @Overr

Re: [PR] feat: more builders and writing manifests [iceberg-go]

zeroshade commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813485988 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2 w

Re: [PR] feat: more builders and writing manifests [iceberg-go]

zeroshade commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813483677 ## manifest.go: ## @@ -876,7 +1030,140 @@ func (m *manifestEntryV2) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV2) DataFile()

Re: [PR] Flink: Maintenance - TableManager + ExpireSnapshots [iceberg]

stevenzwu commented on code in PR #11144: URL: https://github.com/apache/iceberg/pull/11144#discussion_r1813326096 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java: ## @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

wmoustafa commented on PR #11365: URL: https://github.com/apache/iceberg/pull/11365#issuecomment-2433322970 @rdblue @danielcweeks @stevenzwu @RussellSpitzer @bennychow Would be great to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813045425 ## aws/src/integration/java/org/apache/iceberg/aws/AwsIntegTestUtil.java: ## @@ -127,6 +129,47 @@ public static void cleanS3Bucket(S3Client s3, String bucketName,

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813467513 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813443609 ## manifest.go: ## @@ -876,7 +1030,140 @@ func (m *manifestEntryV2) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV2) DataFile

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813445471 ## manifest.go: ## @@ -876,7 +1030,140 @@ func (m *manifestEntryV2) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV2) DataFile

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813446168 ## manifest.go: ## @@ -876,7 +1030,140 @@ func (m *manifestEntryV2) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV2) DataFile

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813442346 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813453372 ## manifest.go: ## @@ -567,6 +569,96 @@ func ReadManifestList(in io.Reader) ([]ManifestFile, error) { return out, dec.Error() } +// WriteManifestListV2

[PR] Spark 3.4: Randomize view/function names in testing [iceberg]

nastra opened a new pull request, #11382: URL: https://github.com/apache/iceberg/pull/11382 this backports #11381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] feat: more builders and writing manifests [iceberg-go]

dwilson1988 commented on code in PR #177: URL: https://github.com/apache/iceberg-go/pull/177#discussion_r1813441342 ## manifest.go: ## @@ -831,14 +946,53 @@ func (m *manifestEntryV1) FileSequenceNum() *int64 { return m.FileSeqNum } -func (m *manifestEntryV1) DataFile(

Re: [PR] Reset Spark Conf for each test in TestCompressionSettings [iceberg]

RussellSpitzer merged PR #11333: URL: https://github.com/apache/iceberg/pull/11333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] ci: Fix CI for bindings python [iceberg-rust]

Xuanwo commented on PR #678: URL: https://github.com/apache/iceberg-rust/pull/678#issuecomment-2433240259 > FYI, CI failing on main > > https://github.com/apache/iceberg-rust/commits/main/ > > > > Is it related to this PR? > > > > error message, >

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813357582 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813357582 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

[PR] (AWS) Docs: List all AWS S3 properties from all language impl. #10674 [iceberg]

Neuw84 opened a new pull request, #11383: URL: https://github.com/apache/iceberg/pull/11383 As @hsiang-c made another pull request building a table [here](https://github.com/apache/iceberg/pull/11321) I didn't want to collide. Therefore, I added: * Added Amazon MSK Connect a

Re: [PR] Core: Add portable Roaring bitmap for row positions [iceberg]

aokolnychyi commented on code in PR #11372: URL: https://github.com/apache/iceberg/pull/11372#discussion_r1813327452 ## core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java: ## @@ -0,0 +1,309 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core: Add portable Roaring bitmap for row positions [iceberg]

aokolnychyi commented on code in PR #11372: URL: https://github.com/apache/iceberg/pull/11372#discussion_r1813325394 ## core/src/test/java/org/apache/iceberg/deletes/TestRoaringPositionBitmap.java: ## @@ -0,0 +1,516 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Core: Add portable Roaring bitmap for row positions [iceberg]

aokolnychyi commented on code in PR #11372: URL: https://github.com/apache/iceberg/pull/11372#discussion_r1813322036 ## core/src/test/java/org/apache/iceberg/deletes/TestRoaringPositionBitmap.java: ## @@ -0,0 +1,323 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Core: Add portable Roaring bitmap for row positions [iceberg]

aokolnychyi commented on code in PR #11372: URL: https://github.com/apache/iceberg/pull/11372#discussion_r1813314555 ## core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java: ## @@ -0,0 +1,317 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813052151 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3URI.java: ## @@ -37,6 +37,9 @@ class S3URI { private static final String QUERY_DELIM = "\\?"; private static

Re: [PR] Spark 3.4: Randomize view/function names in testing [iceberg]

amogh-jahagirdar merged PR #11382: URL: https://github.com/apache/iceberg/pull/11382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on PR #11021: URL: https://github.com/apache/iceberg/pull/11021#issuecomment-2432810694 This mostly looks good to me now, just a few very nit comments. And I think we should update the `aws.md` about using directory buckets. But that can also be a separated PR, up to yo

Re: [PR] Aliyun: Remove spring-boot dependency [iceberg]

jbonofre commented on PR #11291: URL: https://github.com/apache/iceberg/pull/11291#issuecomment-2432883972 @findepi @manuzhang I fixed the spotless. Sorry for the inconvenience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813154671 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

nastra commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813150009 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] AWS: Use testcontainers-minio instead of S3Mock [iceberg]

sullis commented on code in PR #11349: URL: https://github.com/apache/iceberg/pull/11349#discussion_r1813146387 ## aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813140179 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -428,6 +428,21 @@ public class S3FileIOProperties implements Serializable { publ

Re: [PR] Data: Add partition stats writer and reader [iceberg]

ajantha-bhat commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2432771734 @RussellSpitzer: It would be good to have this in 1.7.0. I am waiting from a month for a review. -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] AWS: Support S3 directory bucket listing [iceberg]

jackye1995 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1813117694 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3URI.java: ## @@ -115,4 +118,25 @@ public String scheme() { public String toString() { return location;

  1   2   >