Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat closed pull request #10183: Infra: Track subtasks from Iceberg improvement proposal URL: https://github.com/apache/iceberg/pull/10183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Add a util to read and write partition stats [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat commented on PR #9170: URL: https://github.com/apache/iceberg/pull/9170#issuecomment-2259805653 closing this in favour of https://github.com/apache/iceberg/pull/10176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Core: Add a util to read and write partition stats [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat closed pull request #9170: Core: Add a util to read and write partition stats URL: https://github.com/apache/iceberg/pull/9170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat closed pull request #9437: Spark 3.5: Spark action to compute the partition stats URL: https://github.com/apache/iceberg/pull/9437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat commented on PR #9437: URL: https://github.com/apache/iceberg/pull/9437#issuecomment-2259804075 closing this PR as we decided to move with local algorithm instead of distributive algorithm. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697992679 ## data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [I] Feature: S3 Remote Signing [iceberg-rust]

2024-07-30 Thread via GitHub
Xuanwo commented on issue #506: URL: https://github.com/apache/iceberg-rust/issues/506#issuecomment-2259778445 Of course, welcome! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Feature: S3 Remote Signing [iceberg-rust]

2024-07-30 Thread via GitHub
flaneur2020 commented on issue #506: URL: https://github.com/apache/iceberg-rust/issues/506#issuecomment-2259758262 can i take this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Treat warning as error in CI/Dev [iceberg-python]

2024-07-30 Thread via GitHub
Fokko commented on PR #973: URL: https://github.com/apache/iceberg-python/pull/973#issuecomment-2259753059 @ndrluis Ohh, this is nice. And sorry, it looks like I disabled this out a while ago https://github.com/apache/iceberg-python/commit/0cbb71cb86c1952d40508681ea1ff96f009d5d79 ๐Ÿ™ˆ --

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

2024-07-30 Thread via GitHub
nk1506 commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1697954569 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -48,28 +40,6 @@ public void testNullArguments() { .hasMessage("Invalid JSON string

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

2024-07-30 Thread via GitHub
nk1506 commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1697953557 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -48,28 +40,6 @@ public void testNullArguments() { .hasMessage("Invalid JSON string

Re: [I] Issue when rewriting an unpartitioned table [iceberg-python]

2024-07-30 Thread via GitHub
Fokko commented on issue #979: URL: https://github.com/apache/iceberg-python/issues/979#issuecomment-2259703854 > It looks like this is an issue with tables with any partition evolution, and not specific to an unpartitioned table You're right there :) It only happens when you add a fi

Re: [PR] DOCS: Add more post release notes [iceberg-python]

2024-07-30 Thread via GitHub
Fokko commented on code in PR #983: URL: https://github.com/apache/iceberg-python/pull/983#discussion_r1697919463 ## mkdocs/docs/how-to-release.md: ## @@ -208,6 +208,15 @@ svn add /tmp/iceberg-dist-release/ svn ci -m "PyIceberg " /tmp/iceberg-dist-release/ ``` + + +!!! note

Re: [PR] Bug Fix: use appropriate partition spec for delete [iceberg-python]

2024-07-30 Thread via GitHub
Fokko merged PR #984: URL: https://github.com/apache/iceberg-python/pull/984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Issue when rewriting an unpartitioned table [iceberg-python]

2024-07-30 Thread via GitHub
Fokko closed issue #979: Issue when rewriting an unpartitioned table URL: https://github.com/apache/iceberg-python/issues/979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Manifest list encryption [iceberg]

2024-07-30 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1697877097 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -257,10 +273,48 @@ public Snapshot apply() { .run(index -> manifestFiles[index] =

Re: [PR] Manifest list encryption [iceberg]

2024-07-30 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1697876636 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -237,10 +244,19 @@ public Snapshot apply() { OutputFile manifestList = manifestListPath

Re: [PR] Manifest list encryption [iceberg]

2024-07-30 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1697872976 ## core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java: ## @@ -92,13 +94,45 @@ public ByteBuffer wrapKey(ByteBuffer secretKey) { pu

Re: [PR] Manifest list encryption [iceberg]

2024-07-30 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1697872635 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -147,6 +169,42 @@ static Snapshot fromJson(JsonNode node) { if (node.has(MANIFEST_LIST)) {

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
lirui-apache commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697840726 ## data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-30 Thread via GitHub
jackye1995 commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1697835615 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an iss

Re: [PR] Treat warning as error in CI/Dev [iceberg-python]

2024-07-30 Thread via GitHub
sungwy commented on PR #973: URL: https://github.com/apache/iceberg-python/pull/973#issuecomment-2259529801 Thank you! Running the CI now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Treat warning as error in CI/Dev [iceberg-python]

2024-07-30 Thread via GitHub
ndrluis commented on PR #973: URL: https://github.com/apache/iceberg-python/pull/973#issuecomment-2259525851 @sungwy Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-30 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1697812946 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1446,6 +1480,12 @@ public TableMetadata build() { "Cannot set metadata location with

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-30 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1697811542 ## api/src/main/java/org/apache/iceberg/RemoveUnusedSpecs.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-30 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1697807013 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1425,6 +1460,7 @@ private boolean hasChanges() { || (discardChanges && !changes.isEm

Re: [I] Configure root path in Catalog or FileIO? [iceberg-rust]

2024-07-30 Thread via GitHub
fqaiser94 commented on issue #488: URL: https://github.com/apache/iceberg-rust/issues/488#issuecomment-2259494486 Accidentally closed the previous PR (and discovered you can't reopen closed PRs after a while ๐Ÿคฏ). New PR here: https://github.com/apache/iceberg-rust/pull/511 -- This

Re: [I] Feat: Podman support for integration tests [iceberg-rust]

2024-07-30 Thread via GitHub
liurenjie1024 closed issue #474: Feat: Podman support for integration tests URL: https://github.com/apache/iceberg-rust/issues/474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] feat: podman support [iceberg-rust]

2024-07-30 Thread via GitHub
liurenjie1024 merged PR #489: URL: https://github.com/apache/iceberg-rust/pull/489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] DOCS: Add more post release notes [iceberg-python]

2024-07-30 Thread via GitHub
sungwy commented on code in PR #983: URL: https://github.com/apache/iceberg-python/pull/983#discussion_r1697788769 ## mkdocs/docs/how-to-release.md: ## @@ -243,3 +252,13 @@ Make sure to create a PR to update the [GitHub issues template](https://github.c ## Update the integrati

[PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-07-30 Thread via GitHub
sdd opened a new pull request, #512: URL: https://github.com/apache/iceberg-rust/pull/512 This builds on top of the [concurrent scans PR ](https://github.com/apache/iceberg-rust/pull/373) and so needs to be merged after that. It caches parsed instances of `Manifest` and `ManifestList

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
aokolnychyi commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697760344 ## data/src/main/java/org/apache/iceberg/data/GeneratePartitionStats.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
aokolnychyi commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697730639 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
aokolnychyi commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697297242 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-07-30 Thread via GitHub
aokolnychyi commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1697282703 ## core/src/main/java/org/apache/iceberg/BaseScan.java: ## @@ -289,4 +289,22 @@ private static Schema lazyColumnProjection(TableScanContext context, Schema sche

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2259443025 PTAL @liurenjie1024 and @Xuanwo: I've addressed your helpful feedback and I think this is now ready. Performance tests of this on my performance testing branch show clear gains when scanni

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697765089 ## crates/iceberg/src/expr/predicate.rs: ## @@ -668,6 +668,10 @@ pub enum BoundPredicate { Set(SetExpression), } +/// Newtype to prevent accidentally confusing p

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697764804 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -79,6 +79,11 @@ impl ManifestList { pub fn entries(&self) -> &[ManifestFile] { &self.entries } + +

Re: [PR] DOCS: Add more post release notes [iceberg-python]

2024-07-30 Thread via GitHub
kevinjqliu commented on code in PR #983: URL: https://github.com/apache/iceberg-python/pull/983#discussion_r1697756436 ## mkdocs/docs/how-to-release.md: ## @@ -243,3 +252,13 @@ Make sure to create a PR to update the [GitHub issues template](https://github.c ## Update the integ

Re: [I] Issue when rewriting an unpartitioned table [iceberg-python]

2024-07-30 Thread via GitHub
sungwy commented on issue #979: URL: https://github.com/apache/iceberg-python/issues/979#issuecomment-2259421102 Hi @Fokko - I'm confirming that I can reproduce the issue with the following test: ``` def test_delete_after_partition_evolution_from_partitioned(session_catalog: RestC

Re: [PR] Docs: Use link addresses instead of descriptions in releases.md [iceberg]

2024-07-30 Thread via GitHub
lurnagao-dahua commented on PR #10815: URL: https://github.com/apache/iceberg/pull/10815#issuecomment-2259418041 Thanks for the review and merge @pvary ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP] Build - Ensure all CI Actions Exit Early if One Fails [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on PR #3524: URL: https://github.com/apache/iceberg/pull/3524#issuecomment-2259399192 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatโ€™s incorrect or this pull

Re: [I] In Flink, when we set the PK, the properties cannot be changed [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on issue #3521: URL: https://github.com/apache/iceberg/issues/3521#issuecomment-2259399152 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark: Pushdown Filters / Improve Performance when Importing File Based Tables [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on issue #3532: URL: https://github.com/apache/iceberg/issues/3532#issuecomment-2259399212 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Set the write order of the table to be case sensitive, use WRITE ORDERED BY? Should the default value be changed to false [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on issue #3523: URL: https://github.com/apache/iceberg/issues/3523#issuecomment-2259399175 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark session catalog fix [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] closed pull request #3125: Spark session catalog fix URL: https://github.com/apache/iceberg/pull/3125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Throw exception when uri is configured for hive catalog [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on PR #3126: URL: https://github.com/apache/iceberg/pull/3126#issuecomment-2259398855 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Throw exception when uri is configured for hive catalog [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] closed pull request #3126: Throw exception when uri is configured for hive catalog URL: https://github.com/apache/iceberg/pull/3126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Spark session catalog fix [iceberg]

2024-07-30 Thread via GitHub
github-actions[bot] commented on PR #3125: URL: https://github.com/apache/iceberg/pull/3125#issuecomment-2259398838 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Treat warning as error in CI/Dev [iceberg-python]

2024-07-30 Thread via GitHub
sungwy commented on PR #973: URL: https://github.com/apache/iceberg-python/pull/973#issuecomment-2259391639 @ndrluis shall we rebase this branch with the fix you issued in the other PR #972 ? ``` ERROR tests/integration/test_writes/test_writes.py::test_rest_catalog_with_empty_cata

Re: [I] Use Min, Max, and NumOfNulls from Manifest Files for Spark Column Stats [iceberg]

2024-07-30 Thread via GitHub
huaxingao commented on issue #10791: URL: https://github.com/apache/iceberg/issues/10791#issuecomment-2259391206 @jeesou It seems the manifest file doesn't have `avgLen` and `maxLen`, so I will not be able to have these two for now. Yes, I am thinking of using min, max, and numOfNulls fr

Re: [PR] Bump up Iceberg version to 1.6.0 [iceberg-python]

2024-07-30 Thread via GitHub
sungwy merged PR #982: URL: https://github.com/apache/iceberg-python/pull/982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Fix pydantic warning in the commit process [iceberg-python]

2024-07-30 Thread via GitHub
sungwy merged PR #972: URL: https://github.com/apache/iceberg-python/pull/972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Pydantic warning when appending data [iceberg-python]

2024-07-30 Thread via GitHub
sungwy closed issue #971: Pydantic warning when appending data URL: https://github.com/apache/iceberg-python/issues/971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Flink: a few small fixes or tuning for range partitioner [iceberg]

2024-07-30 Thread via GitHub
stevenzwu commented on code in PR #10823: URL: https://github.com/apache/iceberg/pull/10823#discussion_r1697716288 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SketchRangePartitioner.java: ## @@ -47,18 +46,6 @@ class SketchRangePartitioner implements

Re: [PR] Configure default table root location in catalog [iceberg-rust]

2024-07-30 Thread via GitHub
fqaiser94 closed pull request #494: Configure default table root location in catalog URL: https://github.com/apache/iceberg-rust/pull/494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Bump up version in dev Dockerfile and Issue Template [iceberg-python]

2024-07-30 Thread via GitHub
sungwy merged PR #981: URL: https://github.com/apache/iceberg-python/pull/981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-30 Thread via GitHub
amogh-jahagirdar commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1697706041 ## api/src/main/java/org/apache/iceberg/RemoveUnusedSpecs.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-30 Thread via GitHub
amogh-jahagirdar commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1697706041 ## api/src/main/java/org/apache/iceberg/RemoveUnusedSpecs.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1697699881 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1472 @@ +/* + * Licensed to the Apache Software

Re: [I] Configure root path in Catalog or FileIO? [iceberg-rust]

2024-07-30 Thread via GitHub
fqaiser94 commented on issue #488: URL: https://github.com/apache/iceberg-rust/issues/488#issuecomment-2259316327 Ack, let me see if I can refactor a little bit to match the suggested logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Core: Adds Basic Classes for Iceberg Table Version 3 [iceberg]

2024-07-30 Thread via GitHub
amogh-jahagirdar commented on code in PR #10760: URL: https://github.com/apache/iceberg/pull/10760#discussion_r1697664698 ## core/src/main/java/org/apache/iceberg/ManifestWriter.java: ## @@ -217,6 +217,79 @@ public void close() throws IOException { writer.close(); } +

Re: [PR] Flink: Remove MiniClusterResource in Flink [iceberg]

2024-07-30 Thread via GitHub
tomtongue commented on PR #10817: URL: https://github.com/apache/iceberg/pull/10817#issuecomment-2259305825 Thanks so much for the review and merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] DOCS: Add more notes post release notes [iceberg-python]

2024-07-30 Thread via GitHub
sungwy opened a new pull request, #983: URL: https://github.com/apache/iceberg-python/pull/983 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697662591 ## crates/iceberg/src/scan.rs: ## @@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> { field_ids.push(field_id); } -Ok(TableScan { +

[I] View Default Properties in the Catalog [iceberg]

2024-07-30 Thread via GitHub
RussellSpitzer opened a new issue, #10822: URL: https://github.com/apache/iceberg/issues/10822 ### Feature Request / Improvement Currently we support default table properties with the prefix `table-default` (#4011 ) but these only apply to Tables and not to Views (or theoretically fu

Re: [PR] Flink: backport PR #10748 for limit pushdown [iceberg]

2024-07-30 Thread via GitHub
stevenzwu commented on PR #10813: URL: https://github.com/apache/iceberg/pull/10813#issuecomment-2259270188 thanks @pvary for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Flink: backport PR #10748 for limit pushdown [iceberg]

2024-07-30 Thread via GitHub
stevenzwu merged PR #10813: URL: https://github.com/apache/iceberg/pull/10813 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[I] `org.apache.iceberg.exceptions.NoSuchTableException` / Flink / Rest Catalog [iceberg]

2024-07-30 Thread via GitHub
Al-Moatasem opened a new issue, #10821: URL: https://github.com/apache/iceberg/issues/10821 ### Apache Iceberg version 1.6.0 (latest release) ### Query engine Flink ### Please describe the bug ๐Ÿž Hello, I am trying to write data from PyFlink/SQL API to

Re: [PR] Core: Adds Basic Classes for Iceberg Table Version 3 [iceberg]

2024-07-30 Thread via GitHub
amogh-jahagirdar commented on PR #10760: URL: https://github.com/apache/iceberg/pull/10760#issuecomment-2259268270 Sorry for the delay on reviewing @RussellSpitzer I'm taking a look! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[PR] API: Update Conversions.fromPartitionString API to handle timestamps [iceberg]

2024-07-30 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10820: URL: https://github.com/apache/iceberg/pull/10820 This can be considered as an alternative to https://github.com/apache/iceberg/pull/10724 In #10724, a new API on the DataFile builder was proposed which enabled callers to pass in cu

Re: [PR] DOCS: Add Github Actions Screenshots to Release Notes [iceberg-python]

2024-07-30 Thread via GitHub
sungwy merged PR #975: URL: https://github.com/apache/iceberg-python/pull/975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Declare avro as an api dependency of iceberg-core [iceberg]

2024-07-30 Thread via GitHub
findepi commented on PR #10573: URL: https://github.com/apache/iceberg/pull/10573#issuecomment-2259242140 Merged, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Docs: Use link addresses instead of descriptions in releases.md [iceberg]

2024-07-30 Thread via GitHub
pvary merged PR #10815: URL: https://github.com/apache/iceberg/pull/10815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Declare avro as an api dependency of iceberg-core [iceberg]

2024-07-30 Thread via GitHub
findepi merged PR #10573: URL: https://github.com/apache/iceberg/pull/10573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Docs: Use link addresses instead of descriptions in releases.md [iceberg]

2024-07-30 Thread via GitHub
pvary commented on PR #10815: URL: https://github.com/apache/iceberg/pull/10815#issuecomment-2259240586 Thanks @lurnagao-dahua for the fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-30 Thread via GitHub
pvary commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2259232342 @RussellSpitzer: The compaction started at t1 and it will replace files created before t1. The equality delete is created later at t2. So if we use the same sequence number for the n

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-30 Thread via GitHub
venkata91 commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-2259232252 @pvary Gentle ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

2024-07-30 Thread via GitHub
findepi commented on code in PR #10818: URL: https://github.com/apache/iceberg/pull/10818#discussion_r1697604466 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -48,28 +40,6 @@ public void testNullArguments() { .hasMessage("Invalid JSON strin

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-30 Thread via GitHub
stevenzwu commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1697583182 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkExtended.java: ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1697550150 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on issue #482: URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2259129551 Some additional context. Renjie mentioned: > Cons: > It's mixing logging with tracing. If we use tracing for logging, our logging compability will not be compatible with logg

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1697522014 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkExtended.java: ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1697521191 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkExtended.java: ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1697519616 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkExtended.java: ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-30 Thread via GitHub
rodmeneses commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1697519044 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkExtended.java: ## @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundatio

[PR] Bump up Iceberg version to 1.6.0 [iceberg-python]

2024-07-30 Thread via GitHub
ndrluis opened a new pull request, #982: URL: https://github.com/apache/iceberg-python/pull/982 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] feat: podman support [iceberg-rust]

2024-07-30 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1697497521 ## CONTRIBUTING.md: ## @@ -110,14 +110,16 @@ $ cargo version cargo 1.69.0 (6e9a83356 2023-04-12) ``` - Install docker + Install Docker or Podman -Curr

Re: [PR] feat: podman support [iceberg-rust]

2024-07-30 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1697497521 ## CONTRIBUTING.md: ## @@ -110,14 +110,16 @@ $ cargo version cargo 1.69.0 (6e9a83356 2023-04-12) ``` - Install docker + Install Docker or Podman -Curr

Re: [PR] feat: podman support [iceberg-rust]

2024-07-30 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1697495594 ## crates/test_utils/src/cmd.rs: ## @@ -28,14 +28,27 @@ pub fn run_command(mut cmd: Command, desc: impl ToString) { } } -pub fn get_cmd_output(mut cmd: Comm

[PR] Bump up version in dev Dockerfile and Issue Template [iceberg-python]

2024-07-30 Thread via GitHub
ndrluis opened a new pull request, #981: URL: https://github.com/apache/iceberg-python/pull/981 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] SQL Catalog [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on code in PR #503: URL: https://github.com/apache/iceberg-rust/pull/503#discussion_r1697470138 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,836 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. Se

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697473790 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,159 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697470881 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,159 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697462920 ## api/src/main/java/org/apache/iceberg/transforms/Bucket.java: ## @@ -214,6 +217,20 @@ protected int hash(Long value) { } } + // In order to bucket Timestamp

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697462451 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -515,6 +548,18 @@ public Literal to(Type type) { return (Literal) new TimestampLit

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697461646 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -453,6 +458,34 @@ protected Type.TypeID typeId() { } } + static class TimestampNanoLi

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697460829 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -299,6 +300,8 @@ public Literal to(Type type) { return (Literal) new TimeLiteral(val

Re: [PR] Ensure that RestCatalog passes user config to FileIO [iceberg-rust]

2024-07-30 Thread via GitHub
sdd commented on code in PR #476: URL: https://github.com/apache/iceberg-rust/pull/476#discussion_r1697459277 ## crates/catalog/rest/src/catalog.rs: ## @@ -504,8 +504,15 @@ impl Catalog for RestCatalog { .query::(request) .await?; +let config

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-30 Thread via GitHub
rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1697454916 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -515,6 +523,8 @@ private static String sanitize(Type type, Object value, long now, int to

  1   2   >