Re: [PR] [wip] Table updates [iceberg-python]

2024-07-29 Thread via GitHub
chinmay-bhat commented on code in PR #950: URL: https://github.com/apache/iceberg-python/pull/950#discussion_r1694696015 ## pyiceberg/table/__init__.py: ## @@ -1129,12 +1128,22 @@ def _(update: SetSnapshotRefUpdate, base_metadata: TableMetadata, context: _Tabl return base_

Re: [PR] Flink: Disabling flaky test TestIcebergSourceFailover.testBoundedWithSavepoint [iceberg]

2024-07-29 Thread via GitHub
nastra commented on code in PR #10802: URL: https://github.com/apache/iceberg/pull/10802#discussion_r1694699704 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -137,6 +138,7 @@ protected void assertRecords(Table table, Lis

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.26.21 to 2.26.25 [iceberg]

2024-07-29 Thread via GitHub
nastra merged PR #10800: URL: https://github.com/apache/iceberg/pull/10800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump nessie from 0.93.1 to 0.94.2 [iceberg]

2024-07-29 Thread via GitHub
nastra merged PR #10798: URL: https://github.com/apache/iceberg/pull/10798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Flink: Disabling flaky test TestIcebergSourceFailover.testBoundedWithSavepoint [iceberg]

2024-07-29 Thread via GitHub
nastra commented on code in PR #10802: URL: https://github.com/apache/iceberg/pull/10802#discussion_r1694709068 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -137,6 +138,7 @@ protected void assertRecords(Table table, Lis

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-29 Thread via GitHub
nastra commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2255138973 I don't have anything else and yes #10775 needs to be handled before 1.7 (would be good to start work on it rather sooner than later) -- This is an automated message from the Apache Git

Re: [PR] Support convert orc timestamptz [iceberg]

2024-07-29 Thread via GitHub
pvary commented on code in PR #9905: URL: https://github.com/apache/iceberg/pull/9905#discussion_r1694724039 ## orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java: ## @@ -296,24 +304,36 @@ private static TypeDescription buildOrcProjection( list.elementI

Re: [PR] Support convert orc timestamptz [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2255169998 > > @deniskuzZ: How this is solved in Hive? > > i think it's done in `Hive: Support timestamp with local zone in Hive3 (#1897)` #1897 does not have any ORC specific part. I sti

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
Fokko commented on PR #10780: URL: https://github.com/apache/iceberg/pull/10780#issuecomment-2255174333 > IIUC, this should get published to the central website. Maybe other repos can put a link in there contributing.md to reference this section once it gets published? Yes, that woul

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.17.0 to 3.18.0 [iceberg]

2024-07-29 Thread via GitHub
nastra merged PR #10801: URL: https://github.com/apache/iceberg/pull/10801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump mkdocs-material from 9.5.29 to 9.5.30 [iceberg]

2024-07-29 Thread via GitHub
Fokko merged PR #10796: URL: https://github.com/apache/iceberg/pull/10796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-2255212800 > > This basically resulted in "inconsistencies" between the 1.17-1.18/1.19 tests. > > @pvary Can you clarify what do you mean by the "inconsistencies" here? Differences woul

Re: [I] flink iceberg may occur duplication when succeed to write datafile and commit but checkpoint fail [iceberg]

2024-07-29 Thread via GitHub
pvary commented on issue #10765: URL: https://github.com/apache/iceberg/issues/10765#issuecomment-2255254484 I think this is the log for the 19517: ``` Jul 24, 2024 @ 04:59:45.158 Failed to trigger checkpoint for job ba65ea243c487f4f0fd52c158e4ed985 since Checkpoint triggering task f

Re: [PR] Flink: Disabling flaky test TestIcebergSourceFailover.testBoundedWithSavepoint [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #10802: URL: https://github.com/apache/iceberg/pull/10802#issuecomment-2255275307 Thanks for the review @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Flink: Disabling flaky test TestIcebergSourceFailover.testBoundedWithSavepoint [iceberg]

2024-07-29 Thread via GitHub
pvary merged PR #10802: URL: https://github.com/apache/iceberg/pull/10802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]

2024-07-29 Thread via GitHub
twuebi commented on issue #482: URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2255302470 Hi all, I agree with @sdd and suggest to stick with tracing. It is the default choice for logging in async applications and has the most wide-spread adoption. Regardi

Re: [I] Geospatial Support [iceberg]

2024-07-29 Thread via GitHub
desruisseaux commented on issue #10260: URL: https://github.com/apache/iceberg/issues/10260#issuecomment-2255315231 Citing only the libraries that I know (more may be available): * C/C++ * PROJ can parse all the above. * [ESRI prototype](https://github.com/Esri/ogc-crs-wkt-pa

Re: [I] Geospatial Support [iceberg]

2024-07-29 Thread via GitHub
desruisseaux commented on issue #10260: URL: https://github.com/apache/iceberg/issues/10260#issuecomment-2255338418 > In addition, how does one can tell the string is in a certain CRS format (maybe by reading the first few characters, or try catch exception handling )? For WKT 1 (if a

[I] Site - latest release not showing 1.6.0 [iceberg]

2024-07-29 Thread via GitHub
raphaelauv opened a new issue, #10803: URL: https://github.com/apache/iceberg/issues/10803 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 ![image](https://github.com/user-attachments/assets/49f793ee-ba0a-4387-9cce-5b90bd

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
ajantha-bhat commented on PR #10792: URL: https://github.com/apache/iceberg/pull/10792#issuecomment-2255378369 cc: @fqaiser94, you might be interested in the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Build: Bump mkdocs-awesome-pages-plugin from 2.9.2 to 2.9.3 [iceberg]

2024-07-29 Thread via GitHub
Fokko merged PR #10795: URL: https://github.com/apache/iceberg/pull/10795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

[PR] Docs: Section referring to the ASF roles, guidelines and process [iceberg]

2024-07-29 Thread via GitHub
Fokko opened a new pull request, #10804: URL: https://github.com/apache/iceberg/pull/10804 For context, see devlist thread: https://lists.apache.org/thread/nmblzq7fwjordzzm6wnmrzhfwt05tt13 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #10484: URL: https://github.com/apache/iceberg/pull/10484#issuecomment-2255494823 Hi @stevenzwu, I would like to move forward with this PR. I see the following open questions: 1. Lock implementation - I still think it is important to provide a working imple

Re: [PR] Flink-1.19: Fix the file offset mismatch when Flink reader first seek… [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #10567: URL: https://github.com/apache/iceberg/pull/10567#issuecomment-2255745495 @zhongyujiang: What if we test the reader operator recovery by using `OneInputStreamOperatorTestHarness`? We can store the state of the operator, and later use it to restore the reader. W

Re: [I] Site - latest release not showing 1.6.0 [iceberg]

2024-07-29 Thread via GitHub
Fokko commented on issue #10803: URL: https://github.com/apache/iceberg/issues/10803#issuecomment-2255761007 @raphaelauv Thanks for raising this 👍 @jbonofre A couple of steps that we need to take: It looks like we need to do a couple more steps: - docs: we need to creat

Re: [I] Site - latest release not showing 1.6.0 [iceberg]

2024-07-29 Thread via GitHub
jbonofre commented on issue #10803: URL: https://github.com/apache/iceberg/issues/10803#issuecomment-2255786387 Sorry, my bad, I forgot to publish it. I will do that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

2024-07-29 Thread via GitHub
pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1695119335 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-29 Thread via GitHub
pvary commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2255792704 @szehon-ho: This is a potential bug with compaction and equality delete files. Could you please help me understand if this is a valid scenarion? -- This is an automated message fro

Re: [PR] Flink 1.17: Supports batch queries using time ranges [iceberg]

2024-07-29 Thread via GitHub
pvary commented on PR #7362: URL: https://github.com/apache/iceberg/pull/7362#issuecomment-2255797701 @hililiwei: Are you still working on this? The last failure is maybe due some flaky tests. See: #10671 -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Site - latest release not showing 1.6.0 [iceberg]

2024-07-29 Thread via GitHub
jbonofre commented on issue #10803: URL: https://github.com/apache/iceberg/issues/10803#issuecomment-2255808739 This PR (https://github.com/apache/iceberg/pull/10783) should have updated the latest version on the website. Let me check. -- This is an automated message from the Apache Git S

[I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-07-29 Thread via GitHub
rphadke-trinity opened a new issue, #10808: URL: https://github.com/apache/iceberg/issues/10808 ### Apache Iceberg version 1.6.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Error: UnsupportedOperationException: Byte-buffer read unsu

[PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo opened a new pull request, #499: URL: https://github.com/apache/iceberg-rust/pull/499 This PR will remove an extra config parse logic that maps iceberg props into config directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Add Apache Iceberg 1.6.0 docs [iceberg]

2024-07-29 Thread via GitHub
Fokko merged PR #10807: URL: https://github.com/apache/iceberg/pull/10807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695271890 ## crates/iceberg/src/io/storage_memory.rs: ## @@ -15,31 +15,11 @@ // specific language governing permissions and limitations // under the License. -use st

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Fokko commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695272318 ## Cargo.toml: ## @@ -67,7 +67,7 @@ log = "^0.4" mockito = "^1" murmur3 = "0.5.2" once_cell = "1" -opendal = "0.47" +opendal = "0.48" Review Comment: Nit: I thi

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-07-29 Thread via GitHub
Fokko commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2256000728 @rphadke-trinity Thanks for reporting this, do you also have a full stacktrace? That would help to localize the problem. -- This is an automated message from the Apache Git Service

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695277156 ## Cargo.toml: ## @@ -67,7 +67,7 @@ log = "^0.4" mockito = "^1" murmur3 = "0.5.2" once_cell = "1" -opendal = "0.47" +opendal = "0.48" Review Comment: This PR r

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695276546 ## Cargo.toml: ## @@ -67,7 +67,7 @@ log = "^0.4" mockito = "^1" murmur3 = "0.5.2" once_cell = "1" -opendal = "0.47" +opendal = "0.48" Review Comment: Got, let

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695280401 ## Cargo.toml: ## @@ -67,7 +67,7 @@ log = "^0.4" mockito = "^1" murmur3 = "0.5.2" once_cell = "1" -opendal = "0.47" +opendal = "0.48" Review Comment: Hi, @liur

Re: [I] Configure root path in Catalog or FileIO? [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on issue #488: URL: https://github.com/apache/iceberg-rust/issues/488#issuecomment-2256011677 > Thanks for starting this discussion first. > > My reason for not including a `default_table_root_location` (or `default_warehouse_location`) at the catalog implement

Re: [I] Configure root path in Catalog or FileIO? [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on issue #488: URL: https://github.com/apache/iceberg-rust/issues/488#issuecomment-2256015393 > Sounds reasonable to me. A more complete method is here: https://github.com/apache/iceberg-python/blob/055938d36d46efb94849b1d861cd8a8a111f6ae5/pyiceberg/catalog/__init__.py#L880

Re: [PR] feat: add check compatible func for primitive type [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on PR #492: URL: https://github.com/apache/iceberg-rust/pull/492#issuecomment-2256018387 > It gives me a reflection that do we need the literal type like `PrimitiveLiteral::Timestamp`. Or we can just use `PrimitiveLiteral::Long`.🤔 Yeah, I also think we no longe

Re: [PR] feat: add check compatible func for primitive type [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on PR #492: URL: https://github.com/apache/iceberg-rust/pull/492#issuecomment-2256026279 > > Yes. I think that's why we introduce `Datum`. > > Thank you for the reminder! It makes sense for me to remove it now. I'll open an issue to track it. -- This i

Re: [PR] feat: add check compatible func for primitive type [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on code in PR #492: URL: https://github.com/apache/iceberg-rust/pull/492#discussion_r1695292644 ## crates/iceberg/src/spec/values.rs: ## @@ -76,7 +76,7 @@ pub enum PrimitiveLiteral { /// UTF-8 bytes (without length) String(String), /// 16-b

Re: [PR] feat: add check compatible func for primitive type [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on PR #492: URL: https://github.com/apache/iceberg-rust/pull/492#issuecomment-2256024569 > Yes. I think that's why we introduce `Datum`. Thank you for the reminder! It makes sense for me to remove it now. -- This is an automated message from the Apache Git Service.

Re: [PR] Docs: update document release steps [iceberg]

2024-07-29 Thread via GitHub
ajantha-bhat commented on PR #9921: URL: https://github.com/apache/iceberg/pull/9921#issuecomment-2256030607 Closing this PR as https://github.com/apache/iceberg/pull/10402 has handled the same. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Docs: update document release steps [iceberg]

2024-07-29 Thread via GitHub
ajantha-bhat closed pull request #9921: Docs: update document release steps URL: https://github.com/apache/iceberg/pull/9921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] chore: Bump OpenDAL to 0.48 [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo merged PR #500: URL: https://github.com/apache/iceberg-rust/pull/500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on code in PR #499: URL: https://github.com/apache/iceberg-rust/pull/499#discussion_r1695300548 ## crates/iceberg/src/io/storage_memory.rs: ## @@ -15,31 +15,11 @@ // specific language governing permissions and limitations // under the License. -use std::coll

[PR] Update PyIceberg Verify Release doc [iceberg-python]

2024-07-29 Thread via GitHub
chinmay-bhat opened a new pull request, #976: URL: https://github.com/apache/iceberg-python/pull/976 To allow spinning up containers for `make test-coverage` later, we now run `make install` directly on the unzipped release project directory, which will create a fresh python environment wit

Re: [PR] feat: add check compatible func for primitive type [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 merged PR #492: URL: https://github.com/apache/iceberg-rust/pull/492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Configure default table root location in catalog [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on PR #494: URL: https://github.com/apache/iceberg-rust/pull/494#issuecomment-2256052385 Thanks @fqaiser94 for this quick refactoring. In #488 we had further discussion about it, please take a look. -- This is an automated message from the Apache Git Service. To re

Re: [I] Scan does not work as expected [iceberg-rust]

2024-07-29 Thread via GitHub
ndrluis commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2256052967 @Xuanwo > Hi, does remote signing means presign in s3? Yes and no. I'm not sure if this is the flow, because I haven't found any documentation; this is based on my un

Re: [I] Scan does not work as expected [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2256072271 > I'm not comfortable closing this issue without a regression test that guarantees the expected behavior. +1 on this. Currently we don't have regression tests on the w

Re: [PR] refactor(iceberg): Remove an extra config parse logic [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 merged PR #499: URL: https://github.com/apache/iceberg-rust/pull/499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Add Apache Iceberg 1.6.0 javadoc [iceberg]

2024-07-29 Thread via GitHub
Fokko commented on PR #10809: URL: https://github.com/apache/iceberg/pull/10809#issuecomment-2256107758 The PR freezes my browser, but it feels like there is something off: - I would only expect added lines, no deletions - It affects almost twice the number of files compared to the

Re: [I] Scan does not work as expected [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2256126041 > It's a presign process, but it's not the client's responsibility to presign. The get config will return the s3.signer.uri, and the load table will return s3.remote-signing-enable

Re: [I] discussion: Simplify `PrimitiveLiteral` type. [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on issue #501: URL: https://github.com/apache/iceberg-rust/issues/501#issuecomment-2256129508 Thanks for this. I support this change, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695336217 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which container

Re: [I] discussion: Simplify `PrimitiveLiteral` type. [iceberg-rust]

2024-07-29 Thread via GitHub
ZENOTME commented on issue #501: URL: https://github.com/apache/iceberg-rust/issues/501#issuecomment-2256141727 Also +1 for this and I'm glad to send the PR to do this later if we decide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#issuecomment-2256148753 Hi, @alexyin1 > What is the purpose of ARG BUILDPLATFORM and when/where should that be used? It was only present in one of the Dockerfiles, so I removed it. It's us

Re: [PR] feat: permit Datum Date<->Int type conversion [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 merged PR #496: URL: https://github.com/apache/iceberg-rust/pull/496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: permit Datum Date<->Int type conversion [iceberg-rust]

2024-07-29 Thread via GitHub
liurenjie1024 commented on PR #496: URL: https://github.com/apache/iceberg-rust/pull/496#issuecomment-2256154481 I guess #501 would also simplify this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on PR #10739: URL: https://github.com/apache/iceberg/pull/10739#issuecomment-2256167831 Thanks for the reviews @danielcweeks , @fqaiser94 , and @ajantha-bhat !! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-29 Thread via GitHub
bryanck merged PR #10739: URL: https://github.com/apache/iceberg/pull/10739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2256202203 The validation error looks correct. Since we shouldn't be able to do the rewrite if the equality delete has been written in the middle. When we se using starting sequ

Re: [PR] Add Apache Iceberg 1.6.0 javadoc [iceberg]

2024-07-29 Thread via GitHub
jbonofre commented on PR #10809: URL: https://github.com/apache/iceberg/pull/10809#issuecomment-2256215778 @Fokko it's probably because I update the `latest` folder with 1.6.0. Should I only add 1.6.0 without changing `latest` ? -- This is an automated message from the Apache Git Service.

Re: [PR] Add Apache Iceberg 1.6.0 javadoc [iceberg]

2024-07-29 Thread via GitHub
jbonofre commented on PR #10809: URL: https://github.com/apache/iceberg/pull/10809#issuecomment-2256225891 @Fokko I updated this PR to just add `1.6.0` folder (not touching `latest` folder). -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] discussion: Simplify `PrimitiveLiteral` type. [iceberg-rust]

2024-07-29 Thread via GitHub
sdd commented on issue #501: URL: https://github.com/apache/iceberg-rust/issues/501#issuecomment-2256250188 This would make a lot of sense. I was also never keen on the names. Maybe we could rename Datum to `LogicalValue` and have `PhysicalValue` for primitive, similar to Parquet? -- Th

Re: [I] discussion: Simplify `PrimitiveLiteral` type. [iceberg-rust]

2024-07-29 Thread via GitHub
sdd commented on issue #501: URL: https://github.com/apache/iceberg-rust/issues/501#issuecomment-2256252671 Actually if we did that it would diverge from the Iceberg spec -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Quick notes how to update docs and javadoc at release publication time. [iceberg]

2024-07-29 Thread via GitHub
ajantha-bhat commented on code in PR #10810: URL: https://github.com/apache/iceberg/pull/10810#discussion_r1695445101 ## site/docs/how-to-release.md: ## @@ -270,6 +270,46 @@ git tag -am 'Release Apache Iceberg ' apache-iceberg- apache-i Then release the candidate repository

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-07-29 Thread via GitHub
huaxingao commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2256302276 @laithalzyoud Thanks for your work on this PR! I've noticed there hasn't been activity for a while, and I wanted to check if you're still able to continue working on it. If you're bus

Re: [I] discussion: Simplify `PrimitiveLiteral` type. [iceberg-rust]

2024-07-29 Thread via GitHub
ZENOTME commented on issue #501: URL: https://github.com/apache/iceberg-rust/issues/501#issuecomment-2256319932 I create an init PR #502 for this. I hope it can be a starting point to discuss what's going on if we simplify PrimitiveLiteral. So feel free for any suggestions. -- This is an

[PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang opened a new pull request, #10811: URL: https://github.com/apache/iceberg/pull/10811 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1695462421 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -597,6 +597,24 @@ public TableMetadata replaceProperties(Map rawProperties) { .b

Re: [I] Scan does not work as expected [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2256378008 > Me and @ZENOTME did integration tests in icelake before, and I have to say that without sql engine support, it's painful to maintain those tests. I agree that we need a SQL

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
dramaticlly commented on code in PR #10811: URL: https://github.com/apache/iceberg/pull/10811#discussion_r1695524219 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java: ## @@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695526821 ## crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml: ## @@ -15,6 +15,9 @@ # specific language governing permissions and limitations # under the Licen

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695528164 ## crates/catalog/hms/testdata/hms_catalog/Dockerfile: ## @@ -15,8 +15,6 @@ FROM --platform=$BUILDPLATFORM openjdk:8-jre-slim AS build -ARG BUILDPLATFORM Revi

Re: [PR] Spark Action to Analyze table [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1695534811 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Action to Analyze table [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1695534811 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
Xuanwo commented on PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#issuecomment-2256406009 > Is there a cleaner way of having "CONTRIBUTING.MD" appear in the website and as a toplevel file in the github? It's a little tricky getting both the "github.com" version and the "rust

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-07-29 Thread via GitHub
rphadke-trinity commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2256439997 `org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 12.0 failed 4 times, most recent failure: Lost task 0.3 in stage 12.0 (TID 205) (10.139.

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-07-29 Thread via GitHub
rphadke-trinity commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2256445409 Adding full stracktrace @Fokko ```python org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 12.0 failed 4 times, most recent failu

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1695559265 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
singhpk234 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695514258 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void te

[PR] SQL Catalog [iceberg-rust]

2024-07-29 Thread via GitHub
callum-ryan opened a new pull request, #503: URL: https://github.com/apache/iceberg-rust/pull/503 This is an add-on from @JanKaul's work in a previous PR - we have spoken on slack and he is happy for me to open this PR. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695613576 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void termi

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on PR #10736: URL: https://github.com/apache/iceberg/pull/10736#issuecomment-2256533305 Revapi broken (as expected) please fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] API: Define RepairManifests action interface [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10784: URL: https://github.com/apache/iceberg/pull/10784#discussion_r1695628950 ## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
singhpk234 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695631557 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void te

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695632061 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void ter

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
findepi commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1695640265 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob ma

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
findepi commented on PR #10793: URL: https://github.com/apache/iceberg/pull/10793#issuecomment-2256564863 > I am not sure, but does this mean an integer value like 2 now becomes 2.0? (if using java toString) no, this should be "2" > And in any case, as its not entirely backward

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695651524 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issue,

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695652798 ## CONTRIBUTING.md: ## @@ -110,14 +110,16 @@ $ cargo version cargo 1.69.0 (6e9a83356 2023-04-12) ``` - Install docker + Install Docker or Podman -Curr

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695653411 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issue,

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695652798 ## CONTRIBUTING.md: ## @@ -110,14 +110,16 @@ $ cargo version cargo 1.69.0 (6e9a83356 2023-04-12) ``` - Install docker + Install Docker or Podman -Curr

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695652798 ## CONTRIBUTING.md: ## @@ -110,14 +110,16 @@ $ cargo version cargo 1.69.0 (6e9a83356 2023-04-12) ``` - Install docker + Install Docker or Podman -Curr

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695665021 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void termi

  1   2   3   >