Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

pvary commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2454428555 I think we should backport this change to Flink 1.19, and Flink 1.18 as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

RussellSpitzer merged PR #11386: URL: https://github.com/apache/iceberg/pull/11386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

RussellSpitzer commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2444840468 Thanks @jbonofre for the PR and @Fokko and @pvary for Review. Let's add some additional tests for Spark and Flink in a followup for the old path. -- This is an automated messa

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

RussellSpitzer commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2444803082 Alright! Chatted with folks and we will do the old-path tests in a follow up. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2443246368 @RussellSpitzer the planned Avro reads has been added to Spark (for Iceberg 1.7.x). This one is not a blocker for 1.7.0 but a good to have to benefit the same performance boost as Spar

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

RussellSpitzer commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2442236629 @pvary + @jbonofre how important is this for 1.7? If it is important we need to wrap this up ASAP -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on code in PR #11386: URL: https://github.com/apache/iceberg/pull/11386#discussion_r1818458279 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/data/TestFlinkAvroReaderWriter.java: ## @@ -91,7 +91,7 @@ private void writeAndValidate(Schema schema, Li

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2440687833 I fixed all types (primitves and arrays) reads. Tests should be happy now 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2439933515 I'm fixing a new `ClassCastException`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

pvary commented on code in PR #11386: URL: https://github.com/apache/iceberg/pull/11386#discussion_r1817698482 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/data/TestFlinkAvroReaderWriter.java: ## @@ -91,7 +91,7 @@ private void writeAndValidate(Schema schema, List

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2439382344 I fixed the issue on `ValueReaders` about strings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on code in PR #11386: URL: https://github.com/apache/iceberg/pull/11386#discussion_r1816086840 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkPlannedAvroReader.java: ## @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

pvary commented on code in PR #11386: URL: https://github.com/apache/iceberg/pull/11386#discussion_r1816085856 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkPlannedAvroReader.java: ## @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2435789965 The problem seems to be related to: ``` java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.flink.table.data.StringData (java.lang.String i

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2435037863 It seems the upsert test is stale. I'm investigating (maybe the schema type mapping). -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Flink 1.20: Update Flink to use planned Avro reads [iceberg]

jbonofre commented on PR #11386: URL: https://github.com/apache/iceberg/pull/11386#issuecomment-2434604147 @pvary @aokolnychyi @rdblue this PR mimics what has been done in Spark to use Avro planned reads. -- This is an automated message from the Apache Git Service. To respond to the messa