Re: [PR] Spark 3.5: Update Spark to use planned Avro reads [iceberg]

2024-10-22 Thread via GitHub
rdblue commented on code in PR #11299: URL: https://github.com/apache/iceberg/pull/11299#discussion_r1811023288 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkPlannedAvroReader.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Spark 3.5: Update Spark to use planned Avro reads [iceberg]

2024-10-22 Thread via GitHub
rdblue merged PR #11299: URL: https://github.com/apache/iceberg/pull/11299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark 3.5: Update Spark to use planned Avro reads [iceberg]

2024-10-22 Thread via GitHub
aokolnychyi commented on code in PR #11299: URL: https://github.com/apache/iceberg/pull/11299#discussion_r1811003876 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkPlannedAvroReader.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark 3.5: Update Spark to use planned Avro reads [iceberg]

2024-10-11 Thread via GitHub
rdblue commented on PR #11299: URL: https://github.com/apache/iceberg/pull/11299#issuecomment-2408058064 Here are the benchmark results: ``` ## main Benchmark Mode Cnt Score Error Units IcebergSourceFlatAvroDataRead