Fokko opened a new pull request, #10973: URL: https://github.com/apache/iceberg/pull/10973
This PR switches the encoder to the `BlockingDirectBinaryEncoder` that actually encodes the size of list and map objects in the Avro file. When doing projected reads, it can skip of the fields that are not relevant, without having to skip over each of the element or entry. The `ManifestWriteBenchmark` shows a slight decrease in write speed: ``` Before: 22.51099815 s/op with 0.1981214357719109 error After: 22.2674114166 s/op with 0.47593907631201676 error ``` cc @rustyconover @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
