CTTY opened a new issue, #3669: URL: https://github.com/apache/iceberg-rust/issues/3669
### Is your feature request related to a problem or challenge? `TableScan::plan_files` loads manifests from one task spawned on the io runtime, and `ManifestReader::read` parses each manifest on that same task. The Avro decoding of every manifest in a snapshot therefore runs one after another, while the other runtime workers idle. Profiling planning on a 601-manifest table (TPC-H SF100 `lineitem`, written by pyiceberg) with a 4-worker runtime shows about 80% of plan time in `Manifest::try_from_avro_bytes` on a single core: | table | manifests | `plan_files` median | |---|---:|---:| | SF10 | 60 | 23.9 ms | | SF100 | 601 | 232.9 ms | `concurrency_limit_manifest_files` bounds how many manifests are *fetched* concurrently, but the parse that follows each fetch is still serialized. ### Describe the solution you'd like Run the manifest parse on the table runtime's `cpu` handle, which is what `Runtime::cpu` is documented for, while keeping the fetch on the calling io task. Concretely: - `ManifestReader` takes an optional `Runtime` and, when present, spawns `Manifest::try_from_avro_bytes` on `runtime.cpu()`. - `TableScan` passes the table's runtime through `ObjectCache::get_manifest`, so up to `concurrency_limit_manifest_files` parses run in parallel. `ObjectCache` itself stores no runtime, so `Table::with_runtime` keeps sharing the cache. - A panic or cancellation in a spawned parse surfaces as `ErrorKind::Unexpected` through the `JoinHandle`, and the plan stream ends with that error the same way it does for a read error today. No public API changes; all new items are `pub(crate)`. With this change the medians above drop to 7.0 ms (SF10) and 65.4 ms (SF100), about 3.5x, on the same 4-worker runtime. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
