Ignalina opened a new pull request, #2998: URL: https://github.com/apache/iceberg-rust/pull/2998
Going from 8.4 seconds per commit to 40 milliseconds, by adding one function, gives 207x speedup. Found building skade — an embedded Iceberg engine that commits from inside the writing process: https://codeberg.org/nordisk/skade Every fast-append runs validate_duplicate_files: loads the manifest list, then load_manifest() on every entry. O(live data files) Avro decode, per commit. The cost is the table's own history, re-decoded on every append. files check on check off 1004 1.42 s 61.7 ms 4004 8.42 s 40.6 ms Check off is FLAT across depth. Check on grows superlinearly. It shows up long before that. The bench is in the repo above: skade/examples/write_commit_bench.rs cargo run --release --example write_commit_bench oden, 32 cores, 200k rows: lone-commit 0.71 ms/commit first 5 commits 0.52 ms last 5 commits 0.86 ms 1.65x worse after 40 commits per-file commit 8 899 775 rows/s batched commit 37 770 955 rows/s 4.24x with_check_duplicate already exists on FastAppendAction. Unreachable without a Transaction, and a Transaction commits through a Catalog. An embedded writer with its own commit path cannot get at it. This adds: ```rust pub async fn stage_fast_append( table: &Table, data_files: Vec<DataFile>, ) -> Result<(Vec<TableUpdate>, Vec<TableRequirement>)> pub async fn stage_fast_append_with( table: &Table, data_files: Vec<DataFile>, check_duplicate: bool, ) -> Result<(Vec<TableUpdate>, Vec<TableRequirement>)> ``` Returns updates and requirements. Does not commit. Staged bytes identical either way. The check is a read-only precondition: passes or aborts. Never touches manifest, manifest list or the returned updates. Skipping it changes only whether an already-referenced path is rejected. Callers deriving file names from a per-call unique prefix cannot collide. Callers appending caller-supplied names must leave it on. stage_fast_append keeps true. Going from 8.4 seconds per commit to 40 milliseconds, by adding one function, gives 207x speedup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
