CTTY opened a new issue, #3499: URL: https://github.com/apache/iceberg-rust/issues/3499
### Is your feature request related to a problem or challenge? `SimpleSnapshotProducer` (#3474) writes the data files added by a `FastAppendAction` into one manifest per partition spec, sequentially. iceberg-java's `FastAppend` differs in two ways that matter once appends get large: 1. **Rolling manifests.** Java wraps the manifest writer in `RollingManifestWriter`, which starts a new manifest once the current one reaches `commit.manifest.target-size-bytes` (default 8 MB; size is checked every 250 entries). A single append of many files therefore produces several right-sized manifests instead of one oversized one, which keeps scan planning able to skip and parallelize at manifest granularity. 2. **Parallel manifest writing.** `SnapshotProducer.writeDataManifests` writes the per-spec (and per-roll) manifests concurrently on the worker pool. Each manifest is an independent file, so there is no ordering dependency between writers. Neither is a correctness issue, so they were left out of the milestone-2 producer to keep that PR focused. This issue tracks adding them. ### Describe the solution you'd like Both changes are local to the added-manifest write step in `SimpleSnapshotProducer`; the surrounding structure already accommodates them: - `AddedManifests.manifests` is a `Vec<ManifestFile>` with a group-level reuse guard (schema id, format version), so one spec producing several manifests needs no shape change. - Manifest paths are allocated from `manifest_counter` before any write and recorded in `owned_artifacts`, so a failed or partially written manifest from a parallel writer is already on the cleanup list. - Files are grouped by spec in a `BTreeMap`, so output order stays deterministic as long as results are joined in input order. Proposed steps: 1. **Rolling writer.** Add a `RollingManifestWriter` (or equivalent) over `ManifestWriter` that tracks written bytes and rolls to a new path from the producer's counter when `commit.manifest.target-size-bytes` is exceeded. Return `Vec<ManifestFile>` per spec group and `extend` into `AddedManifests.manifests`. Honor the Java check cadence (every N entries) rather than per entry to keep the size estimate cheap. 2. **Parallel writes.** Once each spec group can yield multiple manifests, write the groups concurrently: allocate all paths up front (serially, under `&mut self`), then spawn one task per manifest through `table.runtime().io().spawn(...)` and collect with `try_join_all` so results stay in input order. `Table` is `Clone`, so each task can own its handle; `DataFile`s are cloned into the task. 3. **Bound concurrency.** Cap in-flight writers (e.g. to the runtime's IO parallelism) so a very large append does not open thousands of output streams at once. The `MergingSnapshotProducer` (milestone 3) should reuse the same rolling writer for its added and delete manifests, so it is worth landing the writer as a shared helper rather than inside the simple producer. Related: epic #3266, milestone-2 PR #3474, issue #3418. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
