sunchao opened a new pull request, #6098: URL: https://github.com/apache/datafusion-comet/pull/6098
## Which issue does this PR close? Closes #6097. ## Rationale for this change The schema cache introduced by #5809 can evict a recently used schema because promoting a hit swaps another entry into its old slot. For example, `A B C D A E D` reparses D even though B is older. The four-entry capacity also leaves the size of retained serialized and parsed schema metadata unrestricted. ## What changes are included in this PR? - Rotate the prefix when promoting a cache hit, preserving the relative recency of the other entries. - Skip cache admission when the estimated serialized-plus-parsed schema size exceeds 1 MiB. Check before evicting entries or copying the serialized key. Oversized valid schemas still decode successfully and are reparsed on subsequent blocks. - Include recursive field sizing and top-level metadata string capacities in the estimate. This is a retention admission estimate, excluding allocator overhead, rather than an exact process-memory limit. The size calculation runs only on cache misses. The change is confined to `native/shuffle/src/ipc.rs`. Both existing decoder entry points retain their validation and complete-stream checks. ## How are these changes tested? Both new regression tests were run against the original cache logic and failed as expected: incorrect eviction produced 1 hit/6 misses instead of 2 hits/5 misses, and the oversized parsed schema remained alive after dropping the decoded batch. The other 16 decoder tests passed. With the fix: - `cargo test --locked --offline --profile ci -p datafusion-comet-shuffle`: all 152 tests passed, including existing warm-allocation checks. - Oversized-schema coverage checks large field names and schema metadata, serialized messages above the limit and messages whose parsed copy crosses it, all four codecs, both decoder entry points, schema release, and preservation of normal cache entries. - `cargo clippy --locked --offline --profile ci -p datafusion-comet-shuffle --all-targets -- -D warnings`. - `rustfmt --check --edition 2021 native/shuffle/src/ipc.rs` and `git diff --check`. Tested with Rust 1.98.1 and the upstream lockfile (Arrow 59.3.0, DataFusion 55.1.0). TPC and JVM suites were not rerun locally for this follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
