sunchao commented on PR #6098: URL: https://github.com/apache/datafusion-comet/pull/6098#issuecomment-5782123655
@andygrove @viirya Addressed in [feccb4df](https://github.com/apache/datafusion-comet/pull/6098/commits/feccb4df4d5aec73fc48956babefc36016c1f1c0). The cache now shares a 4 MiB estimated serialized-plus-parsed budget across at most four entries, evicting least-recently-used entries until a new schema fits. Schemas exceeding the whole budget still decode without disturbing existing entries. The LRU promotion fix and early serialized-size check are preserved. I kept the full retained-size estimate because field/type allocations are real retained memory. The shared budget lets a wide working schema use space left by other entries without raising the previous maximum estimated cache retention. The docs now explain the budget and its independence from the 1 MiB scratch limit. Added regression tests for 8,000-column reuse across every codec and both decoder entry points, multi-entry eviction, exact budget boundaries, nested/schema metadata capacity, and release after reset. Also addressed the helper extraction and test diagnostics comments. All **156 shuffle tests** and **all-target Clippy with `-D warnings`** pass; formatting and diff checks pass. The new 8,000-column / 64-row benchmark compares warm and forced-cold decoding on the current code. Local `ci` profile point estimates were **2.30 ms vs 5.27 ms** uncompressed and **6.71 ms vs 9.82 ms** with LZ4 (10 samples, 1 s warm-up, 2 s measurement). These are local microbenchmark results, not whole-query measurements. The PR description now contains the current validation results and commands; TPC/JVM suites were not rerun locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
