sunchao opened a new issue, #6097: URL: https://github.com/apache/datafusion-comet/issues/6097
### Describe the bug The shuffle schema cache introduced by #5809 has two cache-policy problems in `native/shuffle/src/ipc.rs`: - A hit swaps the selected entry with the front. This changes the relative recency of other entries, so an insertion can evict a more recently used schema. - The four-entry limit does not bound the size of the retained raw schema messages and parsed schemas. Large valid field names or schema metadata can remain cached for the lifetime of a worker thread. The metadata scratch limit does not cover those separate cache entries. ### Steps to reproduce For eviction order, decode schemas in the order `A B C D A E D` with a four-entry cache. Promoting A should leave D newer than B and C, so inserting E should evict B and the final D should hit. The current swap instead moves D to the oldest slot and E evicts it. For retention, decode a valid batch whose schema has a large field name or metadata value, then drop the decoded batch. The cache continues to own the parsed schema and its serialized message. Serialized size alone is insufficient because the parsed schema owns additional copies of strings. ### Expected behavior Cache hits preserve the recency order of other entries. Oversized valid schemas continue to decode, but bypass cache admission without evicting useful normal entries. Admission considers both serialized and parsed schema size. ### Additional context Verified in main at `ccdff0a4bfeee377ffa8654a1ef7843e63b3b8b8`. These are cache reuse and retention issues; the fix can preserve the IPC format and both existing decoder entry points. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
