GitHub user avamingli closed a discussion: Proposal: Binary Data Support for libpq's Extended Protocol with Consumption Tracking
### Description We( @avamingli and @wangliang03) propose enhancing libpq's Extended Protocol to support binary data transmission while adding message tracking capabilities, addressing request identified in previous PR: https://github.com/apache/cloudberry/pull/990#discussion_r2001607856 by @yjhjstz The current struct-based implementation(bitmap) suffers from extened-subtags and lacks proper validation of unconsumed messages. <div align="center"> <img src="https://github.com/user-attachments/assets/301e7de8-dd21-4b7e-a4af-f875f9d527ea"; alt="Description"> </div> Our solution introduces a structured binary format with type/length headers while maintaining backward compatibility, making it more general and easier subtags-extend for for developers. They don't need to consider what messages are of other subtags and only care about the ones created by themselves. The key innovation is a producer-consumer tracking system that will log warnings when messages remain unconsumed at transaction commit, preventing silent data leaks. The implementation is adding opt-in binary support with instrumentation to detect unconsumed messages. Memory management will leverage TopTransactionContext for automatic cleanup while adding reference counting for shared payloads. ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! GitHub link: https://github.com/apache/cloudberry/discussions/1092 ---- This is an automatically sent email for dev@cloudberry.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@cloudberry.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cloudberry.apache.org For additional commands, e-mail: dev-h...@cloudberry.apache.org
