kumarpritam863 opened a new pull request, #13745: URL: https://github.com/apache/iceberg/pull/13745
**Issue** Currently the consumers and producers responsible for consuming and producing from control topic is hardcoded to have StringSerializer and StringDeSerializer for Key and ByteArraySerializer and ByteArrayDeserializer for value. This is although not an issue as the producers and consumers are of <String, byte[]> but limits users in certain ways like the following: - If the size of control message is large kafka throws LargeMessageException and the only way for users is to keep on increasing the topic size at the broker levels. - If user wants a level of encryption and decryption for the control topic data for privacy issues then there is no way we can achieve that. Solution This PR proposes to take the Serializer and Deserializer from config and it will allow only those ** Serializer** and ** Deserializer** which extends from Serializer<byte[]> and Deserializer<byte[]>, this way still not breaking the contact the producers and consumers have. Motivation This change is motivated from the large message issue we faced as we operate at a very large scale and when ever we do bootstrapping with large commit-intervals, we face large message exception and for that we have our own Serializer and Deserializer which handles that very efficiently which we also will be pushing to OSS for wider audience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
