kumarpritam863 opened a new pull request, #13745:
URL: https://github.com/apache/iceberg/pull/13745

   **Issue**
   
   Currently the consumers and producers responsible for consuming and 
producing from control topic is hardcoded to have StringSerializer and 
StringDeSerializer for Key and ByteArraySerializer and ByteArrayDeserializer 
for value. This is although not an issue as the producers and consumers are of 
<String, byte[]> but limits users in certain ways like the following:
   - If the size of control message is large kafka throws LargeMessageException 
and the only way for users is to keep on increasing the topic size at the 
broker levels.
   - If user wants a level of encryption and decryption for the control topic 
data for privacy issues then there is no way we can achieve that.
   Solution
   
   This PR proposes to take the Serializer and Deserializer from config and it 
will allow only those ** Serializer** and ** Deserializer** which extends from 
Serializer<byte[]> and Deserializer<byte[]>, this way still not breaking the 
contact the producers and consumers have.
   Motivation
   
   This change is motivated from the large message issue we faced as we operate 
at a very large scale and when ever we do bootstrapping with large 
commit-intervals, we face large message exception and for that we have our own 
Serializer and Deserializer which handles that very efficiently which we also 
will be pushing to OSS for wider audience.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to