kumarpritam863 commented on PR #11174: URL: https://github.com/apache/iceberg/pull/11174#issuecomment-2418322573
Hi Bryan, Thank you for your response. I completely agree with your view that the logic for Single Message Transforms (SMTs) should not be part of any sink. However, in the context of this PR, I have a few considerations about moving this to an SMT: **Performance Impact**: While SMTs provide flexibility, they can introduce performance overhead. Adding an SMT increases the time complexity by O(number of records), which may seem like O(N), but becomes significant when N involves processing millions of records. **General Applicability**: SMTs are typically designed to work across different connectors, ensuring they remain connector-agnostic. In this case, however, the transformation logic is closely tied to deciding whether to convert to Iceberg's time type. Moving this to an SMT would result in an SMT that is specifically tailored for Kafka Connect Iceberg, which might limit its broader applicability. **Overhead of Moving Logic**: The current logic is a simple conditional check. Moving this to an SMT introduces additional processing complexity, essentially creating a new step or hop, which could come at a higher cost for what is essentially a straightforward type conversion. I'd be interested to hear your thoughts on this approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
