kumarpritam863 commented on PR #11174:
URL: https://github.com/apache/iceberg/pull/11174#issuecomment-2418322573

   Hi Bryan,
   
   Thank you for your response. I completely agree with your view that the 
logic for Single Message Transforms (SMTs) should not be part of any sink. 
However, in the context of this PR, I have a few considerations about moving 
this to an SMT:
   
   **Performance Impact**: While SMTs provide flexibility, they can introduce 
performance overhead. Adding an SMT increases the time complexity by O(number 
of records), which may seem like O(N), but becomes significant when N involves 
processing millions of records.
   
   **General Applicability**: SMTs are typically designed to work across 
different connectors, ensuring they remain connector-agnostic. In this case, 
however, the transformation logic is closely tied to deciding whether to 
convert to Iceberg's time type. Moving this to an SMT would result in an SMT 
that is specifically tailored for Kafka Connect Iceberg, which might limit its 
broader applicability.
   
   **Overhead of Moving Logic**: The current logic is a simple conditional 
check. Moving this to an SMT introduces additional processing complexity, 
essentially creating a new step or hop, which could come at a higher cost for 
what is essentially a straightforward type conversion.
   
   I'd be interested to hear your thoughts on this approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org

Reply via email to