peach12345 commented on issue #13406:
URL: https://github.com/apache/iceberg/issues/13406#issuecomment-3089708625

   Hi @mxm,
   
   yeah, the DynamicSink is really nice! 
   Thanks for confirming that our approach should be a valid one :) 
   
   Still wrestling with schema evolution—especially when the full data 
structure isn’t closed content.
   One of the trickiest parts is handling lists of structs, especially when 
those structs contain optional fields. So that the first struct/object in the 
list does not have all elements.. 
   
   We ended up writing our own schema parser that dynamically generates schemas 
from incoming events and merges them with the existing table schema. It also 
checks the field/row order and reorders the data to match the expected schema.
   
   In the end  we will not support every case and starting with a smaller MVP. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org

Reply via email to