peach12345 commented on issue #13406: URL: https://github.com/apache/iceberg/issues/13406#issuecomment-3089708625
Hi @mxm, yeah, the DynamicSink is really nice! Thanks for confirming that our approach should be a valid one :) Still wrestling with schema evolution—especially when the full data structure isn’t closed content. One of the trickiest parts is handling lists of structs, especially when those structs contain optional fields. So that the first struct/object in the list does not have all elements.. We ended up writing our own schema parser that dynamically generates schemas from incoming events and merges them with the existing table schema. It also checks the field/row order and reorders the data to match the expected schema. In the end we will not support every case and starting with a smaller MVP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org