kumarpritam863 opened a new pull request, #12954: URL: https://github.com/apache/iceberg/pull/12954
**Summary** This PR introduces **backward compatibility** support by implementing proper handling of field deletions during schema evolution. The changes ensure the sink can gracefully handle cases where required fields are missing from incoming records while maintaining data integrity. **Key Changes** Field Deletion Support: - Added comprehensive handling for missing required fields - Implemented nested field deletion tracking through SchemaUpdate.Consumer - Proper propagation of field removal through nested structures (structs, maps, arrays) **Backward Compatibility Improvements:** - New logic to detect and handle schema downgrade scenarios - Added validation for required field removals - Preserved existing data when fields are missing from new records **Refactored Conversion Logic:** - Created dedicated methods for field removal handling - Improved error messaging for schema mismatch cases - Added logging for field deletion operations **Behavior Changes** - New: Missing required fields now trigger schema evolution events instead of failures - New: Field deletions are properly propagated to the metadata layer - Changed: More detailed logging for schema evolution operations - Unchanged: All existing conversion behavior remains compatible **Testing Coverage** Added test cases for: - Simple field removal scenarios - Nested struct field deletions - Field removal in array elements - Map value field deletions - Deeply nested field removal cases **Verified compatibility with:** - Existing table schemas - All supported file formats - Various schema evolution scenarios **Migration Notes** - Backward Compatible: Existing pipelines continue working unchanged - New Capability: Can now handle intentional field removals -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
