ewoodbury commented on code in PR #3255:
URL: https://github.com/apache/iceberg-rust/pull/3255#discussion_r4069962271
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -831,25 +1099,26 @@ impl RecordBatchTransformer {
//
// At this point, all field IDs in the source schema are
trustworthy.
// No conflict detection needed - schema resolution happened
in reader.rs.
- let field_by_id =
field_id_to_source_schema_map.get(field_id).map(
- |(source_field, source_index)| {
- if
source_field.data_type().equals_datatype(target_type) {
- ColumnSource::PassThrough {
- source_index: *source_index,
- }
- } else {
- ColumnSource::Promote {
- target_type: target_type.clone(),
- source_index: *source_index,
- }
- }
- },
- );
-
+ //
// Apply spec's fallback steps for "not present" fields.
// Rule #1 (constants) is handled at the beginning of this
function
- let column_source = if let Some(source) = field_by_id {
- source
+ let column_source = if let Some((source_field, source_index)) =
+ field_id_to_source_schema_map.get(field_id)
+ {
+ if source_field.data_type().equals_datatype(target_type) {
+ ColumnSource::PassThrough {
+ source_index: *source_index,
+ }
+ } else {
+ ColumnSource::Promote {
+ plan: PromotePlan::build(
+ source_field.data_type(),
+ target_type,
+ snapshot_schema,
+ )?,
+ source_index: *source_index,
+ }
+ }
Review Comment:
Yeah - thanks for the explanation, and done in 850c785e3. A present column
always goes through PromotePlan::build, and passes through only when the plan
returns PassThrough. Reorder and same-type rename are covered by
promote_struct_reorders_children_by_id_via_process_record_batch and
promote_struct_renames_child_via_process_record_batch. Drop-and-re-add of the
same name is promote_struct_dropped_and_readded_same_name_nulls_by_id: old id 5
values come back null under id 6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
