ewoodbury commented on code in PR #3255:
URL: https://github.com/apache/iceberg-rust/pull/3255#discussion_r4235413008
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -1147,22 +1422,546 @@ mod test {
use std::collections::HashMap;
use std::sync::Arc;
+ use arrow_array::cast::AsArray;
+ use arrow_array::types::Int32Type;
use arrow_array::{
- Array, Date32Array, Float32Array, Float64Array, Int32Array,
Int64Array, RecordBatch,
- StringArray,
+ Array, ArrayRef, Date32Array, Float32Array, Float64Array, Int32Array,
Int64Array,
+ LargeListArray, ListArray, MapArray, RecordBatch, StringArray,
StructArray,
};
+ use arrow_buffer::{NullBuffer, OffsetBuffer};
use arrow_cast::cast;
- use arrow_schema::{DataType, Field, Schema as ArrowSchema};
+ use arrow_schema::{DataType, Field, Fields, Schema as ArrowSchema};
use super::{field_with_id, schema_to_arrow_schema};
- use crate::arrow::build_partition_constant;
+ use crate::ErrorKind;
use crate::arrow::record_batch_transformer::{
- RecordBatchTransformer, RecordBatchTransformerBuilder,
+ PromotePlan, RecordBatchTransformer, RecordBatchTransformerBuilder,
};
+ use crate::arrow::{DEFAULT_MAP_FIELD_NAME, build_partition_constant};
use crate::spec::{
- ListType, Literal, MapType, NestedField, PrimitiveType, Schema,
Struct, Type,
+ ListType, Literal, MapType, NestedField, PrimitiveType, Schema,
Struct, StructType, Type,
};
+ fn promote(source: &ArrayRef, target: &DataType, schema: &Schema) ->
crate::Result<ArrayRef> {
+ PromotePlan::build(source.data_type(), target, schema,
&HashMap::new())?.apply(source)
+ }
+
+ fn empty_schema() -> Schema {
+ Schema::builder().build().unwrap()
+ }
+
+ fn unevolved_struct_type() -> DataType {
+ DataType::Struct(Fields::from(vec![field_with_id(
+ "x",
+ DataType::Int32,
+ true,
+ 5,
+ )]))
+ }
+
+ fn evolved_struct_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ field_with_id("x", DataType::Int32, true, 5),
+ field_with_id("y", DataType::Int32, true, 6),
+ ]))
+ }
+
+ fn unevolved_struct_data(x_values: Vec<i32>) -> Arc<StructArray> {
+ Arc::new(StructArray::new(
+ Fields::from(vec![field_with_id("x", DataType::Int32, true, 5)]),
+ vec![Arc::new(Int32Array::from(x_values)) as ArrayRef],
+ None,
+ ))
+ }
+
+ fn transform_top_level(column: NestedField, file_column: ArrayRef) ->
crate::Result<ArrayRef> {
Review Comment:
Makes sense! With two suggestions, went ahead and just added that
reader-level test now: `test_read_evolved_nested_fields_by_field_id` in
reader/projection.rs: [direct link
](https://github.com/apache/iceberg-rust/pull/3255/changes/f136c367f07e45228c76d234988ad16c3aa4501c#diff-19136dea678677e1860361bd7d2237dc3cc9668ba87976845126cc64b9927fe6R2056)
It writes a real Parquet file with an old struct, list<struct> and
map<string, struct>, then reads it back through an evolved schema (reorder,
added children, int to long, null list, and map slots) and compares. The test
currently fails on main with `Arrow Schema Error`. If nested field ids go
missing, it errors rather than falling back to position, since every nested
struct gains children. Sliced input can't come out of a Parquet read, so
`promote_sliced_list_with_null_slot` covers that as a unit test in fff2420ca. I
kept the existing unit tests, since several are the cases from earlier reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
