nssalian opened a new pull request, #1238: URL: https://github.com/apache/iceberg-go/pull/1238
Closes [#986](https://github.com/apache/iceberg-go/issues/986). At scan time, reassembles shredded variant columns (metadata + residual value + typed_value) back into the plain unshredded layout, so a shredded variant reads the same as a non-shredded one. arrow-go's VariantArray does the per-row work. The code change is only in the projection visitor's Variant method. ## Fixture based Testing The fixtures come from [apache/parquet-testing](https://github.com/apache/parquet-testing/tree/master/shredded_variant) (written by Iceberg Java's TestVariantReaders). I first ran all 137 upstream cases through arrow-go to see where it stands (128 pass, 4 reject, 2 lenient, 3 spec-invalid), then committed 17 of them. These represent one per distinct reader behaviour instead of all 137: - 9 success cases checked against the Java golden value (array, empty array, scalar, object with a field that falls back to the residual value, object with no value column, nested array, null variant, unshredded data under a shredded schema, and a multi-row column with nulls). Compared as JSON because reassembly is valid but not byte-identical for nested objects. - 5 cases the reader should reject (value/typed_value conflicts, non-object-with-shredded-fields, and a spec-invalid file). - 3 cases where arrow-go reads something Java would reject (uint32 and fixed_size_binary typed_value, plus one spec-invalid file it reads). These pin the current behaviour so we notice if arrow-go changes it. ## Additional Testing - compute.Take / FilterRecordBatch leave a shredded column alone (they run before projection). - rows surviving a positional delete still reassemble correctly. - a leak check on the reassembly path. - a Spark 4 integration test that writes a shredded table and reads it back (checks the file is actually shredded first). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
