manuzhang opened a new issue, #17720: URL: https://github.com/apache/iceberg/issues/17720
### Problem `StructInternalRow.collectionToArrayData` does not handle Iceberg `GEOMETRY` or `GEOGRAPHY` elements in older Spark modules. In Spark 4.1, nested geospatial values in arrays and maps reach the default switch branch and throw: ```text UnsupportedOperationException: Unsupported array element type ``` PR #14984 adds nested conversion in the Spark 4.2 module while adapting to Spark 4.2's `BinaryView` physical representation. The nested array/map support is new capability rather than a Spark 4.2 compatibility requirement and should be evaluated and ported independently to applicable older Spark versions. Spark 4.1 uses `GeometryVal` and `GeographyVal`, so its implementation must construct those values rather than copying the Spark 4.2 `BinaryView` code. Spark 4.0 applicability should be confirmed against that module's geospatial type mapping before adding code. The top-level generic geospatial getter in Spark 4.2 is a separate correctness fix and should remain in #14984. ### Proposed fix - Add `GEOMETRY` and `GEOGRAPHY` handling to `StructInternalRow.collectionToArrayData` in supported older Spark modules. - Convert both `ByteBuffer` and `byte[]` Iceberg WKB inputs into the Spark-version-specific physical value. - Cover geometry and geography in arrays and as map keys/values. - Cover null collection values and nested combinations such as a struct inside a list. - Include representative non-point WKB and relevant SRID behavior. ### Context Identified while reviewing Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
