andygrove opened a new issue, #7033:
URL: https://github.com/apache/datafusion-comet/issues/7033
### Describe the bug
The native Iceberg scan fails any query that reads a struct column, or a
list of structs, when a data file omits one of its optional nested fields, even
though the table schema never changed. Spark reads the missing field as NULL.
Iceberg accepts such files: its write compatibility check allows a missing
optional field at any depth, so a Spark write with
`write.spark.accept-any-schema=true` produces one, and so can other engines.
The scan fails with `Incorrect number of arrays for StructArray fields,
expected 2 got 1`, the error #6504 reported for files written before a nested
field was added. #6543 added a `CometScanRule` fallback for that case, but
`IcebergReflection.nestedFieldsAddedOrRenamed` compares the projected fields
against the table's schema history. Here no schema in the history lacks the
field, so the scan stays native. Reading the iceberg-rust code at the current
pin, `RecordBatchTransformer` passes the file's narrower struct through, and
arrow's struct cast then matches children by position and fails.
### Steps to reproduce
In a `CometTestBase` suite with a Hadoop Iceberg catalog `cat` and
`spark.comet.scan.icebergNative.enabled=true`:
```scala
spark.sql("""CREATE TABLE cat.db.t (id INT, s STRUCT<a: INT, b: INT>, l
ARRAY<STRUCT<x: INT, y: INT>>)
USING iceberg TBLPROPERTIES ('format-version'='2',
'write.spark.accept-any-schema'='true')""")
val full = StructType(Seq(
StructField("id", IntegerType),
StructField("s", StructType(Seq(StructField("a", IntegerType),
StructField("b", IntegerType)))),
StructField("l", ArrayType(StructType(Seq(StructField("x", IntegerType),
StructField("y", IntegerType)))))))
val narrow = StructType(Seq(
StructField("id", IntegerType),
StructField("s", StructType(Seq(StructField("a", IntegerType)))),
StructField("l", ArrayType(StructType(Seq(StructField("x",
IntegerType)))))))
withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, Row(1, 2),
Seq(Row(1, 2)))), 1), full)
.writeTo("cat.db.t").append()
// Iceberg accepts a file that omits the optional fields s.b and
l.element.y.
spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(2, Row(20),
Seq(Row(20)))), 1), narrow)
.writeTo("cat.db.t").append()
}
spark.sql("SELECT id, s FROM cat.db.t").collect()
```
On `main` at 66c63bd307 (Spark 4.1.3, Iceberg 1.11):
| Query | Spark | Comet native scan |
| --- | --- | --- |
| `SELECT id, s` | `[1,[1,2]] [2,[20,null]]` | `Iceberg scan error:
Unexpected => Arrow Schema Error, source: Invalid argument error: Incorrect
number of arrays for StructArray fields, expected 2 got 1` |
| `SELECT id, s.b` | `[1,2] [2,null]` | the same error |
| `SELECT id, l` | `[1,[[1,2]]] [2,[[20,null]]]` | the same error |
| `SELECT *` | the rows above | the same error |
### Expected behavior
The native scan reads a missing optional nested field as NULL, as Spark does.
### Additional context
Found in a nested-type audit of the native scans. Reading the code, the
iceberg-rust bump in #6934 should fix this, because at `b154603`
`RecordBatchTransformer` reconciles struct children by field id and fills
missing ones. That was not verified. Until then, the failure is loud rather
than a wrong result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
