manuzhang opened a new pull request, #18223: URL: https://github.com/apache/iceberg/pull/18223
### Rationale `AvroDataTestBase.testUnknownListType` and `testUnknownMapType` write a `list<unknown>` and a `map<string, unknown>` and expect a clean round trip, which only Avro can do. Unknown columns are omitted from data files. A struct drops the field — `TypeToMessageType.struct` skips children whose `field()` returns null — but a list or map would be left with no element or value type, so `TypeToMessageType.list`/`map` reject the schema, and the ORC schema conversion does the same. Every Parquet and ORC subclass works around this by overriding both tests to assert the rejection, so any new subclass with nested-type support has to rediscover the limitation and copy them again. ### Changes - `AvroDataTestBase`: add `supportsUnknownCollectionElements()`, defaulting to `true`, and route both tests through one helper that either round-trips or asserts the rejection. - `TestSparkOrcReader`, `TestSparkParquetReader`, `TestSparkRecordOrcReaderWriter`, `TestORCDataFrameWrite`, `TestParquetDataFrameWrite`, `TestParquetScan`: drop both overrides and return `false` from the hook. Spark 4.2 only. The same change applies to 3.5, 4.0 and 4.1, and can follow once the approach is agreed. ### Tests The seven affected classes run 393 tests with no failures. All 14 unknown-collection cases execute and none are skipped: `TestSparkAvroReader` covers the round-trip branch and the other six cover the rejection branch. --- **AI Disclosure** - Model: Claude Opus 5 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: extract the duplicated `testUnknownListType`/`testUnknownMapType` overrides in the Spark test hierarchy into a capability hook on `AvroDataTestBase`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
