zhengxiang378928908-code commented on PR #17685: URL: https://github.com/apache/iceberg/pull/17685#issuecomment-6322161383
@pvary since you asked for someone to take this over: I reproduced #17651 and opened #18602 with tests. On current `main` the crash comes from the path that sends records to the table updater (`immediateUpdate` disabled). There, the key is computed with the table spec or table schema id but the record's raw row. To answer @mxm's question: yes, that is the root cause. When `resolvedTableSchema` is null, `data.schema()` is passed as if it were a table schema, so its schema id (0) collides with the table's schema 0 in the `SelectorKey` cache. This PR changes `HashKeyGenerator` to always build the selector from the record schema. That also fixes the updater path, but it breaks the main path: there the row has already been converted to the table layout. With an existing table and records whose columns are in a different order, it fails with the same `ClassCastException` (`testHashDistributionWithReorderedColumns` in #18602; it passes on `main`). So #18602 fixes it in `DynamicRecordProcessor` instead and leaves the main path as it is. @waterWang thanks for the analysis here. If you'd rather continue on this PR, I'm happy to close mine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
