Gabriel39 opened a new pull request, #68481:
URL: https://github.com/apache/doris/pull/68481
### What problem does this PR solve?
On `branch-4.1`, Iceberg scan splits only carry partition metadata when the
table's current spec is partitioned. After an external REPLACE changes a
partitioned table to an unpartitioned table, the old spec remains in the table
metadata but new files use a different spec ID. Missing split metadata makes
the BE default to spec 0, so DELETE can fail with `No partition data for
partitioned table` when committing the position-delete files. Dropping the last
partition field also affects tables containing files from both specs.
Preserve each data file's spec ID and partition JSON regardless of the
current table spec. Keep the existing partition-pruning and partition-counting
behavior unchanged.
### Release note
Fix Iceberg row-level deletes after replacing a partitioned table with an
unpartitioned table or dropping its last partition field.
### Tests
- Added two unit tests using real Iceberg metadata transitions. Both fail on
the original code because the spec ID is missing from the serialized scan
range, and pass with the fix. They also validate conversion to position-delete
metadata for the file's own spec.
- Ran `IcebergScanNodeTest`, `IcebergWriterHelperTest`, and
`IcebergTransactionTest`: **149 tests passed**, no failures or skips.
- FE Checkstyle: passed (`mvn validate -pl fe-core -am
-Dcheckstyle.skip=false`).
- Added `test_iceberg_delete_unpartitioned_evolution` to the external
Iceberg regression suite, covering Spark REPLACE followed by repeated Doris
DELETE, plus mixed-spec DELETE/UPDATE after dropping the last partition field.
It checks both query results against Spark and delete-file spec IDs, with
Parquet and ORC coverage.
- Regression Groovy syntax check passed. The external regression suite has
**not been run end-to-end locally**; it requires the Iceberg/Spark test
environment. Spark REPLACE exercises the metadata transition reported with
Trino RTAS.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Preserve per-file partition metadata for evolved
unpartitioned tables.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
