waterWang opened a new pull request, #17550:
URL: https://github.com/apache/iceberg/pull/17550
## Description
When `RewriteManifestsSparkAction.sortBy()` is called with a partition field
name that contains a dot (e.g., a nested struct field flattened to `a.b`), the
string concatenation in `sortColumn()` produces
`col("data_file.partition.a.b")`. Spark interprets this as a nested field
traversal (`partition` → `a` → `b`) rather than a single field named `a.b`,
causing `AnalysisException: [FIELD_NOT_FOUND] No such struct field a in a.b`.
### Root cause
The `sortColumn()` method at line 320 builds the Spark column reference by
raw string concatenation:
```java
col(DATA_FILE_PARTITION_COLUMN_NAME + "." + p)
```
When `p` contains a dot (e.g., `a.b`), the resulting reference is ambiguous.
### Fix
Wrap the partition field name in backticks so Spark treats it as a single
literal field name:
```java
col(DATA_FILE_PARTITION_COLUMN_NAME + ".`" + p + "`")
```
The sibling class `FixedWriter` (which handles `byte[]`) already does it
correctly.
### Affected versions
- `spark/v3.5`
- `spark/v4.0`
- `spark/v4.1`
### Testing
The existing tests pass because they use simple partition field names
without dots (e.g., `c1`, `c2_trunc`, `c3_bucket`). A test case with a dotted
partition field name would trigger the bug.
Closes #17425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
