stevenzwu commented on PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#issuecomment-2223275495
hmm. the `TestSparkDataFile` failed after this change.
```
Caused by: org.apache.iceberg.exceptions.ValidationException: Invalid
schema: multiple fields for name ts: 9 and 9
at
org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)
at org.apache.iceberg.types.IndexByName.addField(IndexByName.java:195)
at org.apache.iceberg.types.IndexByName.field(IndexByName.java:161)
at org.apache.iceberg.types.IndexByName.field(IndexByName.java:34)
at org.apache.iceberg.types.TypeUtil.visit(TypeUtil.java:611)
at org.apache.iceberg.types.TypeUtil.indexNameById(TypeUtil.java:173)
at org.apache.iceberg.Schema.lazyIdToName(Schema.java:213)
at org.apache.iceberg.Schema.<init>(Schema.java:142)
at org.apache.iceberg.Schema.<init>(Schema.java:99)
at org.apache.iceberg.Schema.<init>(Schema.java:87)
at org.apache.iceberg.Schema.<init>(Schema.java:83)
at org.apache.iceberg.StructTransform.<init>(StructTransform.java:82)
at org.apache.iceberg.PartitionKey.<init>(PartitionKey.java:37)
at
org.apache.iceberg.spark.source.SparkWrite$PartitionedDataWriter.<init>(SparkWrite.java:774)
```
This test class apply two partition spec transformation (identify and hour)
on the same field `ts`. Is that a legitimate use case? If it is yes, then we
have a problem with this proposed change.
```
private static final PartitionSpec SPEC =
PartitionSpec.builderFor(SCHEMA)
.identity("b")
.bucket("i", 2)
.identity("l")
.identity("f")
.identity("d")
.identity("date")
.hour("ts")
.identity("ts")
.identity("tsntz")
.truncate("s", 2)
.identity("bytes")
.bucket("dec_9_0", 2)
.bucket("dec_11_2", 2)
.bucket("dec_38_10", 2)
.build();
```
cc @rdblue @aokolnychyi @RussellSpitzer @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
