twuebi commented on PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#issuecomment-5241351783
> Duplicate temporary IDs remain a separate issue. Multiple zeros can make
the name lookup used by table/metadata.go:2814-2841 nondeterministic. Suggested
fix: reject duplicate temporary IDs before reassignment. Please add persisted
v1/v2/v3 rejection tests covering default and historical specs/orders, shuffled
IDs proving exact matching, duplicate-zero rejection, and round-trip checks
that write paths emit only positive IDs.
@zeroshade I captured what Spark actually posts, by pointing
spark.sql.catalog.probe.uri at a stub REST catalog that logs the create-table
body. For `CREATE TABLE t (ints INT, floats DOUBLE, strings STRING) USING
iceberg PARTITIONED BY (ints, bucket(16, ints))` the request carries
`"partition-spec": {"spec-id":0,"fields":[{"name":"i
nts","transform":"identity","source-id":0,"field-id":1000},{"name":"ints_bucket","transform":"bucket[16]","source-id":0,"field-id":1001}]}`,
so duplicate placeholders are the normal wire form for partitioning one column
two ways and rejecting them would reject that statement. They also resolve
deterministically: `reassignIDs` looks each field up with
`sc.FindColumnName(f.SourceID())` (table/metadata.go:2815,2828) over a
`map[int]string` (schema.go:364-369), so both fields resolve to ints.
Duplicates that are redundant never reach us, Spark refuses `PARTITIONED BY
(years(ts), months(ts))` client-side with `IllegalArgumentException: Cannot add
redundant partition: 1000: ts_year: year(0) conflicts with 1001: ts_month:
month(0)` at
`PartitionSpec$Builder.checkForRedundantPartitions(PartitionSpec.java:424)`,
and identity twice on one placeholder is rejected by our own
`validatePartitionFields`(partitions.go:564). The test cases now use these
captured bodies verbatim.
Separately worth its own issue: our redundancy key is the exact transform
string with no dedupName equivalent, so `year(ts) + month(ts)` on one column
would be accepted if a non-Spark client sent it. If you'd still rather reject
duplicate placeholders outright, I'll switch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
