RyanLin5967 opened a new issue, #4198:
URL: https://github.com/apache/iceberg-python/issues/4198
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
Currently, `UpdateSpec` runs the name check #2305 added against the schema
it loaded, and its commit asserts only the last partition id and default spec
id, so after another writer adds a column `xb`, a stale handle can add a
partition field also named `xb`. Without the concurrent change the same call
raises `ValueError: Cannot create partition with a name that exists in schema:
xb`. Java then refuses the metadata.
```python
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.transforms import BucketTransform
from pyiceberg.types import LongType
cat = SqlCatalog("c", uri="sqlite:///:memory:",
warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
t = cat.create_table("ns.t", schema=pa.schema([("x", pa.int64())]))
with cat.load_table("ns.t").update_schema() as u: # another writer
u.add_column("xb", LongType())
with t.update_spec() as u:
u.add_field("x", BucketTransform(4), "xb")
t = cat.load_table("ns.t")
print(t.schema().column_names, [f.name for f in t.spec().fields])
# ['x', 'xb'] ['xb'], should raise
```
Loading this metadata with Java's `TableMetadataParser` fails with
`IllegalArgumentException: Cannot create identity partition sourced from
different field in schema: xb`. The opposite order does the same: a stale
`update_schema().add_column("xb")` after another writer added partition field
`xb` commits; #4190 is another stale `update_schema` that
`AssertCurrentSchemaId` lets through.
tested on pyiceberg 0.12.0 and main
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
