Saamu192 opened a new issue, #2828:
URL: https://github.com/apache/iceberg-python/issues/2828
### Question
### Branching on Nessie
Hello,
I am having difficulties creating a branch using PyIceberg, and I am not
sure whether this is a misuse of the API or a compatibility issue between
PyIceberg and Nessie.
I am trying to create a branch from an existing snapshot using the following
code.
```
full_path = f"{self.database}.{name}"
table = self.__api.load_table(full_path)
if not snapshot_id:
snapshot = table.current_snapshot()
else:
snapshot = table.snapshot_by_id(snapshot_id)
if not snapshot:
raise KeyError(f"No snapshots found for table {table.name}")
table.manage_snapshots().create_branch(
snapshot_id=snapshot.snapshot_id,
branch_name=branch_name,
max_ref_age_ms=604800000,
max_snapshot_age_ms=259200000,
min_snapshots_to_keep=10,
).commit()
```python
**Error Received**
When calling .commit(), I consistently receive the following error:
```
Traceback (most recent call last):
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/catalog/rest/__init__.py",
line 720, in commit_table
response.raise_for_status()
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/requests/models.py",
line 1026, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url:
http://localhost:19120/iceberg/v1/main/namespaces/dev_external_dp/tables/d_nbs_service_description
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/samu/projects/govbr_pipeline/dags/sdp_govbr/jobs/pyiceberg/stage/stage_to_analytics.py",
line 150, in proccess_stage_to_analytics
iceberg.create_branch(
File "/home/samu/projects/govbr_pipeline/dags/sdp_govbr/sdp_pyiceberg.py",
line 182, in create_branch
).commit()
^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/table/update/__init__.py",
line 72, in commit
self._transaction._apply(*self._commit())
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/table/__init__.py",
line 295, in _apply
self.commit_transaction()
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/table/__init__.py",
line 936, in commit_transaction
self._table._do_commit( # pylint: disable=W0212
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/table/__init__.py",
line 1458, in _do_commit
response = self.catalog.commit_table(self, requirements, updates)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/tenacity/__init__.py",
line 338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/tenacity/__init__.py",
line 477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/tenacity/__init__.py",
line 378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/tenacity/__init__.py",
line 400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File
"/home/samu/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py",
line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File
"/home/samu/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py",
line 401, in __get_result
raise self._exception
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/tenacity/__init__.py",
line 480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/catalog/rest/__init__.py",
line 722, in commit_table
_handle_non_200_response(
File
"/home/samu/projects/govbr_pipeline/.venv/lib/python3.12/site-packages/pyiceberg/catalog/rest/response.py",
line 111, in _handle_non_200_response
raise exception(response) from exc
pyiceberg.exceptions.CommitFailedException: : Requirement failed: ref must
be 'main', but is 'dev_govbr_external_branch_2025_12_15'
```text
**Versions**
- PyIceberg: 0.10.0
- Nessie: 0.105.5
Nessie Docker config:
```
nessie:
image: ghcr.io/projectnessie/nessie:0.105.5
container_name: nessie
networks:
- local-stack-network
environment:
- nessie.version.store.type=IN_MEMORY
- nessie.ref=main
- nessie.catalog.default-warehouse=${WAREHOUSE_NAME}
-
nessie.catalog.warehouses.${WAREHOUSE_NAME}.location=${WAREHOUSE_BUCKET_PATH}
- nessie.catalog.service.s3.default-options.endpoint=http://minio:9000/
-
nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key
- nessie.catalog.service.s3.default-options.path-style-access=true
- nessie.catalog.service.s3.default-options.auth-type=STATIC
- nessie.catalog.secrets.access-key.name=${AWS_ACCESS_KEY_ID}
- nessie.catalog.secrets.access-key.secret=${AWS_SECRET_ACCESS_KEY}
- nessie.catalog.service.s3.default-options.region=${AWS_REGION}
ports:
- "19120:19120"
```yaml
**Instancing Pyiceberg**
```
connection_options = {
"region": dc.MINIO_REGION,
"s3.endpoint": f"http://{dc.MINIO_HOST}:{dc.MINIO_PORT}";,
"s3.access-key-id": dc.MINIO_ACCESS,
"s3.secret-access-key": dc.MINIO_SECRET,
"s3.region": dc.MINIO_REGION,
}
if is_local:
uri = "http://localhost:19120/iceberg/main/";
else:
uri = f"glue.{connection_options['region']}.amazonaws.com/iceberg"
self.__api = load_catalog(
"rest",
**{
"uri": uri,
"snapshot-loading-mode": "refs",
"ref": "main",
**connection_options,
},
)
self.__api.create_namespace_if_not_exists(database)
self.database = database
```python
Is this the expected behavior when creating branches via
manage_snapshots().create_branch() with the REST catalog and Nessie? Am I
missing an additional step to switch references
Any guidance would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
