vakarisbk opened a new issue, #10046:
URL: https://github.com/apache/iceberg/issues/10046
### Apache Iceberg version
1.5.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Table commit fails with an error
`org.apache.iceberg.exceptions.CommitFailedException: Failed to update table
test.test_table from catalog rest_backend` when committing to a table that has
`iceberg_type` NULL in the underlying JDBC catalog.
```
2024-03-27T06:57:32.785 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] -
Error processing REST request
org.apache.iceberg.exceptions.RESTException: Unhandled error:
ErrorResponse(code=409, type=CommitFailedException, message=Failed to update
table test.test_table from catalog rest_backend)
org.apache.iceberg.exceptions.CommitFailedException: Failed to update table
test.test_table from catalog rest_backend
```
iceberg_type is a new column, introduced in 1.5.0 to support iceberg views.
Tables that were created in the previous Iceberg versions have a NULL value in
iceberg_type field.
This behaviour is expected, but it seems to cause commits to fail.
**How to reproduce the issue?**
**1. Create an environment with a rest-catalog, postgresql and spark. Sample
docker-compose for convenience:**
```
version: '3.8'
services:
spark:
ports:
- "15002:15002"
- "4040:4040"
image: spark:3.5.1
entrypoint: >
/opt/spark/bin/spark-submit
--class org.apache.spark.sql.connect.service.SparkConnectServer
--packages
org.apache.spark:spark-connect_2.12:3.5.1,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0
--conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog
--conf
spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.rest.RESTCatalog
--conf spark.sql.catalog.iceberg.uri=http://rest-catalog:8181
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
--conf spark.jars.ivy=/tmp
networks:
- app-network
rest-catalog:
image: tabulario/iceberg-rest:1.5.0
environment:
CATALOG_JDBC_USER: rest_catalog
CATALOG_JDBC_PASSWORD: rest_catalog
CATALOG_URI: jdbc:postgresql://postgres:5432/rest_catalog
ports:
- "8181:8181"
depends_on:
- postgres
networks:
- app-network
postgres:
image: postgres:latest
environment:
POSTGRES_USER: "rest_catalog"
POSTGRES_PASSWORD: "rest_catalog"
POSTGRES_DB: "rest_catalog"
volumes:
- postgres-data:/var/lib/postgresql/data
- ./init-scripts:/docker-entrypoint-initdb.d # Mount init script
directory
ports:
- "5432:5432"
networks:
- app-network
db-init:
image: postgres:latest
depends_on:
- postgres
environment:
POSTGRES_USER: "rest_catalog"
POSTGRES_PASSWORD: "rest_catalog"
POSTGRES_HOST: "postgres"
POSTGRES_DB: "rest_catalog"
networks:
- app-network
command: >
sh -c "
until pg_isready -h postgres -p 5432; do
echo 'Waiting for PostgreSQL to become ready...'
sleep 1
done;
echo 'PostgreSQL is ready. Executing init script.';
PGPASSWORD=$$POSTGRES_PASSWORD psql -h $$POSTGRES_HOST -U
$$POSTGRES_USER -d $$POSTGRES_DB -c 'GRANT ALL PRIVILEGES ON DATABASE
$$POSTGRES_DB TO $$POSTGRES_USER;'
"
networks:
app-network:
driver: bridge
volumes:
postgres-data:
```
**2. Use Spark Connect to create a table and insert sample data:**
```
./bin/pyspark --remote "sc://localhost"
>> spark.sql("create schema iceberg.test")
>> spark.sql("create table iceberg.test.test_table (one string, two
string, three string)")
>> spark.sql("insert into iceberg.test.test_table VALUES ("I", "LOVE",
"ICEBERG")
```
**3. Connect to the postgreSQL database and replace `iceberg_type` field
with `NULL`**
```
>> PGPASSWORD="rest_catalog" ./psql -h localhost -p 5432 -U
rest_catalog -d rest_catalog -c "UPDATE iceberg_tables SET iceberg_type=NULL"
```
**4. Try inserting again:**
```
>> spark.sql("insert into iceberg.test.test_table VALUES ('I', 'LOVE',
'ICEBERG')")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/spark350/python/pyspark/sql/connect/session.py", line 550, in
sql
data, properties = self.client.execute_command(cmd.command(self._client))
File "/opt/spark350/python/pyspark/sql/connect/client/core.py", line 982,
in execute_command
data, _, _, _, properties = self._execute_and_fetch(req)
File "/opt/spark350/python/pyspark/sql/connect/client/core.py", line 1282,
in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(req):
File "/opt/spark350/python/pyspark/sql/connect/client/core.py", line 1263,
in _execute_and_fetch_as_iterator
self._handle_error(error)
File "/opt/spark350/python/pyspark/sql/connect/client/core.py", line 1502,
in _handle_error
self._handle_rpc_error(error)
File "/opt/spark350/python/pyspark/sql/connect/client/core.py", line 1538,
in _handle_rpc_error
raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.SparkConnectGrpcException:
(org.apache.iceberg.exceptions.CommitFailedException) Commit failed: Failed to
update table test.test_table from catalog rest_backend
The same issue is observed in both Spark and Trino.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
