kevinjqliu commented on issue #864: URL: https://github.com/apache/iceberg-python/issues/864#issuecomment-2240087023
Thanks for taking a look at this! I reran the steps above; the issue is actually with the `CREATE OR REPLACE TABLE` statement https://github.com/kevinjqliu/iceberg-rest-catalog/blob/main/tests/pyspark_test.py#L94-L108 I verified that all Pyspark statements work with the tabular REST catalog. https://github.com/kevinjqliu/iceberg-rest-catalog/blob/7c5548133ae266d4fac215b063911c35f08461d9/tests/pyspark_test.py#L75-L116 To reproduce, add this test to `test_rest_manifest.py` and running `pytest tests/integration/test_rest_manifest.py::test_write_round_trip -v -m integration` ``` from pyspark.sql import SparkSession @pytest.mark.integration def test_write_round_trip(catalog: Catalog, spark: SparkSession) -> None: from pyiceberg.exceptions import NoSuchTableError from pyiceberg.schema import Schema from pyiceberg.types import NestedField, UUIDType, FixedType schema = Schema( NestedField(field_id=1, name="uuid_col", field_type=UUIDType(), required=False), NestedField(field_id=2, name="fixed_col", field_type=FixedType(25), required=False), ) identifier = "default.test_table_write_round_trip" try: catalog.drop_table(identifier) except NoSuchTableError: pass catalog.create_table( identifier=identifier, schema=schema ) # write to table with spark sql spark.sql( f""" INSERT INTO integration.{identifier} VALUES ('102cb62f-e6f8-4eb0-9973-d9b012ff0967', CAST('1234567890123456789012345' AS BINARY)), ('ec33e4b2-a834-4cc3-8c4a-a1d3bfc2f226', CAST('1231231231231231231231231' AS BINARY)), ('639cccce-c9d2-494a-a78c-278ab234f024', CAST('12345678901234567ass12345' AS BINARY)), ('c1b0d8e0-0b0e-4b1e-9b0a-0e0b0d0c0a0b', CAST('asdasasdads12312312312111' AS BINARY)), ('923dae77-83d6-47cd-b4b0-d383e64ee57e', CAST('qweeqwwqq1231231231231111' AS BINARY)); """ ) tbl = catalog.load_table(identifier) assert tbl.schema() == schema df = tbl.scan().to_arrow().to_pandas() assert len(df) == 5 assert b"1234567890123456789012345" in df["fixed_col"].to_list() # create a table with spark sql spark.sql( f""" CREATE OR REPLACE TABLE integration.{identifier} USING iceberg AS SELECT 1 AS idx, float('NaN') AS col_numeric UNION ALL SELECT 2 AS idx, null AS col_numeric UNION ALL SELECT 3 AS idx, 1 AS col_numeric """ ) # write to table with spark sql spark.sql( f""" INSERT INTO integration.{identifier} VALUES (4, 999); """ ) ``` > Could we double check if this is an issue in the Rest catalog implementation? The issue **is** with this specific Rest catalog implementation. However, the implementation uses the Pyiceberg library to process catalog requests. `CREATE OR REPLACE TABLE` triggers an `update_table` request to the catalog. And the catalog is implemented by proxying the request directly to the underlying pyiceberg function https://github.com/kevinjqliu/iceberg-rest-catalog/blob/7c5548133ae266d4fac215b063911c35f08461d9/src/iceberg_rest/api/catalog_api.py#L453 I believe the issue is with the `_commit_table` function and how it deals with the incoming `CommitTableRequest` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
