dataengineeratwork opened a new issue, #17037:
URL: https://github.com/apache/iceberg/issues/17037
### Apache Iceberg version
1.4.0
### Query engine
Spark
### Please describe the bug 🐞
We've been using iceberg since long now on AWS, but came across the
following issue recently in our Production environment.
With Upsert query using MERGE INTO stetement, we observed for one of the
iceberg table (non-partitioned), while rewriting the data files, it has removed
some of the records. We have checked with the time-travel and the records were
present earlier, but in one the of rewrite history, the records got removed
from the table.
The table here is in silver layer, where we're merging data from different
data sources into single table, to identify the records, we've "datasource_id"
column present along with some system columns like created_on_dt,
updated_on_dt, w_inserted_dt, w_updated_dt, etl_run_id etc.
We've never observed any such issue with Partitioned tables though. Has
anyone came across similar issue, I want to understand why iceberg is
removing/deleting the unchanged records?
Following is the merge query we use :
```
upsert_sql = f"""
MERGE INTO {table} AS target
USING input_data AS source
ON concat(coalesce(target.integration_id, 'UNKNOWN'),
coalesce(target.datasource_id, 'DFLT')) =
concat(coalesce(source.integration_id, 'UNKNOWN'),
coalesce(source.datasource_id, 'DFLT'))
WHEN MATCHED THEN
UPDATE SET {', '.join(f"target.{col} = source.{col}" for
col in input_df.columns if col not in exclude_columns)}
WHEN NOT MATCHED THEN
INSERT ({', '.join(input_df.columns)})
VALUES ({', '.join(f"source.{col}" for col in
input_df.columns)})
"""
# Execute the upsert SQL statement
spark.sql(upsert_sql)
cleaned_df = cleaned_df.dropDuplicates(["integration_id"])
upsert_data_iceberg(spark, cleaned_df, sl_athena_output_db_name,
iceberg_table_name, s3_silver_output_location, logger)
```
Before snapshot:
snapshot-id = 5244992060615715741
total-records = 99,375,307
total-data-files = 936
Then the next snapshot:
snapshot-id = 7953544942377759370
added-records = 49,077,409
deleted-records = 55,719,302
added-data-files = 4
deleted-data-files = 38
total-records = 92,733,414
Net loss:
99,375,307
-92,733,414
------------
6,641,893 rows
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
