hedger9487 opened a new pull request, #3867:
URL: https://github.com/apache/iceberg-python/pull/3867
Closes #3598
### Rationale & Motivation
During `Table.upsert()`, matched rows are compared cell-by-cell in pure
Python (`upsert_util.get_rows_to_update`) to determine whether any non-key
columns have changed, avoiding unnecessary rewrite IO.
However, for wide tables (e.g. 50–200 columns), comparing every single
non-key column across matched rows incurs substantial CPU overhead:
In production, the vast majority of matched rows are unchanged. Determining
that a row has not changed requires scanning all $M$ non-key columns ($O(N
\times M)$).
In workflows where users maintain dedicated change-tracking columns in their
tables (such as an upstream row hash, update timestamp, or version column), or
only care about changes to specific business fields, comparing every other
column is unnecessary.
This PR introduces the optional `difference_cols` parameter to allow users
to specify which non-key columns determine whether a matched row is considered
changed. Changes to columns outside `difference_cols` are intentionally ignored
for update detection.
---
### Key Invariants & Design
1. **Detection Authority with Full-Row Writes**:
`difference_cols` only restricts which columns are inspected to determine
*if* a matched row has changed. When a row is detected as changed, **all
columns of the source row are written**, preserving complete row data integrity.
2. **Fail-Fast Validation (`validate_difference_cols`)**:
- Defensively rejects empty lists (`difference_cols=[]`) with:
`"difference_cols must contain at least one column; use None to compare all
non-key columns"`.
- Rejects non-existent columns with deterministic error messages.
- Rejects columns that overlap with `join_cols`, since join columns are
guaranteed to match for matched rows and therefore cannot provide meaningful
change detection.
- Validated upfront in `Transaction.upsert` to fail fast before
performing storage or network operations.
3. **Backward Compatibility**:
- Defaults to `difference_cols=None`, which preserves the existing
change-detection behavior of checking all non-key columns.
- Existing callers that do not provide `difference_cols` require no
changes.
---
### Benchmark Results (10 Measured Trials)
Measured using in-memory Arrow tables isolated to `get_rows_to_update`.
*Note: The benchmark measures change-detection overhead only and excludes
catalog, filesystem, and Parquet I/O.*
- **Hardware / OS**: macOS (Apple Silicon ARM64)
- **Environment**: Python 3.14.7, PyArrow 25.0.1
- **Dataset**: 10,000 rows $\times$ 200 string columns (95% unchanged rows,
5% changed rows; 3 warmup iterations, 10 measured trials)
- **Control**: For the changed rows, `col_0` was modified so that the same
500 rows were identified as changed under both configurations.
| Metric | Baseline (All 200 columns) | With `difference_cols=["col_0"]` |
Improvement |
| :--- | :--- | :--- | :--- |
| **Execution Time (Mean)** | 34.1514s ($\pm 0.3602$s) | 0.6358s ($\pm
0.0027$s) | **53.71x faster** |
| **Execution Time (Median)** | 34.1264s | 0.6358s | **53.68x faster (98.1%
reduction)** |
| **Changed Rows Detected** | 500 rows | 500 rows | **Identical set
detected** |
---
### Proposed Changes
#### 1. `pyiceberg/table/upsert_util.py`
- Added `validate_difference_cols(column_names, join_cols,
difference_cols)`: validates non-key constraint, non-emptiness, and column
existence.
- Updated `get_rows_to_update`: uses `difference_cols` when provided;
otherwise falls back to existing `list(all_columns - join_cols_set)`.
#### 2. `pyiceberg/table/__init__.py`
- Added `difference_cols: list[str] | None = None` parameter and docstrings
to `Table.upsert` and `Transaction.upsert`.
- Added upfront `validate_difference_cols` before remote data scan execution.
#### 3. `tests/table/test_upsert.py`
Added 9 comprehensive unit and integration tests:
- `test_get_rows_to_update_with_difference_cols`: Verifies that only
`difference_cols` are used for change detection and that changed rows retain
all source columns.
- `test_get_rows_to_update_difference_cols_validation`: Verifies
`ValueError` on empty list, unknown column, and join key overlap.
- `test_get_rows_to_update_with_multiple_difference_cols`: Verifies
composite difference columns (`["col1", "col2"]`).
- `test_get_rows_to_update_difference_cols_with_nulls`: Verifies `None`
$\leftrightarrow$ value transitions.
- `test_get_rows_to_update_difference_cols_ignore_other_columns`: Verifies
that changes to non-difference columns are ignored for update detection (NOOP).
- `test_upsert_with_difference_cols`: End-to-end catalog upsert test
verifying rows skipped, updated, and inserted with actual Parquet files,
confirming full-row writes.
- `test_upsert_schema_evolution_with_difference_cols`: Verifies interaction
between table schema evolution and `difference_cols`.
- `test_upsert_with_invalid_difference_cols`: Verifies fail-fast rejection
via `table.upsert`.
- `test_upsert_transaction_with_difference_cols`: Verifies transaction
support.
---
### Verification Checklist
- [x] `uv run prek run -a` passes cleanly.
- [x] All tests in `tests/table/test_upsert.py` pass.
- [x] All tests in `tests/table/` pass locally.
- [x] Added 9 comprehensive unit and integration tests covering validation,
nulls, ignored columns, transactions, and schema evolution.
- [x] Rigorous statistical benchmark demonstrates 53.68x speedup on wide
tables.
- [x] Backward compatible by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
