mattfaltyn opened a new issue, #2287:
URL: https://github.com/apache/iceberg-go/issues/2287
### Apache Iceberg version
main (development), `a4fb4d0dafaeaf455990c1c3142462107662e5ec`; reproduced
locally with Go 1.26.3 on macOS arm64.
### Please describe the bug 🐞
`compaction.Analyze` passes `Scan.PlanFiles` results directly to
`Config.PlanCompaction`. Parquet planning can return several byte-range tasks
for one physical file, but the compaction planner counts and bin-packs each
task as a separate file, using the full file size for every range.
`ExecuteCompactionGroup` reads only the assigned range and then schedules
the whole source file for replacement. When partial progress puts ranges in
separate commit batches, the first batch can commit only part of the file's
rows. A later batch fails because the source file has already been removed from
the current snapshot. Returning that error does not undo the first durable
commit.
#### Reproduction
This needs no concurrent writer or external service:
1. Create a local, unpartitioned v2 table with one required `id: long`
column and property `read.split.target-size=1`.
2. Add one valid Parquet file containing IDs 1–12 in twelve one-row row
groups. The tiny split size keeps the reproduction small; `PlanFiles` returns
twelve ranges with the same data-file path.
3. Let `size` be that file's size and call `compaction.Analyze` with:
```go
cfg := compaction.Config{
TargetFileSizeBytes: size + 1,
MinFileSizeBytes: size + 1,
MaxFileSizeBytes: size * 2,
MinInputFiles: 1,
DeleteFileThreshold: 1,
PackingLookback: compaction.DefaultPackingLookback,
}
```
4. Convert each returned `compaction.Group` to `table.CompactionTaskGroup`,
preserving its tasks and size. The planner returns twelve one-task groups,
although there is only one physical file.
5. Call `txn.RewriteDataFiles(ctx, groups,
table.RewriteDataFilesOptions{PartialProgress: true, MaxCommits: 12})` without
injecting cancellation or a catalog failure.
6. Scan `result.Table`, the latest committed table returned with the error.
Observed twice, and repeated independently:
```text
rewritten groups=1
error=stage partial rewrite batch: cannot delete files that do not belong to
the table
rows after failed later group: 1 of 12
```
With the same rows and compaction settings but an unsplit file task,
compaction succeeds and retains all twelve rows. With the five-input threshold,
repeated source paths can instead fail duplicate-removal validation before
committing; this is not a claim that the default configuration always loses
data.
#### Expected behavior
Compaction must plan and replace complete physical files. Each committed
group must preserve all live source rows, including when later partial-progress
batches fail. Counts, sizes, and the minimum-input threshold should count a
source file once. Incomplete explicit worker groups should fail before
replacing a whole file with partial rows.
#### Relevant code
- [Analyze passes scan tasks to the
planner](https://github.com/apache/iceberg-go/blob/a4fb4d0dafaeaf455990c1c3142462107662e5ec/table/compaction/analyze.go#L33-L42)
- [Per-task packing with whole-file
sizes](https://github.com/apache/iceberg-go/blob/a4fb4d0dafaeaf455990c1c3142462107662e5ec/table/compaction/compaction.go#L245-L262)
- [Range
reads](https://github.com/apache/iceberg-go/blob/a4fb4d0dafaeaf455990c1c3142462107662e5ec/table/rewrite_data_files.go#L477-L480)
followed by [whole-file
removal](https://github.com/apache/iceberg-go/blob/a4fb4d0dafaeaf455990c1c3142462107662e5ec/table/rewrite_data_files.go#L528-L535)
The reproduction shows rows missing from the committed active snapshot, not
immediate physical deletion of the original file. An older retained snapshot
may permit recovery.
I checked existing issues and PRs, including #1939, #2041 and #2276; those
do not address this file/range-grain mismatch. I plan to work on a focused fix
with regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
