ShivamSharma43 opened a new issue, #2489:
URL: https://github.com/apache/iceberg-go/issues/2489
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
When the parent snapshot's summary doesn't have a `total-*` property, or has
one that doesn't parse as a number, iceberg-go treats the previous total as 0
and writes a new total anyway. The new snapshot then reports a total that is
far too small. For example, take a table with thousands of data files whose
current snapshot summary has only `operation`, or only some of the totals
(written by another engine). After appending one file with iceberg-go, the new
snapshot says `total-data-files=1`, `total-records=<rows in that file>`, and so
on. Later commits build on those wrong numbers.
Reproduced on main (a929eec) with the internal helper that computes the
totals:
```go
result, _ := updateSnapshotSummaries(Summary{Operation: OpAppend,
Properties: iceberg.Properties{
"added-data-files": "1",
"added-records": "5",
"added-files-size": "100",
}}, iceberg.Properties{
// The parent only tracked the record total; the other totals are
unknown.
"total-records": "1000",
})
fmt.Println(result.Properties)
```
Output:
```
map[added-data-files:1 added-files-size:100 added-records:5
total-data-files:1 total-delete-files:0 total-equality-deletes:0
total-files-size:100 total-position-deletes:0 total-records:1005]
```
Expected: only `total-records=1005`. The other five totals are unknown, so
they should be left out. An unparseable previous total (for example
`"total-records": "not-a-number"`) is also turned into a new total that starts
from 0.
**Cause:** `updateTotals` in `updateSnapshotSummaries`
(`table/snapshots.go`) reads the previous total with
`previous.GetInt64(totalProp, 0)`. That returns 0 for a missing or unparseable
value, so a total is always written.
**Java:** `SnapshotProducer.updateTotal` writes a total only if the previous
total is present, parses, and is non-negative ("ignore and do not add total"
otherwise):
https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L938-L970
PyIceberg's `update_snapshot_summaries` also writes a total only if the
previous one is present:
https://github.com/apache/iceberg-python/blob/718c7908b36117974257e3d214959bffcada14a1/pyiceberg/table/snapshots.py#L385-L397
**Proposed fix:** in `updateTotals`, skip a total when the previous value is
missing, unparseable or negative, as Java does. A table with no parent snapshot
(`previous == nil`) still starts every total at 0.
I'd like to work on this. I have a fix and a unit test ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
