hcrosse commented on code in PR #2445:
URL: https://github.com/apache/iceberg-go/pull/2445#discussion_r4645793058
##########
table/equality_delete_reader.go:
##########
@@ -1657,6 +1657,97 @@ func (m *equalityDeleteMask) applySingleBytesDictionary(
return true
}
+const (
+ dictionaryEntryUnknown uint8 = iota
+ dictionaryEntryKept
+ dictionaryEntryDeleted
+)
+
+// dictionaryEntryMemo caches one delete set's results across a file's
+// batches while they share a dictionary object. Batches must be filtered one
+// at a time. Holding the dictionary without Retain still prevents its address
+// from being reused, and on a match the current batch keeps it alive. Reuse
+// relies on Arrow arrays being immutable.
+type dictionaryEntryMemo struct {
+ dict arrow.ArrayData
+ results []uint8
+}
+
+// apply looks up each dictionary entry that a row references at most once
+// per dictionary. It returns false, leaving m unchanged, for value types other
+// than string and binary.
+func (c *dictionaryEntryMemo) apply(
+ m *equalityDeleteMask,
+ column *array.Dictionary,
+ deleteSet *singleBytesEqualityDeleteSet,
+ mem memory.Allocator,
+) bool {
+ dict := column.Dictionary()
+ if data := column.Data().Dictionary(); c.dict != data {
Review Comment:
Good catch, thank you for the review and for running the benchmark locally.
Moved the memo reset behind the type switch in `dictionaryEntryMemo.apply` and
added a `FixedSizeBinary` test so unsupported dictionary types do not allocate
or clear `results`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]