zeroshade commented on code in PR #1769:
URL: https://github.com/apache/iceberg-go/pull/1769#discussion_r3831551625
##########
table/partitioned_fanout_writer.go:
##########
@@ -486,16 +487,35 @@ func (n *partitionMapNode) getOrCreate(partitionRec
partitionRecord, fieldInfo [
}
partVal = &partitionInfo{
- rows: make([]int64, 0, 128), // modest starting
capacity
+ rows: make([]int64, 0,
initialPartitionRowCapacity(numRows, n.partitionCount)),
Review Comment:
This estimate depends on partition discovery order and can land just below
Go's slice-growth thresholds. In a 32,768-row probe where 300 partitions are
discovered first and partitions 256–299 later grow to roughly 110–128 rows,
this branch increased usage from 1,369,248 to 1,455,136 B/op (+6.3%) and from
11,950 to 11,994 allocs/op versus the merge base. The included skew benchmark
only exercises the favorable hot-partition-first ordering. Could we quantize
these capacities to growth-safe boundaries, or otherwise prevent sub-128 slices
from growing beyond the previous 128 allocation? Please also add a
late-discovered/skewed benchmark covering this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
