nmarasoiu opened a new issue, #13742:
URL: https://github.com/apache/iceberg/issues/13742
### Proposed Change
Summary
Apache Iceberg's current BigDecimal-based decimal handling creates
significant performance bottlenecks in decimal-heavy analytics workloads due to
excessive object allocation for operations that could be performed on primitive
representations.
Problem Description
Every decimal operation in Iceberg currently creates new BigDecimal objects,
even for simple comparisons and filtering operations. This results in:
3-4 object allocations per decimal value (BigDecimal + BigInteger + internal
arrays)
Memory overhead: 80-120 bytes per decimal vs 8-16 bytes for packed
representation
GC pressure: Millions of short-lived BigDecimal objects in filtering
operations
Non-vectorizable operations: Cannot leverage SIMD instructions
10-100x performance degradation for decimal comparison operations
Current Usage Analysis
Investigation of Iceberg's actual decimal usage reveals that the
mathematical capabilities of BigDecimal are largely unused:
What Iceberg Actually Does:
Serialization/deserialization between binary formats and BigDecimal
Filtering/comparisons using BigDecimal.compareTo() for predicate evaluation
Basic scale adjustments for literal conversions
Simple truncation operations for partitioning
What Iceberg Does NOT Do:
Arithmetic operations (add, subtract, multiply, divide) in hot paths
Complex mathematical calculations
Statistical aggregations on decimals in the core engine
Performance Impact
For a typical filtering operation:
java// Current approach - creates 3-4 objects per comparison
new BigDecimal(new BigInteger(bytes), scale).compareTo(literal)
// Optimal approach - zero allocations
comparePackedDecimals(bytes, scale, literalBytes, literalScale)
This affects every decimal predicate evaluation, literal comparison, and
partition filtering operation.
Proposed Solution
Implement a packed decimal representation for comparison operations:
Packed Storage: Store decimals as long unscaledValue + byte scale (9 bytes
total)
Zero-allocation Comparisons: Direct primitive comparison with scale
normalization
Backward Compatibility: Maintain BigDecimal API for mathematical operations
Incremental Adoption: Start with literal comparisons and filtering operations
Example Implementation Pattern:
javapublic static int compareDecimals(long unscaled1, int scale1, long
unscaled2, int scale2) {
if (scale1 == scale2) return Long.compare(unscaled1, unscaled2);
return normalizeAndCompare(unscaled1, scale1, unscaled2, scale2);
}
Benefits
10-100x faster decimal comparisons for typical analytics workloads
Massive reduction in GC pressure from eliminating object allocations
Better cache locality with primitive operations
SIMD compatibility for future vectorization
Maintains full precision for existing use cases
Implementation Strategy
Phase 1: Optimize literal comparisons in predicate evaluation
Phase 2: Extend to partition filtering and truncation operations
Phase 3: Consider vectorized operations for batch processing
Phase 4: Storage format optimizations
Compatibility
No breaking changes to existing APIs
Backward compatible with current BigDecimal usage
Internal optimization transparent to users
Maintains precision and scale semantics
References
Similar optimizations have been successfully implemented in:
PostgreSQL's NUMERIC type (base-10000 digits)
QuestDB's decimal handling
Apache Arrow's decimal256 operations
This optimization would significantly improve Iceberg's performance for
financial, scientific, and analytics workloads that heavily use decimal types.
Environment:
All versions of Apache Iceberg
Affects all query engines (Spark, Trino, Flink, etc.)
Most noticeable in decimal-heavy analytics workloads
### Proposal document
_No response_
### Specifications
- [ ] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]