nmarasoiu opened a new issue, #13742:
URL: https://github.com/apache/iceberg/issues/13742

   ### Proposed Change
   
   Summary
   Apache Iceberg's current BigDecimal-based decimal handling creates 
significant performance bottlenecks in decimal-heavy analytics workloads due to 
excessive object allocation for operations that could be performed on primitive 
representations.
   Problem Description
   Every decimal operation in Iceberg currently creates new BigDecimal objects, 
even for simple comparisons and filtering operations. This results in:
   
   3-4 object allocations per decimal value (BigDecimal + BigInteger + internal 
arrays)
   Memory overhead: 80-120 bytes per decimal vs 8-16 bytes for packed 
representation
   GC pressure: Millions of short-lived BigDecimal objects in filtering 
operations
   Non-vectorizable operations: Cannot leverage SIMD instructions
   10-100x performance degradation for decimal comparison operations
   
   Current Usage Analysis
   Investigation of Iceberg's actual decimal usage reveals that the 
mathematical capabilities of BigDecimal are largely unused:
   What Iceberg Actually Does:
   
   Serialization/deserialization between binary formats and BigDecimal
   Filtering/comparisons using BigDecimal.compareTo() for predicate evaluation
   Basic scale adjustments for literal conversions
   Simple truncation operations for partitioning
   
   What Iceberg Does NOT Do:
   
   Arithmetic operations (add, subtract, multiply, divide) in hot paths
   Complex mathematical calculations
   Statistical aggregations on decimals in the core engine
   
   Performance Impact
   For a typical filtering operation:
   java// Current approach - creates 3-4 objects per comparison
   new BigDecimal(new BigInteger(bytes), scale).compareTo(literal)
   
   // Optimal approach - zero allocations
   comparePackedDecimals(bytes, scale, literalBytes, literalScale)
   This affects every decimal predicate evaluation, literal comparison, and 
partition filtering operation.
   Proposed Solution
   Implement a packed decimal representation for comparison operations:
   
   Packed Storage: Store decimals as long unscaledValue + byte scale (9 bytes 
total)
   Zero-allocation Comparisons: Direct primitive comparison with scale 
normalization
   Backward Compatibility: Maintain BigDecimal API for mathematical operations
   Incremental Adoption: Start with literal comparisons and filtering operations
   
   Example Implementation Pattern:
   javapublic static int compareDecimals(long unscaled1, int scale1, long 
unscaled2, int scale2) {
       if (scale1 == scale2) return Long.compare(unscaled1, unscaled2);
       return normalizeAndCompare(unscaled1, scale1, unscaled2, scale2);
   }
   Benefits
   
   10-100x faster decimal comparisons for typical analytics workloads
   Massive reduction in GC pressure from eliminating object allocations
   Better cache locality with primitive operations
   SIMD compatibility for future vectorization
   Maintains full precision for existing use cases
   
   Implementation Strategy
   
   Phase 1: Optimize literal comparisons in predicate evaluation
   Phase 2: Extend to partition filtering and truncation operations
   Phase 3: Consider vectorized operations for batch processing
   Phase 4: Storage format optimizations
   
   Compatibility
   
   No breaking changes to existing APIs
   Backward compatible with current BigDecimal usage
   Internal optimization transparent to users
   Maintains precision and scale semantics
   
   References
   Similar optimizations have been successfully implemented in:
   
   PostgreSQL's NUMERIC type (base-10000 digits)
   QuestDB's decimal handling
   Apache Arrow's decimal256 operations
   
   This optimization would significantly improve Iceberg's performance for 
financial, scientific, and analytics workloads that heavily use decimal types.
   
   Environment:
   
   All versions of Apache Iceberg
   Affects all query engines (Spark, Trino, Flink, etc.)
   Most noticeable in decimal-heavy analytics workloads
   
   
   ### Proposal document
   
   _No response_
   
   ### Specifications
   
   - [ ] Table
   - [ ] View
   - [ ] REST
   - [ ] Puffin
   - [ ] Encryption
   - [ ] Other


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to