aik-jahoda opened a new issue, #96:
URL: https://github.com/apache/arrow-dotnet/issues/96
### Describe the enhancement requested
The current implementation of `DecimalXArray.Builder` shows significantly
worse performance compared to other array builders in arrow-dotnet, primarily
due to BigInteger arithmetic overhead.
| Method | Mean | Error | StdDev | Gen0 |
Gen1 | Gen2 | Allocated |
|-------------------------
|------------:|----------:|----------:|-------:|-------:|-------:|----------:|
| Int32ArrayBuilder_Append | 317.7 ns | 8.32 ns | 24.27 ns | 0.0234 |
- | - | 296 B |
| Int64ArrayBuilder | 318.1 ns | 9.22 ns | 26.74 ns | 0.0310 |
- | - | 392 B |
| StringArrayBuilder | 1,620.8 ns | 32.50 ns | 84.47 ns | 0.1850 |
0.1717 | 0.0916 | 1928 B |
| Decimal32ArrayBuilder | 3,514.0 ns | 68.52 ns | 135.26 ns | 0.0458 |
0.0420 | 0.0153 | 528 B |
| Decimal64ArrayBuilder | 4,400.3 ns | 87.73 ns | 208.50 ns | 0.0992 |
- | - | 1256 B |
| Decimal128ArrayBuilder | 5,949.8 ns | 118.44 ns | 216.58 ns | 0.2289 |
0.1068 | 0.0534 | 2456 B |
| Decimal256ArrayBuilder | 10,367.5 ns | 245.93 ns | 713.47 ns | 0.2747 |
0.1526 | 0.1068 | 3184 B |
The results are based on the following benchmark:
https://github.com/aik-jahoda/arrow-dotnet/commit/a2cf2a6d0ca65f029fe80c61f0e70e4218178292
###Root cause
The main performance bottleneck comes from reliance on BigInteger
arithmetic, which is much slower than using native integer math.
Proposed Directions
I’d like to open a discussion on which approach would best align with
arrow-dotnet’s goals and .NET ecosystem support:
### 1. Pre-allocate `BigInteger` powers of 10
Pros:
- Straightforward to implement
- Provides performance gains for all decimal variants across all .NET
versions
Cons:
- Only partial improvement, may still lag compared to native integer
arithmetic
### 2. Use native integral types (`Int128` in .net 7+, `Int64` in older)
Pros:
- High performance for `Decimal32`, `Decimal64` for all .net versions and
`Decimal128` for .net 7+
Cons:
- Performance inconsistency across .NET versions (on pre-.net 7
`Decimal128` must fall back to `BigInteger`)
- `Decimal256` still requires `BigInteger` on all .net versions
### 3. Create/import custom `Int128`/`Int256` types
Pros:
- Consistent performance across all decimal variants
Cons:
- Significant complexity in implementation and maintenance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
