varun-lakhyani commented on PR #16729: URL: https://github.com/apache/iceberg/pull/16729#issuecomment-4825489924
### Benchmark Setup (JMH) **Machine:** AWS EC2 (same region as the S3 bucket to minimize network latency) | Property | Value | |----------|-------| | Instance type | r5.4xlarge | | vCPUs | 16 | | Memory | 128 GB | | Network | Up to 10 Gbps | | Storage | 50 GB gp3 EBS | | AMI | Amazon Linux 2023 | | Region | ap-south-1 (Mumbai) | **Benchmark:** [IcebergDataCompactionBenchmark](https://github.com/apache/iceberg/blob/839b22647ce5aa08e27220bbe85cde1292129fea/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergDataCompactionBenchmark.java#L57) extended with `aalEnabled` and `eagerThreshold` params for this comparison - Warmup iterations: **5** - Measurement iterations: **10** - Total rows: **20,000,000 (20 million)** ### Results | Number of Files | Default (s) | AAL = true (s) | EagerInputFile = true (s) | EagerInputFile Improvement vs Default | EagerInputFile Improvement vs AAL | |----------------:|------------:|---------------:|--------------------------:|-----------------------:|-------------------:| | 250 | 45.104 | 29.671 | 27.311 | **39.45%** | **7.95%** | | 500 | 77.788 | 54.030 | 46.553 | **40.15%** | **13.84%** | | 1000 | 163.238 | 107.138 | 95.429 | **41.54%** | **10.93%** | | 2000 | 312.252 | 195.158 | 179.351 | **42.56%** | **8.10%** | #### Graphical Comparison <img width="2970" height="1769" alt="iceberg_compaction_runtime_axis" src="https://github.com/user-attachments/assets/469dedde-4693-4620-861b-b92e770764d5"; /> ### S3 Request Analysis The number of GET/HEAD requests observed from the S3 bucket logs is computed as: > `Total Requests / (Total Iterations × Number of Files)` Configuration | GET/file | HEAD/file | Total Calls/file -- | -- | -- | -- Default | 3 | 0 | 3 AAL = true | 1 | 1 | 2 EagerInputFile = true | 1 | 0 | 1 <p></p> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
