ZENOTME commented on issue #1604:
URL: https://github.com/apache/iceberg-rust/issues/1604#issuecomment-3194148763
I draft the implementation of design above #1610, test on local machine(mac
m1 pro). Although the latency is high and not stable but we can see that the
performance of iceberg-rust is much closer to datafusion-parquet. I will build
a local test env to bench it next step.
```
====== full scan tpch_10.customer (1500000 rows, 1 files) =======
> before parallel scan
scan/iceberg-rust/tpch_10.customer
time: [10.834 s 11.197 s 11.593 s]
thrpt: [129.38 Kelem/s 133.97 Kelem/s 138.46
Kelem/s]
scan/datafusion-parquet/tpch_10.customer
time: [4.9707 s 5.1641 s 5.3602 s]
thrpt: [279.84 Kelem/s 290.46 Kelem/s 301.77
Kelem/s]
> after parallel scan
scan/iceberg-rust/tpch_10.customer
time: [6.4354 s 6.8607 s 7.3866 s]
thrpt: [203.07 Kelem/s 218.64 Kelem/s 233.09
Kelem/s]
scan/datafusion-parquet/tpch_10.customer
time: [5.1607 s 5.5025 s 5.9684 s]
thrpt: [251.33 Kelem/s 272.61 Kelem/s 290.66
Kelem/s]
```
```
====== full scan tpch_30.customer (4500000 rows, 3 files) =======
> before parallel scan
scan/iceberg-rust/tpch_30.customer
time: [15.670 s 18.931 s 24.221 s]
thrpt: [185.79 Kelem/s 237.71 Kelem/s 287.18
Kelem/s]
scan/datafusion-parquet/tpch_30.customer
time: [7.8278 s 8.3205 s 9.0664 s]
thrpt: [496.34 Kelem/s 540.83 Kelem/s 574.88
Kelem/s]
> after parallel scan
scan/iceberg-rust/tpch_30.customer
time: [8.2926 s 10.530 s 14.204 s]
thrpt: [316.80 Kelem/s 427.37 Kelem/s 542.65
Kelem/s]
scan/datafusion-parquet/tpch_30.customer
time: [7.9077 s 8.2183 s 8.5933 s]
thrpt: [523.67 Kelem/s 547.56 Kelem/s 569.07
Kelem/s]
```
```
====== full scan tpch_50.customer (7500000 rows, 5 files) =======
> before parallel scan
scan/iceberg-rust/tpch_50.customer
time: [15.612 s 19.579 s 26.554 s]
thrpt: [282.44 Kelem/s 383.06 Kelem/s 480.39
Kelem/s]
scan/datafusion-parquet/tpch_50.customer
time: [12.567 s 14.950 s 18.726 s]
thrpt: [400.50 Kelem/s 501.67 Kelem/s 596.80
Kelem/s]
> after parallel scan
scan/iceberg-rust/tpch_50.customer
time: [12.020 s 13.179 s 14.382 s]
thrpt: [521.48 Kelem/s 569.10 Kelem/s 623.94
Kelem/s]
scan/datafusion-parquet/tpch_50.customer
time: [12.035 s 21.058 s 38.281 s]
thrpt: [195.92 Kelem/s 356.16 Kelem/s 623.18
Kelem/s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
