tschwarzinger commented on issue #26169:
URL: https://github.com/apache/datafusion/issues/26169#issuecomment-6113886480
Ok I've started my PC before saying something wrong.
Query:
```sql
SELECT
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice,
sum(l_quantity)
FROM
customer,
orders,
lineitem
WHERE
o_orderkey IN (
SELECT
l_orderkey
FROM
lineitem
GROUP BY
l_orderkey
HAVING
sum(l_quantity) > 300)
AND c_custkey = o_custkey
AND o_orderkey = l_orderkey
GROUP BY
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice
ORDER BY
o_totalprice DESC,
o_orderdate
LIMIT 100;
```
Polars (streaming):
```text
SORT BY [slice: (0, 100), descending: [true, false], nulls_last: [false,
true]] [col("o_totalprice"), col("o_orderdate")]
AGGREGATE[maintain_order: false]
[col("l_quantity").sum(null_on_empty=true)] BY [col("c_name"),
col("c_custkey"), col("o_orderkey"), col("o_orderdate"), col("o_totalprice")]
FROM
simple π 6/6 ["c_name", "c_custkey", ... 4 other columns]
INNER JOIN:
BUILD SIDE: PreferLeft
LEFT PLAN ON: [col("o_orderkey")]
INNER JOIN:
BUILD SIDE: PreferRight
LEFT PLAN ON: [col("c_custkey")]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet]
PROJECT 2/8 COLUMNS
SELECTION: col("c_custkey").dynamic_predicate()
ESTIMATED ROWS: 1500000
RIGHT PLAN ON: [col("o_custkey")]
simple π 4/4 ["o_orderkey", "o_totalprice", ... 2 other columns]
SEMI JOIN:
BUILD SIDE: PreferRight
LEFT PLAN ON: [col("o_orderkey")]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet]
PROJECT 4/9 COLUMNS
SELECTION: col("o_orderkey").dynamic_predicate()
ESTIMATED ROWS: 15000000
RIGHT PLAN ON: [col("l_orderkey")]
simple π 1/1 ["l_orderkey"]
FILTER col("__POLARS_HAVING_0")
FROM
AGGREGATE[maintain_order: false]
[(col("l_quantity").sum(null_on_empty=true) >
300).alias("__POLARS_HAVING_0")] BY [col("l_orderkey")]
FROM
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet]
PROJECT 2/16 COLUMNS
ESTIMATED ROWS: 59986052
END SEMI JOIN
END INNER JOIN
RIGHT PLAN ON: [col("l_orderkey")]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet]
PROJECT 2/16 COLUMNS
SELECTION: col("l_orderkey").dynamic_predicate()
ESTIMATED ROWS: 59986052
END INNER JOIN
```
Polars (in-memory):
```text
SORT BY [slice: (0, 100), descending: [true, false], nulls_last: [false,
true]] [col("o_totalprice"), col("o_orderdate")]
AGGREGATE[maintain_order: false]
[col("l_quantity").sum(null_on_empty=true)] BY [col("c_name"),
col("c_custkey"), col("o_orderkey"), col("o_orderdate"), col("o_totalprice")]
FROM
simple π 6/6 ["c_name", "c_custkey", ... 4 other columns]
INNER JOIN:
LEFT PLAN ON: [col("o_orderkey")]
INNER JOIN:
BUILD SIDE: PreferLeft
LEFT PLAN ON: [col("c_custkey")]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet]
PROJECT 2/8 COLUMNS
ESTIMATED ROWS: 1500000
RIGHT PLAN ON: [col("o_custkey")]
simple π 4/4 ["o_orderkey", "o_totalprice", ... 2 other columns]
SEMI JOIN:
LEFT PLAN ON: [col("o_orderkey")]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet]
PROJECT 4/9 COLUMNS
ESTIMATED ROWS: 15000000
RIGHT PLAN ON: [col("l_orderkey")]
simple π 1/1 ["l_orderkey"]
FILTER col("__POLARS_HAVING_0")
FROM
AGGREGATE[maintain_order: false]
[(col("l_quantity").sum(null_on_empty=true) >
300).alias("__POLARS_HAVING_0")] BY [col("l_orderkey")]
FROM
CACHE[id: d50bd95e-fd24-4dc5-abdb-26543a5d8a6f]
Parquet SCAN
[/home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet]
PROJECT 2/16 COLUMNS
ESTIMATED ROWS: 59986052
END SEMI JOIN
END INNER JOIN
RIGHT PLAN ON: [col("l_orderkey")]
simple π 2/2 ["l_quantity", "l_orderkey"]
CACHE[id: d50bd95e-fd24-4dc5-abdb-26543a5d8a6f]
END INNER JOIN
```
DataFusion:
```text
SortPreservingMergeExec: [o_totalprice@4 DESC, o_orderdate@3 ASC NULLS
LAST], fetch=100, metrics=[output_rows=100, elapsed_compute=44.56µs,
output_bytes=20.1 KB, output_batches=1]
SortExec: TopK(fetch=100), expr=[o_totalprice@4 DESC, o_orderdate@3 ASC
NULLS LAST], preserve_partitioning=[true], metrics=[output_rows=624,
elapsed_compute=1.14ms, output_bytes=56.1 KB, output_batches=24,
row_replacements=624]
AggregateExec: mode=SinglePartitioned, gby=[c_name@1 as c_name,
c_custkey@0 as c_custkey, o_orderkey@2 as o_orderkey, o_orderdate@4 as
o_orderdate, o_totalprice@3 as o_totalprice], aggr=[sum(lineitem.l_quantity)],
metrics=[output_rows=624, elapsed_compute=1.97ms, output_bytes=66.6 KB,
output_batches=24, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0,
peak_mem_used=267.9 K, aggregate_arguments_time=19.82µs,
aggregation_time=82.10µs, emitting_time=160.07µs,
time_calculating_group_ids=1.24ms]
HashJoinExec: mode=Partitioned, join_type=RightSemi,
on=[(l_orderkey@0, o_orderkey@2)], metrics=[output_rows=4.37 K,
elapsed_compute=13.96s, output_bytes=12.9 MB, output_batches=24,
array_map_created_count=0, build_input_batches=24, build_input_rows=624,
input_batches=7.33 K, input_rows=59.99 M, build_mem_used=1.59 M,
build_time=6.96s, join_time=364.55ms, avg_fanout=100% (4.37 K/4.37 K),
probe_hit_rate=0.01% (4.37 K/59.99 M)]
FilterExec: sum(lineitem.l_quantity)@1 > Some(30000),25,2,
projection=[l_orderkey@0], metrics=[output_rows=624, elapsed_compute=231.40ms,
output_bytes=1536.0 KB, output_batches=24, selectivity=0% (624/15.00 M)]
AggregateExec: mode=FinalPartitioned, gby=[l_orderkey@0 as
l_orderkey], aggr=[sum(lineitem.l_quantity)], metrics=[output_rows=15.00 M,
elapsed_compute=6.63s, output_bytes=43.3 GB, output_batches=1.85 K,
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=1.91 B,
aggregate_arguments_time=2.75ms, aggregation_time=1.41s, emitting_time=15.80ms,
time_calculating_group_ids=5.19s]
RepartitionExec: partitioning=Hash([l_orderkey@0], 24),
input_partitions=24, metrics=[output_rows=15.00 M, elapsed_compute=4.40ms,
output_bytes=346.5 MB, output_batches=1.85 K, spill_count=0, spilled_bytes=0.0
B, spilled_rows=0, fetch_time=10.85s, repartition_time=731.14ms,
send_time=2.64s]
AggregateExec: mode=Partial, gby=[l_orderkey@0 as l_orderkey],
aggr=[sum(lineitem.l_quantity)], metrics=[output_rows=15.00 M,
elapsed_compute=8.69s, output_bytes=43.1 GB, output_batches=1.84 K,
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0,
peak_mem_used=958.3 M, aggregate_arguments_time=5.34ms, aggregation_time=1.27s,
emitting_time=6.26ms, time_calculating_group_ids=7.39s, reduction_factor=25.01%
(15.00 M/59.99 M)]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:0..106115546],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:106115546..212231092],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:212231092..318346638],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:318346638..424462184],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:424462184..530577730],
...]}, projection=[l_orderkey, l_quantity], file_type=parquet,
metrics=[output_rows=59.99 M, elapsed_compute=15.39ms, output_bytes=1382.7 MB,
output_batches=7.48 K, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=524 total → 524 matched,
row_groups_pruned_bloom_filter=524 total → 524 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched
, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=205.8 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=22.21µs, metadata_load_time=2.79ms,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=40.45ms,
time_elapsed_processing=1.45s, time_elapsed_scanning_total=10.77s,
time_elapsed_scanning_until_data=147.24ms, output_rows_skew=0.03%,
scan_efficiency_ratio=8.08% (205.8 M/2.55 B)]
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@2,
l_orderkey@0)], projection=[c_custkey@0, c_name@1, o_orderkey@2,
o_totalprice@3, o_orderdate@4, l_quantity@6], metrics=[output_rows=59.99 M,
elapsed_compute=15.45s, output_bytes=540.0 GB, output_batches=7.33 K,
array_map_created_count=0, build_input_batches=1.85 K, build_input_rows=15.00
M, input_batches=7.33 K, input_rows=59.99 M, build_mem_used=3.10 B,
build_time=5.63s, join_time=9.71s, avg_fanout=100% (59.99 M/59.99 M),
probe_hit_rate=100% (59.99 M/59.99 M)]
RepartitionExec: partitioning=Hash([o_orderkey@2], 24),
input_partitions=24, metrics=[output_rows=15.00 M, elapsed_compute=5.00ms,
output_bytes=2.5 GB, output_batches=1.85 K, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=7.18s, repartition_time=1.00s, send_time=7.73s]
HashJoinExec: mode=Partitioned, join_type=Inner,
on=[(c_custkey@0, o_custkey@1)], projection=[c_custkey@0, c_name@1,
o_orderkey@2, o_totalprice@4, o_orderdate@5], metrics=[output_rows=15.00 M,
elapsed_compute=4.60s, output_bytes=11.5 GB, output_batches=1.84 K,
array_map_created_count=0, build_input_batches=192, build_input_rows=1.50 M,
input_batches=1.84 K, input_rows=15.00 M, build_mem_used=242.0 M,
build_time=195.65ms, join_time=4.39s, avg_fanout=100% (15.00 M/15.00 M),
probe_hit_rate=100% (15.00 M/15.00 M)]
RepartitionExec: partitioning=Hash([c_custkey@0], 24),
input_partitions=24, metrics=[output_rows=1.50 M, elapsed_compute=421.35µs,
output_bytes=179.8 MB, output_batches=192, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=301.66ms, repartition_time=31.25ms,
send_time=232.35ms]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:0..5780685],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:5780685..11561370],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:11561370..17342055],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:17342055..23122740],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:23122740..28903425],
...]}, projection=[c_custkey, c_name], file_type=parquet,
metrics=[output_rows=1.50 M, elapsed_compute=723.00µs, output_bytes=228.2 MB,
output_batches=210, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=35 total → 35 matched,
row_groups_pruned_bloom_filter=35 total → 35 matched, page_index_pages_pruned=0
total → 0 matched, page_index_rows_pruned=0 total → 0 matched,
limit_pruned_row_grou
ps=0 total → 0 matched, batches_split=0, bytes_scanned=19.30 M,
file_open_errors=0, file_scan_errors=0, files_opened=24, files_processed=24,
num_predicate_creation_errors=0, predicate_evaluation_errors=0,
pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=1.64µs, metadata_load_time=845.94µs,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=9.64ms,
time_elapsed_processing=147.08ms, time_elapsed_scanning_total=560.13ms,
time_elapsed_scanning_until_data=228.43ms, output_rows_skew=10.91%,
scan_efficiency_ratio=13.91% (19.30 M/138.7 M)]
RepartitionExec: partitioning=Hash([o_custkey@1], 24),
input_partitions=24, metrics=[output_rows=15.00 M, elapsed_compute=6.10ms,
output_bytes=518.9 MB, output_batches=1.84 K, spill_count=0, spilled_bytes=0.0
B, spilled_rows=0, fetch_time=2.81s, repartition_time=426.87ms, send_time=4.97s]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:0..28214315],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:28214315..56428630],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:56428630..84642945],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:84642945..112857260],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:112857260..141071575],
...]}, projection=[o_orderkey, o_custkey, o_totalprice, o_orderdate],
file_type=parquet, predicate=DynamicFilter [ CASE hash_repartition % 24 WHEN 0
THEN o_custkey@1 >= 20 AND o_custkey@1 <= 1499995 AND hash_lookup WHEN 1 THEN
o_custkey@1 >= 1 AND o_custkey@1 <= 1499982 AND hash_lookup WHEN 2 THEN
o_custkey@1 >= 13 AND o_custkey@1 <= 1499952 AND hash_lookup WHEN 3 THEN
o_custkey@1 >= 9 AND o_custkey@1 <= 1499974 AND hash_lookup WHEN 4 THE
N o_custkey@1 >= 2 AND o_custkey@1 <= 1499994 AND hash_lookup WHEN 5 THEN
o_custkey@1 >= 6 AND o_custkey@1 <= 1499998 AND hash_lookup WHEN 6 THEN
o_custkey@1 >= 12 AND o_custkey@1 <= 1499986 AND hash_lookup WHEN 7 THEN
o_custkey@1 >= 11 AND o_custkey@1 <= 1499990 AND hash_lookup WHEN 8 THEN
o_custkey@1 >= 5 AND o_custkey@1 <= 1499993 AND hash_lookup WHEN 9 THEN
o_custkey@1 >= 40 AND o_custkey@1 <= 1499950 AND hash_lookup WHEN 10 THEN
o_custkey@1 >= 10 AND o_custkey@1 <= 1499987 AND hash_lookup WHEN 11 THEN
o_custkey@1 >= 41 AND o_custkey@1 <= 1499988 AND hash_lookup WHEN 12 THEN
o_custkey@1 >= 53 AND o_custkey@1 <= 1499977 AND hash_lookup WHEN 13 THEN
o_custkey@1 >= 24 AND o_custkey@1 <= 1499996 AND hash_lookup WHEN 14 THEN
o_custkey@1 >= 26 AND o_custkey@1 <= 1499922 AND hash_lookup WHEN 15 THEN
o_custkey@1 >= 43 AND o_custkey@1 <= 1499997 AND hash_lookup WHEN 16 THEN
o_custkey@1 >= 39 AND o_custkey@1 <= 1499963 AND hash_lookup WHEN 17 THEN
o_custkey@1 >= 3 AND o_custkey@1 <= 14999
80 AND hash_lookup WHEN 18 THEN o_custkey@1 >= 77 AND o_custkey@1 <= 1499984
AND hash_lookup WHEN 19 THEN o_custkey@1 >= 16 AND o_custkey@1 <= 1499999 AND
hash_lookup WHEN 20 THEN o_custkey@1 >= 4 AND o_custkey@1 <= 1499979 AND
hash_lookup WHEN 21 THEN o_custkey@1 >= 17 AND o_custkey@1 <= 1499964 AND
hash_lookup WHEN 22 THEN o_custkey@1 >= 15 AND o_custkey@1 <= 1500000 AND
hash_lookup WHEN 23 THEN o_custkey@1 >= 30 AND o_custkey@1 <= 1499983 AND
hash_lookup ELSE false END ] AND DynamicFilter [ CASE hash_repartition % 24
WHEN 0 THEN o_orderkey@0 >= 1544643 AND o_orderkey@0 <= 59237092 AND
o_orderkey@0 IN (SET) ([48314274, 52826116, 53407847, 37683008, 46837987,
15738658, 40753383, 28077922, 57974274, 10442629, 8506019, 20662370, 50839616,
3043270, 11028230, 59237092, 1544643, 56089923, 21057986, 54796358, 49172291,
34343042, 30008928, 44883872, 19730208, 19730468, 37531878]) WHEN 1 THEN
o_orderkey@0 >= 2366755 AND o_orderkey@0 <= 57137254 AND o_orderkey@0 IN (SET)
([46486080, 5293568
5, 18353314, 32991271, 8075589, 31018979, 5832321, 20714883, 41064486,
38093280, 41325120, 51929955, 16801991, 13432994, 17016006, 38488484, 2366755,
39492870, 12251526, 21973952, 22470437, 57137254, 24690087, 24923202,
40021697]) WHEN 2 THEN o_orderkey@0 >= 6651397 AND o_orderkey@0 <= 59138305 AND
o_orderkey@0 IN (SET) ([46678469, 18152960, 30567815, 18367712, 55342658,
58547488, 59138305, 15960071, 9727175, 16020773, 18762278, 23660263, 23711776,
32161635, 33899427, 56516384, 34590402, 42262528, 6651397, 45053063, 12378086])
WHEN 3 THEN o_orderkey@0 >= 1474818 AND o_orderkey@0 <= 56741287 AND
o_orderkey@0 IN (SET) ([53395493, 30702500, 18449123, 55807047, 37828134,
8590657, 41274690, 31587747, 51155008, 51409728, 53623108, 1474818, 54375559,
38643041, 21213895, 54915267, 4527553, 7125602, 36667107, 7456839, 56741287,
22040514, 24378656, 22291718, 37023521]) WHEN 4 THEN o_orderkey@0 >= 2329187
AND o_orderkey@0 <= 53938401 AND o_orderkey@0 IN (SET) ([30864451, 50277955,
10805828, 47
071686, 37874913, 31450375, 31601378, 51532000, 31855814, 53938401, 28916832,
16783360, 44143908, 17272195, 21296295, 29805379, 2329187, 42146816, 44800711,
39949734, 12563331]) WHEN 5 THEN o_orderkey@0 >= 4259524 AND o_orderkey@0 <=
57743845 AND o_orderkey@0 IN (SET) ([17927718, 57743845, 37775877, 51063047,
9615783, 28584225, 52032003, 29294434, 21312615, 38926278, 44331334, 4259524,
12099683, 24319555, 57173283, 49619845, 49704768]) WHEN 6 THEN o_orderkey@0 >=
551136 AND o_orderkey@0 <= 55864096 AND o_orderkey@0 IN (SET) ([46147206,
15373056, 15394692, 53397509, 2662214, 30728833, 40376484, 33140323, 12809282,
20315522, 5489475, 551136, 55864096, 28378659, 23129479, 13393635, 23245603,
47452993, 41662242, 28506404, 53650310, 43438883, 43906817, 52415075, 1971680,
33868740, 48943904, 54765253, 34130272, 39395047, 44890534, 39455748, 39732833,
12237830, 5007490, 27027302, 24581062]) WHEN 7 THEN o_orderkey@0 >= 4478371 AND
o_orderkey@0 <= 53972800 AND o_orderkey@0 IN (SET) ([4808560
3, 18188388, 43328641, 33002339, 25332608, 12906020, 28092902, 8574884,
8991333, 9436480, 53972800, 32054309, 21140833, 6210853, 44542087, 6517825,
4478371, 24281478, 49453025, 49519297, 49608135]) WHEN 8 THEN o_orderkey@0 >=
2942469 AND o_orderkey@0 <= 55722822 AND o_orderkey@0 IN (SET) ([42891712,
53101121, 33026341, 27729440, 33162084, 33274246, 55722822, 10889601, 31041542,
2942469, 31192034, 23338693, 41396289, 18869634, 32122243, 52312869, 52566243,
38397669, 13841255, 21530176, 6292038, 14891232, 15036034, 22408480, 37282215])
WHEN 9 THEN o_orderkey@0 >= 1263015 AND o_orderkey@0 <= 58441889 AND
o_orderkey@0 IN (SET) ([45581186, 46098247, 15572326, 43207233, 32937312,
7887841, 55229349, 57823175, 22789733, 58441889, 20676097, 2869152, 1263015,
1742403, 29326886, 3342468, 3883783, 14392483, 4720454, 57324993]) WHEN 10 THEN
o_orderkey@0 >= 29158 AND o_orderkey@0 <= 59993957 AND o_orderkey@0 IN (SET)
([45889221, 47773829, 35542498, 20038309, 53246051, 43322471, 29158, 27658564, 1
2845285, 5296167, 37707840, 31121184, 41277440, 9304229, 51253283, 16215716,
51841767, 29122912, 59993957, 38562980, 14046658, 4134341, 19546182, 14544192,
32624836, 19851237, 4745607, 4806726, 36828227]) WHEN 11 THEN o_orderkey@0 >=
983201 AND o_orderkey@0 <= 56952741 AND o_orderkey@0 IN (SET) ([47953029,
46685344, 17668135, 17701571, 30417318, 18302693, 30775045, 28184576, 31826535,
31932481, 51759617, 983201, 18927141, 28800353, 16416640, 19157927, 43980355,
32393895, 34201984, 6203745, 6581665, 19427463, 35893095, 14875232, 26465922,
56952741]) WHEN 12 THEN o_orderkey@0 >= 857959 AND o_orderkey@0 <= 58390470 AND
o_orderkey@0 IN (SET) ([15157120, 17929698, 32937731, 8231942, 13165024,
58390470, 50776837, 31165796, 15955906, 857959, 43416326, 17301668, 13683589,
17361729, 17612003, 44723233, 44886083, 4290656, 12353700, 26471747, 27531430,
49461377, 49671844]) WHEN 13 THEN o_orderkey@0 >= 6882 AND o_orderkey@0 <=
55876839 AND o_orderkey@0 IN (SET) ([47688132, 48300454, 48301091, 4
8395044, 53092067, 7810823, 6882, 40449575, 5156581, 13185729, 10603495,
10740227, 565574, 55876839, 22897223, 20693186, 43609317, 43952871, 43997089,
20971013, 13563296, 38728164, 48920836, 29749060, 34216164, 42378407, 39353991,
35922529, 4267751, 15104033, 20016800, 4702759, 4722021, 37128899, 27247074])
WHEN 14 THEN o_orderkey@0 >= 7062629 AND o_orderkey@0 <= 59930883 AND
o_orderkey@0 IN (SET) ([45460769, 35383045, 37591043, 43245189, 20324257,
32699424, 32766721, 30338976, 30356293, 40685988, 25859617, 58504644, 31105221,
10906593, 9782273, 43423427, 28546531, 59313570, 18923648, 23794694, 59930883,
48862754, 17366275, 56217669, 56240807, 14211942, 44257860, 7062629, 24518212])
WHEN 15 THEN o_orderkey@0 >= 2232932 AND o_orderkey@0 <= 58982979 AND
o_orderkey@0 IN (SET) ([47886951, 52679876, 52993379, 25243237, 33414662,
33450849, 58000519, 40876003, 46834406, 5849444, 8754087, 3037414, 58982979,
31831749, 11231077, 54188961, 52458145, 32338533, 54849831, 2232932, 34472643,
34937
604, 26011682, 4439686, 36327201, 7379172, 26977863]) WHEN 16 THEN
o_orderkey@0 >= 2745894 AND o_orderkey@0 <= 55747171 AND o_orderkey@0 IN (SET)
([17812705, 46794464, 12701702, 50172704, 10408580, 40705506, 25797412,
55747171, 47058404, 2745894, 2995076, 51031042, 51184261, 11484962, 6082306,
42359809, 3861123, 32579840, 27034435, 7549735, 27580742, 49931207]) WHEN 17
THEN o_orderkey@0 >= 502886 AND o_orderkey@0 <= 54840071 AND o_orderkey@0 IN
(SET) ([30768103, 40462630, 8319046, 502886, 20386147, 22855844, 38363558,
31536773, 9816576, 28911364, 54185891, 48526950, 44178691, 38465697, 33928775,
17457767, 54840071, 2199712, 5984582, 6091649, 3767271, 42290181, 44769410,
26587169, 7165026, 22017989, 36910982, 37004583, 22513536, 49739366]) WHEN 18
THEN o_orderkey@0 >= 1481925 AND o_orderkey@0 <= 58919845 AND o_orderkey@0 IN
(SET) ([42681408, 18200676, 30332516, 40356833, 46892804, 10858721, 20524164,
2806245, 41068385, 58919845, 31677313, 51553093, 53601824, 11599171, 16424550,
32148
069, 43846177, 1481925, 52469894, 39154146, 42021606, 6515330, 39710883,
36420615, 26707522, 22098118, 22253443, 24979495]) WHEN 19 THEN o_orderkey@0 >=
735366 AND o_orderkey@0 <= 58684996 AND o_orderkey@0 IN (SET) ([47880193,
47969124, 48307430, 5200102, 57968195, 10678725, 735366, 58684996, 967334,
18871616, 19217698, 32305890, 33615872, 21135524, 13837220, 29731045, 34181762,
6222658, 34748294, 19628135, 35999840, 15071267, 39702437, 19960768, 49474597,
49896961]) WHEN 20 THEN o_orderkey@0 >= 1845057 AND o_orderkey@0 <= 59417473
AND o_orderkey@0 IN (SET) ([42997059, 46757697, 30218567, 30615650, 8028004,
40714112, 2761378, 5746311, 8433440, 23062983, 9128386, 41588929, 51041954,
11141606, 51748135, 59417473, 18965830, 19232581, 29089474, 52426951, 38422560,
1845057, 38914978, 34509573, 14614786, 12331750, 36866503, 56968864, 37307554,
27458500, 49864035]) WHEN 21 THEN o_orderkey@0 >= 8073922 AND o_orderkey@0 <=
59057792 AND o_orderkey@0 IN (SET) ([45897379, 8073922, 25109123, 552
26209, 55385158, 28309383, 40944001, 58647078, 58673379, 59057792, 9506693,
43491654, 23906277, 54409761, 38656259, 34142818, 56497062, 12285222, 26458949,
22220514, 24813703, 57524647]) WHEN 22 THEN o_orderkey@0 >= 4739650 AND
o_orderkey@0 <= 59565889 AND o_orderkey@0 IN (SET) ([45968358, 46096647,
46226405, 35098820, 55084354, 55799200, 50380160, 30976483, 20649506, 28447970,
9827201, 23671233, 16089345, 23813318, 23957318, 59565889, 33836992, 48881602,
33954756, 34764101, 44232641, 34551296, 14389831, 19738853, 36168806, 12314084,
4739650, 39837541, 6899299, 26584295, 49847329]) WHEN 23 THEN o_orderkey@0 >=
1436544 AND o_orderkey@0 <= 56788359 AND o_orderkey@0 IN (SET) ([45231655,
15191270, 35527463, 53066306, 7611718, 46777222, 10184165, 10612710, 28346247,
23166179, 8639938, 11120293, 16139072, 23861472, 19094241, 1436544, 52340710,
52357185, 29109127, 2096705, 13971749, 14577123, 11785570, 7235014, 56788359,
27529377]) ELSE false END ] AND DynamicFilter [ empty ], metrics=[out
put_rows=15.00 M, elapsed_compute=7.73ms, output_bytes=517.6 MB,
output_batches=1.85 K, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=154 total → 154 matched,
row_groups_pruned_bloom_filter=154 total → 154 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=329.4 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=2.82µs, metadata_load_time=682.59µs,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=94.44ms,
time_elapsed_processing=1.58s, time_elapsed_scanning_total=12.44s,
time_elapsed_scanning_until_data=390.64ms, output_rows_skew=0.61%,
scan_efficiency_ratio=48.65% (329.4 M/677.1 M)]
RepartitionExec: partitioning=Hash([l_orderkey@0], 24),
input_partitions=24, metrics=[output_rows=59.99 M, elapsed_compute=11.65ms,
output_bytes=1375.1 MB, output_batches=7.33 K, spill_count=0, spilled_bytes=0.0
B, spilled_rows=0, fetch_time=1.84s, repartition_time=908.00ms, send_time=1.82s]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:0..106115546],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:106115546..212231092],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:212231092..318346638],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:318346638..424462184],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:424462184..530577730],
...]}, projection=[l_orderkey, l_quantity], file_type=parquet,
predicate=DynamicFilter [ CASE hash_repartition % 24 WHEN 0 THEN l_orderkey@0
>= 101 AND l_orderkey@0 <= 59999974 AND hash_lookup WHEN 1 THEN l_orderkey@0 >=
1 AND l_orderkey@0 <= 59999462 AND hash_lookup WHEN 2 THEN l_orderkey@0 >= 513
AND l_orderkey@0 <= 59999939 AND hash_lookup WHEN 3 THEN l_orderkey@0 >= 66 AND
l_orderkey@0 <= 59999908 AND hash_lookup WHEN 4 TH
EN l_orderkey@0 >= 2 AND l_orderkey@0 <= 59999623 AND hash_lookup WHEN 5 THEN
l_orderkey@0 >= 6 AND l_orderkey@0 <= 59999975 AND hash_lookup WHEN 6 THEN
l_orderkey@0 >= 256 AND l_orderkey@0 <= 59999969 AND hash_lookup WHEN 7 THEN
l_orderkey@0 >= 160 AND l_orderkey@0 <= 59999874 AND hash_lookup WHEN 8 THEN
l_orderkey@0 >= 5 AND l_orderkey@0 <= 59999878 AND hash_lookup WHEN 9 THEN
l_orderkey@0 >= 67 AND l_orderkey@0 <= 59999430 AND hash_lookup WHEN 10 THEN
l_orderkey@0 >= 257 AND l_orderkey@0 <= 59999968 AND hash_lookup WHEN 11 THEN
l_orderkey@0 >= 386 AND l_orderkey@0 <= 59999972 AND hash_lookup WHEN 12 THEN
l_orderkey@0 >= 68 AND l_orderkey@0 <= 59999719 AND hash_lookup WHEN 13 THEN
l_orderkey@0 >= 1063 AND l_orderkey@0 <= 59999943 AND hash_lookup WHEN 14 THEN
l_orderkey@0 >= 352 AND l_orderkey@0 <= 59999905 AND hash_lookup WHEN 15 THEN
l_orderkey@0 >= 192 AND l_orderkey@0 <= 59999841 AND hash_lookup WHEN 16 THEN
l_orderkey@0 >= 39 AND l_orderkey@0 <= 59999942 AND hash_lookup WHEN 1
7 THEN l_orderkey@0 >= 3 AND l_orderkey@0 <= 59999396 AND hash_lookup WHEN 18
THEN l_orderkey@0 >= 289 AND l_orderkey@0 <= 59999971 AND hash_lookup WHEN 19
THEN l_orderkey@0 >= 135 AND l_orderkey@0 <= 59999940 AND hash_lookup WHEN 20
THEN l_orderkey@0 >= 4 AND l_orderkey@0 <= 59999879 AND hash_lookup WHEN 21
THEN l_orderkey@0 >= 38 AND l_orderkey@0 <= 60000000 AND hash_lookup WHEN 22
THEN l_orderkey@0 >= 675 AND l_orderkey@0 <= 59999745 AND hash_lookup WHEN 23
THEN l_orderkey@0 >= 32 AND l_orderkey@0 <= 59999970 AND hash_lookup ELSE false
END ], metrics=[output_rows=59.99 M, elapsed_compute=12.72ms,
output_bytes=1382.7 MB, output_batches=7.48 K,
files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=524 total → 524 matched,
row_groups_pruned_bloom_filter=524 total → 524 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=20
5.8 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=2.45µs, metadata_load_time=2.00ms,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=50.13ms,
time_elapsed_processing=1.14s, time_elapsed_scanning_total=12.49s,
time_elapsed_scanning_until_data=52.21ms, output_rows_skew=0.03%,
scan_efficiency_ratio=8.08% (205.8 M/2.55 B)]
```
So, we seem to compute the join of all (`customer` x `order` x `lineitem`),
most of which are thrown away at right semi join which filters out all small
(`sum(l_quantity) < 300`) orders.
Enabling Pushdown Filters:
```text
SortPreservingMergeExec: [o_totalprice@4 DESC, o_orderdate@3 ASC NULLS
LAST], fetch=100, metrics=[output_rows=100, elapsed_compute=40.41µs,
output_bytes=20.1 KB, output_batches=1]
SortExec: TopK(fetch=100), expr=[o_totalprice@4 DESC, o_orderdate@3 ASC
NULLS LAST], preserve_partitioning=[true], metrics=[output_rows=624,
elapsed_compute=549.00µs, output_bytes=56.1 KB, output_batches=24,
row_replacements=624]
AggregateExec: mode=SinglePartitioned, gby=[c_name@1 as c_name,
c_custkey@0 as c_custkey, o_orderkey@2 as o_orderkey, o_orderdate@4 as
o_orderdate, o_totalprice@3 as o_totalprice], aggr=[sum(lineitem.l_quantity)],
metrics=[output_rows=624, elapsed_compute=872.93µs, output_bytes=66.6 KB,
output_batches=24, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0,
peak_mem_used=267.9 K, aggregate_arguments_time=15.42µs,
aggregation_time=34.82µs, emitting_time=76.25µs,
time_calculating_group_ids=487.47µs]
HashJoinExec: mode=Partitioned, join_type=RightSemi,
on=[(l_orderkey@0, o_orderkey@2)], metrics=[output_rows=4.37 K,
elapsed_compute=15.09s, output_bytes=12.9 MB, output_batches=24,
array_map_created_count=0, build_input_batches=24, build_input_rows=624,
input_batches=24, input_rows=4.37 K, build_mem_used=1.59 M, build_time=7.71s,
join_time=743.03µs, avg_fanout=100% (4.37 K/4.37 K), probe_hit_rate=100% (4.37
K/4.37 K)]
FilterExec: sum(lineitem.l_quantity)@1 > Some(30000),25,2,
projection=[l_orderkey@0], metrics=[output_rows=624, elapsed_compute=196.44ms,
output_bytes=1536.0 KB, output_batches=24, selectivity=0% (624/15.00 M)]
AggregateExec: mode=FinalPartitioned, gby=[l_orderkey@0 as
l_orderkey], aggr=[sum(lineitem.l_quantity)], metrics=[output_rows=15.00 M,
elapsed_compute=7.40s, output_bytes=43.3 GB, output_batches=1.85 K,
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=1.91 B,
aggregate_arguments_time=3.36ms, aggregation_time=1.53s, emitting_time=14.12ms,
time_calculating_group_ids=5.84s]
RepartitionExec: partitioning=Hash([l_orderkey@0], 24),
input_partitions=24, metrics=[output_rows=15.00 M, elapsed_compute=5.27ms,
output_bytes=346.5 MB, output_batches=1.85 K, spill_count=0, spilled_bytes=0.0
B, spilled_rows=0, fetch_time=11.79s, repartition_time=846.65ms,
send_time=3.58s]
AggregateExec: mode=Partial, gby=[l_orderkey@0 as l_orderkey],
aggr=[sum(lineitem.l_quantity)], metrics=[output_rows=15.00 M,
elapsed_compute=9.07s, output_bytes=43.1 GB, output_batches=1.84 K,
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0,
peak_mem_used=958.3 M, aggregate_arguments_time=7.74ms, aggregation_time=1.34s,
emitting_time=5.77ms, time_calculating_group_ids=7.69s, reduction_factor=25.01%
(15.00 M/59.99 M)]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:0..106115546],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:106115546..212231092],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:212231092..318346638],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:318346638..424462184],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:424462184..530577730],
...]}, projection=[l_orderkey, l_quantity], file_type=parquet,
metrics=[output_rows=59.99 M, elapsed_compute=22.61ms, output_bytes=1382.7 MB,
output_batches=7.48 K, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=524 total → 524 matched,
row_groups_pruned_bloom_filter=524 total → 524 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched
, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=205.8 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=1.72µs, metadata_load_time=2.28ms,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=50.13ms,
time_elapsed_processing=1.88s, time_elapsed_scanning_total=11.69s,
time_elapsed_scanning_until_data=60.62ms, output_rows_skew=0.03%,
scan_efficiency_ratio=8.08% (205.8 M/2.55 B)]
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@2,
l_orderkey@0)], projection=[c_custkey@0, c_name@1, o_orderkey@2,
o_totalprice@3, o_orderdate@4, l_quantity@6], metrics=[output_rows=4.37 K,
elapsed_compute=7.80ms, output_bytes=12.9 MB, output_batches=24,
array_map_created_count=0, build_input_batches=24, build_input_rows=624,
input_batches=24, input_rows=4.37 K, build_mem_used=10.44 M, build_time=3.66ms,
join_time=1.05ms, avg_fanout=100% (4.37 K/4.37 K), probe_hit_rate=100% (4.37
K/4.37 K)]
RepartitionExec: partitioning=Hash([o_orderkey@2], 24),
input_partitions=24, metrics=[output_rows=624, elapsed_compute=151.58µs,
output_bytes=9.9 MB, output_batches=24, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=29.17s, repartition_time=963.27µs, send_time=4.97ms]
HashJoinExec: mode=Partitioned, join_type=Inner,
on=[(c_custkey@0, o_custkey@1)], projection=[c_custkey@0, c_name@1,
o_orderkey@2, o_totalprice@4, o_orderdate@5], metrics=[output_rows=624,
elapsed_compute=274.77ms, output_bytes=9.9 MB, output_batches=24,
array_map_created_count=0, build_input_batches=192, build_input_rows=1.50 M,
input_batches=24, input_rows=624, build_mem_used=242.0 M, build_time=261.84ms,
join_time=1.90ms, avg_fanout=100% (624/624), probe_hit_rate=100% (624/624)]
RepartitionExec: partitioning=Hash([c_custkey@0], 24),
input_partitions=24, metrics=[output_rows=1.50 M, elapsed_compute=506.78µs,
output_bytes=179.8 MB, output_batches=192, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=235.35ms, repartition_time=40.98ms,
send_time=326.35ms]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:0..5780685],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:5780685..11561370],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:11561370..17342055],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:17342055..23122740],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/customer.parquet:23122740..28903425],
...]}, projection=[c_custkey, c_name], file_type=parquet,
metrics=[output_rows=1.50 M, elapsed_compute=896.82µs, output_bytes=228.2 MB,
output_batches=210, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=35 total → 35 matched,
row_groups_pruned_bloom_filter=35 total → 35 matched, page_index_pages_pruned=0
total → 0 matched, page_index_rows_pruned=0 total → 0 matched,
limit_pruned_row_grou
ps=0 total → 0 matched, batches_split=0, bytes_scanned=19.30 M,
file_open_errors=0, file_scan_errors=0, files_opened=24, files_processed=24,
num_predicate_creation_errors=0, predicate_evaluation_errors=0,
pushdown_rows_matched=0, pushdown_rows_pruned=0,
predicate_cache_inner_records=0, predicate_cache_records=0,
bloom_filter_eval_time=3.68µs, metadata_load_time=15.30ms,
page_index_eval_time=48ns, row_pushdown_eval_time=48ns,
statistics_eval_time=48ns, time_elapsed_opening=34.13ms,
time_elapsed_processing=205.74ms, time_elapsed_scanning_total=574.53ms,
time_elapsed_scanning_until_data=126.31ms, output_rows_skew=10.91%,
scan_efficiency_ratio=13.91% (19.30 M/138.7 M)]
RepartitionExec: partitioning=Hash([o_custkey@1], 24),
input_partitions=24, metrics=[output_rows=624, elapsed_compute=393.10µs,
output_bytes=6.8 MB, output_batches=24, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=24.27s, repartition_time=10.80ms, send_time=3.66ms]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:0..28214315],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:28214315..56428630],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:56428630..84642945],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:84642945..112857260],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/orders.parquet:112857260..141071575],
...]}, projection=[o_orderkey, o_custkey, o_totalprice, o_orderdate],
file_type=parquet, predicate=DynamicFilter [ CASE hash_repartition % 24 WHEN 0
THEN o_custkey@1 >= 20 AND o_custkey@1 <= 1499995 AND hash_lookup WHEN 1 THEN
o_custkey@1 >= 1 AND o_custkey@1 <= 1499982 AND hash_lookup WHEN 2 THEN
o_custkey@1 >= 13 AND o_custkey@1 <= 1499952 AND hash_lookup WHEN 3 THEN
o_custkey@1 >= 9 AND o_custkey@1 <= 1499974 AND hash_lookup WHEN 4 THE
N o_custkey@1 >= 2 AND o_custkey@1 <= 1499994 AND hash_lookup WHEN 5 THEN
o_custkey@1 >= 6 AND o_custkey@1 <= 1499998 AND hash_lookup WHEN 6 THEN
o_custkey@1 >= 12 AND o_custkey@1 <= 1499986 AND hash_lookup WHEN 7 THEN
o_custkey@1 >= 11 AND o_custkey@1 <= 1499990 AND hash_lookup WHEN 8 THEN
o_custkey@1 >= 5 AND o_custkey@1 <= 1499993 AND hash_lookup WHEN 9 THEN
o_custkey@1 >= 40 AND o_custkey@1 <= 1499950 AND hash_lookup WHEN 10 THEN
o_custkey@1 >= 10 AND o_custkey@1 <= 1499987 AND hash_lookup WHEN 11 THEN
o_custkey@1 >= 41 AND o_custkey@1 <= 1499988 AND hash_lookup WHEN 12 THEN
o_custkey@1 >= 53 AND o_custkey@1 <= 1499977 AND hash_lookup WHEN 13 THEN
o_custkey@1 >= 24 AND o_custkey@1 <= 1499996 AND hash_lookup WHEN 14 THEN
o_custkey@1 >= 26 AND o_custkey@1 <= 1499922 AND hash_lookup WHEN 15 THEN
o_custkey@1 >= 43 AND o_custkey@1 <= 1499997 AND hash_lookup WHEN 16 THEN
o_custkey@1 >= 39 AND o_custkey@1 <= 1499963 AND hash_lookup WHEN 17 THEN
o_custkey@1 >= 3 AND o_custkey@1 <= 14999
80 AND hash_lookup WHEN 18 THEN o_custkey@1 >= 77 AND o_custkey@1 <= 1499984
AND hash_lookup WHEN 19 THEN o_custkey@1 >= 16 AND o_custkey@1 <= 1499999 AND
hash_lookup WHEN 20 THEN o_custkey@1 >= 4 AND o_custkey@1 <= 1499979 AND
hash_lookup WHEN 21 THEN o_custkey@1 >= 17 AND o_custkey@1 <= 1499964 AND
hash_lookup WHEN 22 THEN o_custkey@1 >= 15 AND o_custkey@1 <= 1500000 AND
hash_lookup WHEN 23 THEN o_custkey@1 >= 30 AND o_custkey@1 <= 1499983 AND
hash_lookup ELSE false END ] AND DynamicFilter [ CASE hash_repartition % 24
WHEN 0 THEN o_orderkey@0 >= 1544643 AND o_orderkey@0 <= 59237092 AND
o_orderkey@0 IN (SET) ([37683008, 28077922, 52826116, 40753383, 15738658,
53407847, 50839616, 57974274, 10442629, 3043270, 20662370, 48314274, 56089923,
21057986, 1544643, 59237092, 8506019, 49172291, 37531878, 11028230, 34343042,
30008928, 54796358, 46837987, 19730208, 19730468, 44883872]) WHEN 1 THEN
o_orderkey@0 >= 2366755 AND o_orderkey@0 <= 57137254 AND o_orderkey@0 IN (SET)
([32991271, 3809328
0, 52935685, 13432994, 18353314, 8075589, 20714883, 5832321, 38488484,
39492870, 21973952, 22470437, 31018979, 41064486, 51929955, 57137254, 41325120,
2366755, 40021697, 16801991, 46486080, 17016006, 24690087, 24923202, 12251526])
WHEN 2 THEN o_orderkey@0 >= 6651397 AND o_orderkey@0 <= 59138305 AND
o_orderkey@0 IN (SET) ([55342658, 18152960, 18367712, 15960071, 30567815,
16020773, 58547488, 23660263, 59138305, 23711776, 56516384, 33899427, 18762278,
6651397, 9727175, 34590402, 42262528, 46678469, 32161635, 12378086, 45053063])
WHEN 3 THEN o_orderkey@0 >= 1474818 AND o_orderkey@0 <= 56741287 AND
o_orderkey@0 IN (SET) ([37828134, 53395493, 55807047, 18449123, 30702500,
36667107, 38643041, 21213895, 1474818, 4527553, 22040514, 8590657, 51155008,
22291718, 37023521, 51409728, 56741287, 53623108, 41274690, 31587747, 24378656,
54375559, 54915267, 7125602, 7456839]) WHEN 4 THEN o_orderkey@0 >= 2329187 AND
o_orderkey@0 <= 53938401 AND o_orderkey@0 IN (SET) ([37874913, 50277955,
30864451, 10
805828, 21296295, 39949734, 51532000, 28916832, 53938401, 29805379, 31450375,
44143908, 2329187, 31601378, 42146816, 31855814, 16783360, 17272195, 47071686,
44800711, 12563331]) WHEN 5 THEN o_orderkey@0 >= 4259524 AND o_orderkey@0 <=
57743845 AND o_orderkey@0 IN (SET) ([37775877, 17927718, 57743845, 38926278,
21312615, 4259524, 28584225, 51063047, 29294434, 49619845, 52032003, 57173283,
49704768, 9615783, 24319555, 44331334, 12099683]) WHEN 6 THEN o_orderkey@0 >=
551136 AND o_orderkey@0 <= 55864096 AND o_orderkey@0 IN (SET) ([40376484,
12809282, 2662214, 28378659, 15373056, 15394692, 33140323, 53397509, 13393635,
55864096, 551136, 23129479, 20315522, 23245603, 30728833, 5489475, 39395047,
39455748, 48943904, 33868740, 28506404, 39732833, 34130272, 43438883, 53650310,
43906817, 1971680, 41662242, 52415075, 46147206, 5007490, 54765253, 24581062,
27027302, 47452993, 44890534, 12237830]) WHEN 7 THEN o_orderkey@0 >= 4478371
AND o_orderkey@0 <= 53972800 AND o_orderkey@0 IN (SET) ([2809290
2, 12906020, 33002339, 25332608, 18188388, 43328641, 48085603, 21140833,
4478371, 8574884, 6210853, 8991333, 6517825, 9436480, 49453025, 49519297,
49608135, 53972800, 24281478, 32054309, 44542087]) WHEN 8 THEN o_orderkey@0 >=
2942469 AND o_orderkey@0 <= 55722822 AND o_orderkey@0 IN (SET) ([27729440,
33026341, 2942469, 53101121, 33162084, 33274246, 42891712, 55722822, 13841255,
23338693, 10889601, 38397669, 21530176, 22408480, 37282215, 18869634, 6292038,
14891232, 31041542, 15036034, 31192034, 41396289, 52312869, 52566243,
32122243]) WHEN 9 THEN o_orderkey@0 >= 1263015 AND o_orderkey@0 <= 58441889 AND
o_orderkey@0 IN (SET) ([32937312, 55229349, 2869152, 15572326, 57823175,
43207233, 22789733, 7887841, 45581186, 20676097, 58441889, 3342468, 3883783,
1263015, 4720454, 14392483, 29326886, 1742403, 57324993, 46098247]) WHEN 10
THEN o_orderkey@0 >= 29158 AND o_orderkey@0 <= 59993957 AND o_orderkey@0 IN
(SET) ([27658564, 37707840, 12845285, 29158, 20038309, 53246051, 35542498,
43322471, 1
4046658, 47773829, 5296167, 16215716, 45889221, 38562980, 4134341, 4745607,
4806726, 36828227, 51253283, 14544192, 29122912, 59993957, 9304229, 31121184,
51841767, 41277440, 19546182, 32624836, 19851237]) WHEN 11 THEN o_orderkey@0 >=
983201 AND o_orderkey@0 <= 56952741 AND o_orderkey@0 IN (SET) ([28184576,
17668135, 17701571, 18302693, 30417318, 35893095, 47953029, 30775045, 16416640,
983201, 26465922, 34201984, 28800353, 6203745, 18927141, 51759617, 14875232,
56952741, 6581665, 19157927, 43980355, 31826535, 31932481, 19427463, 46685344,
32393895]) WHEN 12 THEN o_orderkey@0 >= 857959 AND o_orderkey@0 <= 58390470 AND
o_orderkey@0 IN (SET) ([15157120, 32937731, 17929698, 50776837, 13683589,
13165024, 15955906, 8231942, 58390470, 857959, 4290656, 26471747, 43416326,
49461377, 31165796, 49671844, 17301668, 17361729, 17612003, 44723233, 44886083,
12353700, 27531430]) WHEN 13 THEN o_orderkey@0 >= 6882 AND o_orderkey@0 <=
55876839 AND o_orderkey@0 IN (SET) ([40449575, 6882, 5156581, 530920
67, 13185729, 13563296, 55876839, 22897223, 7810823, 565574, 47688132,
35922529, 10603495, 20693186, 10740227, 48300454, 48301091, 48395044, 38728164,
20971013, 4267751, 39353991, 48920836, 4702759, 4722021, 34216164, 37128899,
43609317, 43952871, 43997089, 29749060, 15104033, 42378407, 27247074,
20016800]) WHEN 14 THEN o_orderkey@0 >= 7062629 AND o_orderkey@0 <= 59930883
AND o_orderkey@0 IN (SET) ([37591043, 32699424, 32766721, 40685988, 30338976,
30356293, 45460769, 25859617, 35383045, 43245189, 14211942, 20324257, 58504644,
23794694, 56217669, 56240807, 48862754, 59313570, 28546531, 18923648, 59930883,
43423427, 31105221, 10906593, 9782273, 24518212, 44257860, 17366275, 7062629])
WHEN 15 THEN o_orderkey@0 >= 2232932 AND o_orderkey@0 <= 58982979 AND
o_orderkey@0 IN (SET) ([52679876, 52993379, 25243237, 33414662, 33450849,
40876003, 58000519, 47886951, 3037414, 36327201, 5849444, 26011682, 58982979,
4439686, 8754087, 2232932, 52458145, 54188961, 11231077, 34472643, 31831749,
349376
04, 54849831, 32338533, 46834406, 7379172, 26977863]) WHEN 16 THEN
o_orderkey@0 >= 2745894 AND o_orderkey@0 <= 55747171 AND o_orderkey@0 IN (SET)
([12701702, 2745894, 40705506, 50172704, 17812705, 25797412, 55747171,
10408580, 2995076, 3861123, 51031042, 51184261, 6082306, 49931207, 11484962,
42359809, 46794464, 47058404, 27034435, 32579840, 7549735, 27580742]) WHEN 17
THEN o_orderkey@0 >= 502886 AND o_orderkey@0 <= 54840071 AND o_orderkey@0 IN
(SET) ([40462630, 22855844, 502886, 38363558, 20386147, 8319046, 30768103,
38465697, 3767271, 48526950, 22017989, 33928775, 36910982, 5984582, 26587169,
37004583, 6091649, 28911364, 22513536, 49739366, 9816576, 2199712, 44178691,
54185891, 31536773, 42290181, 54840071, 17457767, 7165026, 44769410]) WHEN 18
THEN o_orderkey@0 >= 1481925 AND o_orderkey@0 <= 58919845 AND o_orderkey@0 IN
(SET) ([40356833, 2806245, 30332516, 42681408, 18200676, 20524164, 16424550,
10858721, 36420615, 58919845, 1481925, 39154146, 22098118, 39710883, 22253443,
267075
22, 51553093, 6515330, 53601824, 41068385, 43846177, 52469894, 31677313,
42021606, 11599171, 32148069, 46892804, 24979495]) WHEN 19 THEN o_orderkey@0 >=
735366 AND o_orderkey@0 <= 58684996 AND o_orderkey@0 IN (SET) ([5200102,
13837220, 57968195, 735366, 47880193, 35999840, 47969124, 10678725, 48307430,
967334, 58684996, 21135524, 33615872, 39702437, 34181762, 18871616, 6222658,
19217698, 49474597, 29731045, 49896961, 15071267, 34748294, 32305890, 19628135,
19960768]) WHEN 20 THEN o_orderkey@0 >= 1845057 AND o_orderkey@0 <= 59417473
AND o_orderkey@0 IN (SET) ([30218567, 2761378, 40714112, 42997059, 23062983,
8028004, 30615650, 38422560, 5746311, 38914978, 8433440, 51041954, 36866503,
59417473, 14614786, 37307554, 18965830, 51748135, 29089474, 9128386, 56968864,
19232581, 1845057, 41588929, 49864035, 11141606, 52426951, 34509573, 46757697,
12331750, 27458500]) WHEN 21 THEN o_orderkey@0 >= 8073922 AND o_orderkey@0 <=
59057792 AND o_orderkey@0 IN (SET) ([28309383, 55226209, 55385158, 25
109123, 8073922, 45897379, 58647078, 58673379, 59057792, 38656259, 23906277,
56497062, 22220514, 26458949, 34142818, 43491654, 40944001, 9506693, 57524647,
54409761, 24813703, 12285222]) WHEN 22 THEN o_orderkey@0 >= 4739650 AND
o_orderkey@0 <= 59565889 AND o_orderkey@0 IN (SET) ([55084354, 50380160,
35098820, 55799200, 16089345, 20649506, 36168806, 23671233, 23813318, 23957318,
48881602, 4739650, 33836992, 28447970, 33954756, 14389831, 39837541, 26584295,
59565889, 30976483, 49847329, 9827201, 45968358, 46096647, 34551296, 34764101,
46226405, 6899299, 44232641, 19738853, 12314084]) WHEN 23 THEN o_orderkey@0 >=
1436544 AND o_orderkey@0 <= 56788359 AND o_orderkey@0 IN (SET) ([15191270,
28346247, 53066306, 7611718, 45231655, 35527463, 13971749, 10184165, 23166179,
16139072, 10612710, 1436544, 23861472, 8639938, 14577123, 56788359, 29109127,
19094241, 2096705, 52340710, 11120293, 52357185, 46777222, 11785570, 7235014,
27529377]) ELSE false END ] AND DynamicFilter [ empty ], metrics=[out
put_rows=624, elapsed_compute=3.44ms, output_bytes=24.6 KB,
output_batches=154, files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=154 total → 154 matched,
row_groups_pruned_bloom_filter=154 total → 154 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=311.2 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=624,
pushdown_rows_pruned=15.00 M, predicate_cache_inner_records=38.65 M,
predicate_cache_records=2.68 K, bloom_filter_eval_time=1.52ms,
metadata_load_time=2.92ms, page_index_eval_time=48ns,
row_pushdown_eval_time=19.10s, statistics_eval_time=48ns,
time_elapsed_opening=91.31ms, time_elapsed_processing=22.91s,
time_elapsed_scanning_total=24.20s, time_elapsed_scanning_until_data=2.74s,
output_rows_skew=
2.02%, scan_efficiency_ratio=45.95% (311.2 M/677.1 M)]
RepartitionExec: partitioning=Hash([l_orderkey@0], 24),
input_partitions=24, metrics=[output_rows=4.37 K, elapsed_compute=231.88µs,
output_bytes=4.5 MB, output_batches=24, spill_count=0, spilled_bytes=0.0 B,
spilled_rows=0, fetch_time=19.32s, repartition_time=6.12ms, send_time=4.10ms]
DataSourceExec: file_groups={24 groups:
[[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:0..106115546],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:106115546..212231092],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:212231092..318346638],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:318346638..424462184],
[home/tschwarzinger/bartgeier/polars-2.0-benchmark/data/tpch-10/lineitem.parquet:424462184..530577730],
...]}, projection=[l_orderkey, l_quantity], file_type=parquet,
predicate=DynamicFilter [ CASE hash_repartition % 24 WHEN 0 THEN l_orderkey@0
>= 1544643 AND l_orderkey@0 <= 59237092 AND l_orderkey@0 IN (SET) ([49172291,
21057986, 28077922, 30008928, 48314274, 59237092, 56089923, 1544643, 15738658,
37531878, 20662370, 11028230, 40753383, 52826116, 53407847, 3043270, 57974274,
10442629, 37683008, 44883872, 8506019, 34343042, 5479
6358, 50839616, 46837987, 19730468, 19730208]) WHEN 1 THEN l_orderkey@0 >=
2366755 AND l_orderkey@0 <= 57137254 AND l_orderkey@0 IN (SET) ([41064486,
41325120, 13432994, 24690087, 57137254, 38093280, 31018979, 18353314, 46486080,
12251526, 40021697, 38488484, 22470437, 24923202, 51929955, 8075589, 39492870,
5832321, 52935685, 32991271, 2366755, 20714883, 21973952, 17016006, 16801991])
WHEN 2 THEN l_orderkey@0 >= 6651397 AND l_orderkey@0 <= 59138305 AND
l_orderkey@0 IN (SET) ([15960071, 12378086, 42262528, 16020773, 18367712,
56516384, 6651397, 23711776, 9727175, 55342658, 30567815, 59138305, 32161635,
45053063, 58547488, 23660263, 34590402, 33899427, 18152960, 46678469,
18762278]) WHEN 3 THEN l_orderkey@0 >= 1474818 AND l_orderkey@0 <= 56741287 AND
l_orderkey@0 IN (SET) ([54375559, 37023521, 7456839, 30702500, 24378656,
41274690, 51409728, 37828134, 53623108, 36667107, 8590657, 4527553, 7125602,
53395493, 56741287, 21213895, 22040514, 51155008, 55807047, 31587747, 22291718,
38643041
, 18449123, 54915267, 1474818]) WHEN 4 THEN l_orderkey@0 >= 2329187 AND
l_orderkey@0 <= 53938401 AND l_orderkey@0 IN (SET) ([10805828, 21296295,
28916832, 2329187, 44800711, 50277955, 30864451, 31450375, 17272195, 16783360,
53938401, 37874913, 51532000, 31855814, 42146816, 47071686, 12563331, 31601378,
39949734, 44143908, 29805379]) WHEN 5 THEN l_orderkey@0 >= 4259524 AND
l_orderkey@0 <= 57743845 AND l_orderkey@0 IN (SET) ([29294434, 37775877,
57173283, 21312615, 24319555, 49619845, 57743845, 38926278, 51063047, 52032003,
28584225, 44331334, 9615783, 49704768, 17927718, 12099683, 4259524]) WHEN 6
THEN l_orderkey@0 >= 551136 AND l_orderkey@0 <= 55864096 AND l_orderkey@0 IN
(SET) ([23129479, 43906817, 41662242, 12237830, 39732833, 46147206, 24581062,
44890534, 52415075, 34130272, 33140323, 551136, 5007490, 15373056, 53397509,
43438883, 28506404, 39455748, 5489475, 55864096, 30728833, 40376484, 39395047,
23245603, 15394692, 20315522, 27027302, 13393635, 2662214, 12809282, 1971680,
2837
8659, 47452993, 53650310, 33868740, 54765253, 48943904]) WHEN 7 THEN
l_orderkey@0 >= 4478371 AND l_orderkey@0 <= 53972800 AND l_orderkey@0 IN (SET)
([49453025, 32054309, 44542087, 43328641, 28092902, 9436480, 49519297,
49608135, 4478371, 18188388, 8574884, 53972800, 12906020, 25332608, 6517825,
33002339, 24281478, 6210853, 48085603, 21140833, 8991333]) WHEN 8 THEN
l_orderkey@0 >= 2942469 AND l_orderkey@0 <= 55722822 AND l_orderkey@0 IN (SET)
([33274246, 18869634, 15036034, 38397669, 52312869, 52566243, 31192034,
53101121, 10889601, 13841255, 32122243, 41396289, 14891232, 2942469, 23338693,
33162084, 21530176, 33026341, 31041542, 27729440, 55722822, 37282215, 22408480,
42891712, 6292038]) WHEN 9 THEN l_orderkey@0 >= 1263015 AND l_orderkey@0 <=
58441889 AND l_orderkey@0 IN (SET) ([3883783, 29326886, 58441889, 15572326,
14392483, 7887841, 57823175, 3342468, 43207233, 55229349, 45581186, 32937312,
2869152, 1742403, 22789733, 20676097, 1263015, 57324993, 46098247, 4720454])
WHEN 10 THEN
l_orderkey@0 >= 29158 AND l_orderkey@0 <= 59993957 AND l_orderkey@0 IN (SET)
([29122912, 59993957, 14046658, 4134341, 38562980, 14544192, 4806726, 12845285,
20038309, 5296167, 53246051, 4745607, 31121184, 47773829, 32624836, 51841767,
9304229, 51253283, 29158, 37707840, 27658564, 43322471, 16215716, 36828227,
19851237, 35542498, 41277440, 45889221, 19546182]) WHEN 11 THEN l_orderkey@0 >=
983201 AND l_orderkey@0 <= 56952741 AND l_orderkey@0 IN (SET) ([34201984,
6581665, 18927141, 32393895, 56952741, 17701571, 35893095, 31826535, 31932481,
43980355, 30775045, 28184576, 983201, 46685344, 30417318, 26465922, 19427463,
18302693, 51759617, 16416640, 19157927, 47953029, 6203745, 17668135, 14875232,
28800353]) WHEN 12 THEN l_orderkey@0 >= 857959 AND l_orderkey@0 <= 58390470 AND
l_orderkey@0 IN (SET) ([44723233, 17301668, 32937731, 13683589, 8231942,
50776837, 26471747, 15157120, 49461377, 44886083, 43416326, 857959, 17361729,
31165796, 12353700, 49671844, 17612003, 58390470, 13165024, 17929
698, 27531430, 15955906, 4290656]) WHEN 13 THEN l_orderkey@0 >= 6882 AND
l_orderkey@0 <= 55876839 AND l_orderkey@0 IN (SET) ([47688132, 35922529,
42378407, 43952871, 43997089, 565574, 27247074, 22897223, 5156581, 48300454,
13563296, 55876839, 4702759, 29749060, 48395044, 4722021, 40449575, 48920836,
4267751, 48301091, 10603495, 34216164, 13185729, 20971013, 37128899, 20016800,
43609317, 53092067, 15104033, 6882, 10740227, 39353991, 7810823, 20693186,
38728164]) WHEN 14 THEN l_orderkey@0 >= 7062629 AND l_orderkey@0 <= 59930883
AND l_orderkey@0 IN (SET) ([56240807, 37591043, 43423427, 48862754, 44257860,
43245189, 23794694, 18923648, 30356293, 28546531, 59930883, 35383045, 7062629,
17366275, 40685988, 45460769, 20324257, 14211942, 24518212, 30338976, 9782273,
56217669, 32766721, 32699424, 10906593, 58504644, 59313570, 31105221,
25859617]) WHEN 15 THEN l_orderkey@0 >= 2232932 AND l_orderkey@0 <= 58982979
AND l_orderkey@0 IN (SET) ([52679876, 7379172, 3037414, 4439686, 5849444,
36327201
, 58000519, 52993379, 26977863, 11231077, 8754087, 54849831, 34937604,
46834406, 2232932, 33450849, 47886951, 26011682, 33414662, 58982979, 40876003,
54188961, 32338533, 52458145, 25243237, 31831749, 34472643]) WHEN 16 THEN
l_orderkey@0 >= 2745894 AND l_orderkey@0 <= 55747171 AND l_orderkey@0 IN (SET)
([10408580, 2745894, 6082306, 47058404, 51031042, 7549735, 51184261, 40705506,
17812705, 55747171, 42359809, 25797412, 12701702, 3861123, 50172704, 32579840,
49931207, 2995076, 27034435, 27580742, 11484962, 46794464]) WHEN 17 THEN
l_orderkey@0 >= 502886 AND l_orderkey@0 <= 54840071 AND l_orderkey@0 IN (SET)
([40462630, 44769410, 49739366, 9816576, 54185891, 38363558, 20386147,
48526950, 44178691, 7165026, 17457767, 8319046, 37004583, 36910982, 42290181,
22855844, 22513536, 31536773, 6091649, 22017989, 54840071, 502886, 3767271,
2199712, 38465697, 30768103, 28911364, 5984582, 33928775, 26587169]) WHEN 18
THEN l_orderkey@0 >= 1481925 AND l_orderkey@0 <= 58919845 AND l_orderkey@0 IN
(SET)
([11599171, 26707522, 39154146, 31677313, 30332516, 20524164, 32148069,
6515330, 58919845, 22253443, 36420615, 42681408, 53601824, 43846177, 46892804,
39710883, 51553093, 16424550, 40356833, 1481925, 42021606, 10858721, 22098118,
18200676, 2806245, 41068385, 24979495, 52469894]) WHEN 19 THEN l_orderkey@0 >=
735366 AND l_orderkey@0 <= 58684996 AND l_orderkey@0 IN (SET) ([35999840,
39702437, 47969124, 10678725, 13837220, 18871616, 47880193, 5200102, 735366,
19628135, 19960768, 57968195, 58684996, 19217698, 967334, 34748294, 29731045,
49896961, 15071267, 32305890, 49474597, 6222658, 34181762, 33615872, 48307430,
21135524]) WHEN 20 THEN l_orderkey@0 >= 1845057 AND l_orderkey@0 <= 59417473
AND l_orderkey@0 IN (SET) ([38914978, 1845057, 18965830, 11141606, 29089474,
41588929, 12331750, 56968864, 37307554, 49864035, 5746311, 42997059, 38422560,
9128386, 19232581, 51041954, 46757697, 8028004, 2761378, 40714112, 23062983,
27458500, 34509573, 14614786, 30615650, 52426951, 8433440, 51748135,
59417473, 36866503, 30218567]) WHEN 21 THEN l_orderkey@0 >= 8073922 AND
l_orderkey@0 <= 59057792 AND l_orderkey@0 IN (SET) ([55385158, 59057792,
38656259, 9506693, 22220514, 43491654, 23906277, 54409761, 55226209, 57524647,
45897379, 25109123, 8073922, 28309383, 58673379, 56497062, 58647078, 40944001,
24813703, 26458949, 34142818, 12285222]) WHEN 22 THEN l_orderkey@0 >= 4739650
AND l_orderkey@0 <= 59565889 AND l_orderkey@0 IN (SET) ([33836992, 44232641,
34551296, 34764101, 55084354, 23813318, 6899299, 39837541, 46226405, 19738853,
50380160, 59565889, 35098820, 20649506, 23671233, 55799200, 28447970, 36168806,
12314084, 48881602, 49847329, 14389831, 23957318, 45968358, 46096647, 16089345,
26584295, 30976483, 9827201, 33954756, 4739650]) WHEN 23 THEN l_orderkey@0 >=
1436544 AND l_orderkey@0 <= 56788359 AND l_orderkey@0 IN (SET) ([11785570,
19094241, 53066306, 52357185, 27529377, 8639938, 35527463, 7611718, 10612710,
28346247, 13971749, 23861472, 23166179, 52340710, 11120293, 46777222,
56788359, 16139072, 14577123, 15191270, 10184165, 1436544, 7235014, 45231655,
29109127, 2096705]) ELSE false END ], metrics=[output_rows=4.37 K,
elapsed_compute=3.04ms, output_bytes=104.1 KB, output_batches=372,
files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=524 total → 524 matched,
row_groups_pruned_bloom_filter=524 total → 524 matched,
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0,
bytes_scanned=168.9 M, file_open_errors=0, file_scan_errors=0, files_opened=24,
files_processed=24, num_predicate_creation_errors=0,
predicate_evaluation_errors=0, pushdown_rows_matched=4.37 K,
pushdown_rows_pruned=59.98 M, predicate_cache_inner_records=59.18 M,
predicate_cache_records=4.37 K, bloom_filter_eval_time=5.39µs,
metadata_load_time=2.40ms, page_index_eval_time=48ns,
row_pushdown_eval_time=16.61s, statistics_eval_time=48ns, time_elapsed_openi
ng=199.90ms, time_elapsed_processing=18.30s,
time_elapsed_scanning_total=19.14s, time_elapsed_scanning_until_data=2.51s,
output_rows_skew=2.36%, scan_efficiency_ratio=6.63% (168.9 M/2.55 B)]
```
I think this hides much of the join ordering problems. For example, the
second `lineitems` scan only returns `4.37 K` elements. But the processing time
in the scans increases by quite some time, eliminating most of the reads (e.g.,
`row_pushdown_eval_time=16.61s` in the last scan). It seems like DF can't prune
any of the row groups: `files_ranges_pruned_statistics=24 total → 24 matched,
row_groups_pruned_statistics=524 total → 524 matched,
row_groups_pruned_bloom_filter=524 total → 524 matched,`. Not sure how to get
this info in Polars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
