This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/rand_distr-0.6.0
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit be9d9625e4 chore(deps): bump rand_distr from 0.5.1 to 0.6.0
add 3a23bb2531 perf: Optimize `array_agg()` using `GroupsAccumulator`
(#20504)
add 8df75c3f04 Document guidance on how to evaluate breaking API changes
(#20584)
add 8482e2e448 feat: support extension planner for `TableScan` (#20548)
add 6713439497 perf: Optimize `array_to_string()`, support more types
(#20553)
add 95de1bf410 Add metrics for parquet sink (#20307)
add 1f37a33ce5 Update DataFusion meetups page on docs (#20629)
add 93d177db4e Extend dynamic filter to joins that preserve probe side ON
(#20447)
add 0af9ff5998 Improve sqllogicteset speed by creating only a single large
file rather than 2 (#20586)
add 43584caf11 cli: Fix datafusion-cli hint edge cases (#20609)
add 02dae77420 Speedup sqllogictests by running long running tests first
(#20576)
add a5f490e4c3 Add `ExecutionPlan::apply_expressions()` (#20337)
add 12314c5564 perf: Optimize `array_to_string` to avoid a copy (#20639)
add 0bf376735e fix: make the `sql` feature truly optional (#20625)
add 657887dca9 Fix custom metric display (#20643)
add 476c200f00 refactor: Set expected runtime config in error message when
the used disk space during the spilling process has exceeded the allocation
limit (#20375)
add 3238a7e45b more families for the CI (#20663)
add 0ca9d6586a CI: Add CodeQL workflow for GitHub Actions security
scanning (#20636)
add d1a305821f perf: Apply logical regexp optimizations to Utf8View and
LargeUtf8 inputs (#20581)
add 88fa0dfce6 Add `Field` to `Expr::Cast` -- allow logical expressions to
express a cast to an extension type (#18136)
add d2df7a57c4 perf: Optimize `array_concat` using `MutableArrayData`
(#20620)
add 7f42c8c40e chore(deps): bump astral-sh/setup-uv from 7.3.0 to 7.3.1
(#20660)
add bcc52cd4c2 chore(deps): bump taiki-e/install-action from 2.68.8 to
2.68.16 (#20661)
add 2cbee474d7 fix: use try_shrink instead of shrink in try_resize (#20424)
add 79b5b24099 fix: Provide more generic API for the capacity limit
parsing (#20372)
add 3b028fe677 Improve formatting of datatypes (#20605)
add a1148409bf docs: Update `datafusion-cli` doc for
`top-memory-consumers` config (#20390)
add 92d0a5c527 Add explain plans for ClickBench queries (#20666)
add 10b5f22411 [main] Update version to 52.2.0 (#20573)
add 909608aa5f fix: Fix bug in `array_has` scalar path with sliced arrays
(#20677)
add 028e351595 Add files_processed and files_scanned metrics to
FileStreamMetrics (#20592)
add d025869526 fix: `HashJoin` panic with String dictionary keys (don't
flatten keys) (#20505)
add b092bd4117 Update releases links with releases in 2025-2026 (#20630)
add d412ba5d18 Speedup push_down_filter_regression.slt by using
uncompressed parquet (#20652)
add b0349ffefc feat: support nanosecond date_part (#20674)
add 0f093f405c Implement cardinality_effect for window execs and UnionExec
(#20321)
add 5d27860438 feat: parse `JsonAccess` as a binary operator, add
`Operator::Colon` (#20628)
add cbe5cb36ce ci: Harden labeler workflow, remove unnecessary checkout
from pull_request_target job (#20637)
add ddfc282008 Add tests for sqllogictest prioritization (#20656)
add 340ef606cf perf: Optimize `to_char` to allocate less, fix NULL
handling (#20635)
add 27c9cdaecc correct parquet leaf index mapping when schema contains
struct cols (#20698)
add 1f0232c995 Reattach parquet metadata cache after deserializing in
datafusion-proto (#20574)
add 848cd6387a Eliminate deterministic group by keys with deterministic
transformations (#20706)
add 46ac990878 Wire up with_new_state with DataSource (#20718)
add 29e849519e chore: Enable `assigning_clones` clippy lint (#20670)
add c919054241 perf: short-circuit and collect_bool for IN list with
column references (#20694)
add 953bdf4100 feat: Support Spark `array_contains` builtin function
(#20685)
add 00e36e8b72 fix: Return `probe_side.len()` for RightMark/Anti count(*)
queries (#20710)
add 13cebf8432 FFI_TableOptions are using default values only (#20721)
add 631c918039 perf: sort replace free()->try_grow() pattern with
try_resize() to reduce memory pool interactions (#20729)
add 03d17e8473 Improve documentation for `AggregateUdfImpl::simplify` and
`WindowUDFImpl::simplify` (#20712)
add a95da7028b doc: Add more context to `Precision` (#20713)
add dd988f6efc Fix test that's broken on Windows due to naive path
handling (#20692)
add b3976d6320 Fix DELETE/UPDATE filter extraction when predicates are
pushed down into TableScan (#19884)
add 678d1ad7f4 Minor: Add comment explaining rationale to avoid
dependencies on functions (#20667)
add 33c922f254 use linker optimization for extended sqllogictests (#20740)
add d72b0b8061 fix: preserve None projection semantics across FFI boundary
in ForeignTableProvider::scan (#20393)
add 02ce571910 Push even local limits past windows (#20752)
add 0ac434d0e5 Add case-heavy LEFT JOIN benchmark and debug timing/logging
for PushDownFilter hot paths (#20664)
add 5211a8b7c5 Fix repartition from dropping data when spilling (#20672)
add bfa0ea8056 Hash join buffering on probe side (#19761)
add 8fe926d387 test: Add `datafusion-cli` `fair` and `unbounded`
memory-pool test coverage (#20565)
add 92078d932d Copy limits before repartitions (#20736)
add 4dbb4498fc ser/de fetch in FilterExec (#20738)
add 1eb520675f feat: Integrate CastColumnExpr into PhysicalExprAdapter
(#20269)
add 37b9a466c4 feat: `partition_statistics()` for HashJoinExec (#20711)
add 15bc6bd5c8 feat: make DefaultLogicalExtensionCodec support
serialisation of buil… (#20638)
add bb421db72a Add tests for simplifying multiple aggregate expressions
(#20723)
add 33b9afa911 Allow SQL `TypePlanner` to plan SQL types as extension
types (#20676)
add b51edffc44 Update reverse UDF to emit utf8view when input is utf8view
(#20604)
add 9b3d6a4909 Make lower and upper emit Utf8View for Utf8View input
(#20616)
add 097f04c2ec fix(spark): handle divide-by-zero in Spark `mod`/`pmod`
with ANSI mode support (#20461)
add aca8c14478 Fix FilterExec converting Absent column stats to
Exact(NULL) (#20391)
add 44dfa7b8a5 Clean up date_part preimage implementation (#20350)
add 84a22eaa32 Wrap Arc to Statistics for `partition_statistics` API
(#20570)
add fd97799ddc Make Physical CastExpr Field-aware and unify cast semantics
across physical expressions (#20814)
add 75c7da5da3 Pass ConfigOptions to scalar UDFs via FFI (#20454)
add 39226c3f30 [datafusion-cli] Replace mutex with AtomicU64 for stream
duration tracking in instrumentedObjectStore (#20802)
add af79d14e7d Make translate emit Utf8View for Utf8View input (#20624)
add 23b88fbed7 Allow filters on struct fields to be pushed down into
Parquet scan (#20822)
add 6f86c8d878 Used constant with mapping instead of write! to display
scalar value bytes (#20719)
add 1f87930940 fix: sqllogictest cannot convert <subquery> to Substrait
(#19739)
add daa8f52c16 fix: interval analysis error when have two filterexec that
inner filter proves zero selectivity (#20743)
add fc514c2506 perf: Optimize set operations to avoid RowConverter
deserialization overhead (#20623)
add 31a4037da5 chore(deps): bump taiki-e/install-action from 2.68.16 to
2.68.25 (#20842)
add 64b5228790 chore(deps): bump github/codeql-action from 4.32.5 to
4.32.6 (#20843)
add 8e02b8ede5 chore: Ignore RUSTSEC-2024-0421 (#20850)
add 5af7361a98 fix: SanityCheckPlan error with window functions and NVL
filter (#20231)
add 9b7cddaa04 chore(deps): bump quinn-proto from 0.11.13 to 0.11.14
(#20859)
add 48199b9da3 Use `ParquetPushDecoder` in `ParquetOpener` (#20839)
add 86cb815e03 [Minor] Remove redundant ProjectionExec nodes in sort-based
plans (#20780)
add 2589fa8ac0 doc: Add documentation for pushing limit into plan (#20271)
add 4bac1cf878 impl ser/de for preserve_order in RepartitionExec (#20798)
add da05287c0f Fix FileStream scanning_total to include sync next-file
open time (#20627)
add 95a3dfd153 chore: Ignore RUSTSEC-2024-0014 (#20862)
add ed793f0de0 chore: clean up dependencies (#20861)
add 1efcbf57dd Add benchmark for struct field filter pushdown in Parquet
(#20829)
add 21cf60a9d3 Add Null Type Coercions for Placeholders (#20543)
add d68b8009e4 Minor: Deprecate unused `PartitionedFileStream` (#20869)
add f8fb5bd505 fix: Avoid unnecessary type casts in `concat_ws` (#20436)
add 981b5c37bf chore(deps): bump substrait from 0.62 to 0.63.0 (#20876)
add 129c58fa43 fix: Remove `!=0` check from
`supports_collect_by_thresholds` (#20730)
add 8d9b080882 [Minor] propagate distinct_count as inexact through unions
(#20846)
add 4b022c0e33 fix: do not recompute hash join exec properties if not
required (#20900)
add 6b7152387c [main] Bump to 52.3.0 and changelog (#20790) (#20849)
add 385d9dbe71 try to remove redundant alias in expression rewriter and
select (#20867)
add 8b412deaca fix: Optimize `!~ '.*'` case to `col IS NULL AND
Boolean(NULL)` instead of `Eq ""` (#20702)
add 57b275a8d3 feat: correct struct column names for `arrays_zip` return
type (#20886)
add fcb1c935c7 Fix duplicate group keys after hash aggregation spill
(#20724) (#20858)
add 422b54579f fix: Track metrics in hash joins with empty build sides
(#20810)
add 3c56e5d77f perf: Use batched row conversion for `array_has_any`,
`array_has_all` (#20588)
add b7e421325c Include .proto files in datafusion-proto-common
distribution (#20921)
add d2278a90b4 Check sqllogictests for any dangling config settings
(#17914) (#20838)
add 10d8bcba63 Add support for ListView in unnest (#20760)
add d09ff92def feat: Reduce allocations for aggregating `Statistics`
(#20768)
add 2c871b2f30 Project only accessed struct leaves in Parquet row filter
pushdown (#20854)
add 9c3c01aa54 refactor: Improve `SessionContext::parse_duration` API
(#20816)
add c74976f8be minor: Move PreparedAccessPlan to same module as
ParquetAccessPlan (#20929)
add 5db04b8503 chore(deps): bump pyjwt from 2.11.0 to 2.12.0 (#20938)
add 6d3a846483 Rewrite `SUM(expr + scalar)` --> `SUM(expr) +
scalar*COUNT(expr)` (#20749)
add 9b7d0924bc Add AGENTS.md / CLAUDE.md (#20939)
add 538a2019f8 perf: Optimize array set ops on sliced arrays (#20693)
add 1f59d327ba fix: dfbench respects DATAFUSION_RUNTIME_MEMORY_LIMIT env
var (#20631)
add ab2823475d Support `columns_sorted` in row_filters (#20497)
add 8609288ae0 Add --simulate-latency / SIMULATE_LATENCY option to
dfbench / ./bench.sh (#20954)
add b61aee77c8 Minor: make signatures of `SessionContext::register_*`
methods consistent (#20873)
add 3ece9ec2e4 test: add reproducer for Dictionary InList pushdown type
mismatch (#2… (#20960)
add c6f7145e31 Extract shared `ParquetReadPlan` for leaf column resolution
(#20913)
add 5d37babf36 chore: Remove usage of `paste` crate (#20946)
add c0664e7a95 chore(deps): bump rand_distr from 0.5.1 to 0.6.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (be9d9625e4)
\
N -- N -- N refs/heads/dependabot/cargo/main/rand_distr-0.6.0
(c0664e7a95)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/audit.yml | 4 +-
.github/workflows/{stale.yml => codeql.yml} | 45 +-
.github/workflows/docs.yaml | 2 +-
.github/workflows/docs_pr.yaml | 2 +-
.github/workflows/extended.yml | 15 +-
.github/workflows/labeler.yml | 2 -
.github/workflows/rust.yml | 32 +-
AGENTS.md | 34 +
CLAUDE.md | 1 +
Cargo.lock | 110 ++-
Cargo.toml | 79 +-
benchmarks/Cargo.toml | 2 +
benchmarks/README.md | 6 +-
benchmarks/bench.sh | 43 +-
benchmarks/src/clickbench.rs | 4 +-
benchmarks/src/h2o.rs | 4 +-
benchmarks/src/hj.rs | 4 +-
benchmarks/src/imdb/run.rs | 14 +-
benchmarks/src/nlj.rs | 4 +-
benchmarks/src/smj.rs | 4 +-
benchmarks/src/sort_tpch.rs | 4 +-
benchmarks/src/tpcds/run.rs | 10 +-
benchmarks/src/tpch/run.rs | 14 +-
benchmarks/src/util/latency_object_store.rs | 157 ++++
benchmarks/src/util/mod.rs | 1 +
benchmarks/src/util/options.rs | 112 ++-
datafusion-cli/src/exec.rs | 1 +
datafusion-cli/src/helper.rs | 33 +-
datafusion-cli/src/object_storage/instrumented.rs | 209 +++--
datafusion-cli/tests/cli_integration.rs | 49 +-
...ory_consumers_with_mem_pool_type@no_track.snap} | 2 +
[email protected]} | 2 +
.../[email protected] | 36 +
.../custom_data_source/custom_datasource.rs | 17 +
.../memory_pool_execution_plan.rs | 17 +
.../examples/proto/composed_extension_codec.rs | 19 +
.../examples/relation_planner/table_sample.rs | 24 +-
datafusion/catalog/src/memory/table.rs | 15 +-
datafusion/common/Cargo.toml | 5 +-
datafusion/common/benches/stats_merge.rs | 85 ++
datafusion/common/src/config.rs | 30 +-
datafusion/common/src/error.rs | 141 ++-
datafusion/common/src/join_type.rs | 29 +
datafusion/common/src/scalar/mod.rs | 21 +-
datafusion/common/src/stats.rs | 712 +++++++++++----
datafusion/common/src/types/logical.rs | 120 ++-
datafusion/common/src/types/native.rs | 120 ++-
datafusion/common/src/utils/aggregate.rs | 149 ++++
datafusion/common/src/utils/mod.rs | 1 +
datafusion/core/Cargo.toml | 13 +-
datafusion/core/benches/aggregate_query_sql.rs | 11 +
datafusion/core/benches/data_utils/mod.rs | 18 +-
datafusion/core/benches/sql_planner_extended.rs | 239 ++++-
datafusion/core/benches/topk_repartition.rs | 90 ++
datafusion/core/src/dataframe/parquet.rs | 133 +++
datafusion/core/src/datasource/listing/mod.rs | 7 +-
datafusion/core/src/datasource/mod.rs | 4 +-
datafusion/core/src/execution/context/mod.rs | 281 +++++-
datafusion/core/src/physical_planner.rs | 443 ++++++++--
.../tests/custom_sources_cases/dml_planning.rs | 426 ++++++++-
datafusion/core/tests/custom_sources_cases/mod.rs | 25 +-
.../provider_filter_pushdown.rs | 19 +-
.../core/tests/custom_sources_cases/statistics.rs | 31 +-
datafusion/core/tests/dataframe/mod.rs | 36 +-
datafusion/core/tests/fuzz_cases/once_exec.rs | 17 +
datafusion/core/tests/memory_limit/mod.rs | 15 +-
datafusion/core/tests/parquet/expr_adapter.rs | 146 ++-
datafusion/core/tests/parquet/page_pruning.rs | 620 +++++++------
datafusion/core/tests/parquet/row_group_pruning.rs | 642 +++++++-------
.../physical_optimizer/enforce_distribution.rs | 18 +-
.../tests/physical_optimizer/filter_pushdown.rs | 375 ++++++++
.../tests/physical_optimizer/join_selection.rs | 35 +-
.../physical_optimizer/partition_statistics.rs | 365 ++++++--
.../tests/physical_optimizer/pushdown_utils.rs | 29 +
.../core/tests/physical_optimizer/test_utils.rs | 44 +-
datafusion/core/tests/sql/path_partition.rs | 10 +-
datafusion/core/tests/sql/runtime_config.rs | 23 +-
.../core/tests/user_defined/insert_operation.rs | 17 +
.../core/tests/user_defined/user_defined_plan.rs | 20 +-
.../user_defined/user_defined_scalar_functions.rs | 5 +-
datafusion/datasource-arrow/src/source.rs | 15 +
datafusion/datasource-avro/src/source.rs | 15 +
datafusion/datasource-csv/src/source.rs | 15 +
datafusion/datasource-json/Cargo.toml | 4 +-
datafusion/datasource-json/src/source.rs | 15 +
datafusion/datasource-parquet/Cargo.toml | 6 +
.../benches/parquet_struct_filter_pushdown.rs | 353 ++++++++
datafusion/datasource-parquet/src/access_plan.rs | 61 +-
datafusion/datasource-parquet/src/file_format.rs | 35 +-
datafusion/datasource-parquet/src/opener.rs | 355 ++++----
datafusion/datasource-parquet/src/row_filter.rs | 978 ++++++++++++++++++---
datafusion/datasource-parquet/src/sort.rs | 79 +-
datafusion/datasource-parquet/src/source.rs | 21 +
datafusion/datasource/src/file.rs | 22 +
datafusion/datasource/src/file_scan_config.rs | 37 +-
datafusion/datasource/src/file_stream.rs | 80 +-
datafusion/datasource/src/memory.rs | 29 +-
datafusion/datasource/src/mod.rs | 4 +
datafusion/datasource/src/sink.rs | 16 +-
datafusion/datasource/src/source.rs | 57 +-
datafusion/datasource/src/test_util.rs | 9 +-
datafusion/execution/Cargo.toml | 2 +-
datafusion/execution/src/disk_manager.rs | 3 +-
datafusion/execution/src/memory_pool/mod.rs | 4 +-
datafusion/expr-common/Cargo.toml | 4 +-
datafusion/expr-common/src/interval_arithmetic.rs | 38 +-
datafusion/expr-common/src/operator.rs | 17 +-
datafusion/expr-common/src/signature.rs | 195 +++-
.../expr-common/src/type_coercion/aggregates.rs | 11 +-
datafusion/expr-common/src/type_coercion/binary.rs | 9 +-
datafusion/expr/Cargo.toml | 1 -
datafusion/expr/src/expr.rs | 66 +-
datafusion/expr/src/expr_rewriter/order_by.rs | 258 +++++-
datafusion/expr/src/expr_schema.rs | 29 +-
datafusion/expr/src/function.rs | 40 +-
datafusion/expr/src/logical_plan/plan.rs | 54 +-
datafusion/expr/src/planner.rs | 18 +
datafusion/expr/src/test/function_stub.rs | 2 -
datafusion/expr/src/tree_node.rs | 8 +-
datafusion/expr/src/type_coercion/functions.rs | 57 +-
datafusion/expr/src/udaf.rs | 132 ++-
datafusion/expr/src/udf.rs | 4 +-
datafusion/expr/src/udwf.rs | 27 +-
datafusion/expr/src/utils.rs | 100 +++
datafusion/ffi/src/execution_plan.rs | 33 +
datafusion/ffi/src/session/mod.rs | 107 ++-
datafusion/ffi/src/table_provider.rs | 77 +-
datafusion/ffi/src/tests/async_provider.rs | 17 +
datafusion/ffi/src/tests/mod.rs | 3 +
datafusion/ffi/src/tests/udf_udaf_udwf.rs | 44 +-
datafusion/ffi/src/udf/mod.rs | 39 +-
datafusion/ffi/tests/ffi_udf.rs | 48 +-
.../src/aggregate/groups_accumulator/nulls.rs | 4 +-
datafusion/functions-aggregate/Cargo.toml | 1 -
datafusion/functions-aggregate/src/array_agg.rs | 718 ++++++++++++++-
datafusion/functions-aggregate/src/macros.rs | 2 -
datafusion/functions-aggregate/src/sum.rs | 50 +-
datafusion/functions-nested/Cargo.toml | 10 +-
.../functions-nested/benches/array_concat.rs | 94 ++
.../functions-nested/benches/array_set_ops.rs | 130 +++
.../functions-nested/benches/array_to_string.rs | 188 ++++
datafusion/functions-nested/src/array_has.rs | 325 ++++++-
datafusion/functions-nested/src/arrays_zip.rs | 4 +-
datafusion/functions-nested/src/concat.rs | 101 ++-
datafusion/functions-nested/src/except.rs | 119 ++-
datafusion/functions-nested/src/macros.rs | 6 -
datafusion/functions-nested/src/position.rs | 81 +-
datafusion/functions-nested/src/set_ops.rs | 252 +++++-
datafusion/functions-nested/src/string.rs | 431 ++++-----
datafusion/functions-table/Cargo.toml | 1 -
datafusion/functions-table/src/lib.rs | 34 +-
datafusion/functions-window/Cargo.toml | 1 -
datafusion/functions-window/src/cume_dist.rs | 1 +
datafusion/functions-window/src/lead_lag.rs | 2 +
datafusion/functions-window/src/macros.rs | 60 +-
datafusion/functions-window/src/nth_value.rs | 3 +
datafusion/functions-window/src/ntile.rs | 1 +
datafusion/functions-window/src/rank.rs | 3 +
datafusion/functions-window/src/row_number.rs | 1 +
datafusion/functions/benches/to_char.rs | 61 +-
datafusion/functions/src/core/arrow_cast.rs | 8 +-
datafusion/functions/src/datetime/date_part.rs | 95 +-
datafusion/functions/src/datetime/to_char.rs | 214 ++---
datafusion/functions/src/string/common.rs | 10 +-
datafusion/functions/src/string/concat.rs | 42 +-
datafusion/functions/src/string/concat_ws.rs | 382 +++++++-
datafusion/functions/src/string/lower.rs | 25 +-
datafusion/functions/src/string/upper.rs | 25 +-
datafusion/functions/src/strings.rs | 12 +-
datafusion/functions/src/unicode/reverse.rs | 64 +-
datafusion/functions/src/unicode/translate.rs | 150 +++-
datafusion/optimizer/Cargo.toml | 8 +
datafusion/optimizer/src/analyzer/type_coercion.rs | 2 +-
.../optimizer/src/eliminate_group_by_constant.rs | 111 ++-
datafusion/optimizer/src/eliminate_outer_join.rs | 4 +-
datafusion/optimizer/src/push_down_filter.rs | 65 +-
.../src/simplify_expressions/expr_simplifier.rs | 81 +-
.../src/simplify_expressions/linear_aggregates.rs | 229 +++++
.../optimizer/src/simplify_expressions/mod.rs | 1 +
.../optimizer/src/simplify_expressions/regex.rs | 132 ++-
.../src/simplify_expressions/simplify_exprs.rs | 210 ++++-
.../physical-expr-adapter/src/schema_rewriter.rs | 191 ++--
.../physical-expr-common/src/metrics/value.rs | 10 +-
datafusion/physical-expr/Cargo.toml | 1 -
.../src/equivalence/properties/dependency.rs | 40 +-
.../src/equivalence/properties/mod.rs | 55 +-
datafusion/physical-expr/src/expressions/binary.rs | 2 +-
datafusion/physical-expr/src/expressions/cast.rs | 260 +++++-
.../physical-expr/src/expressions/cast_column.rs | 8 +
.../physical-expr/src/expressions/in_list.rs | 264 ++++--
.../physical-expr/src/expressions/negative.rs | 19 +-
datafusion/physical-expr/src/planner.rs | 89 +-
datafusion/physical-optimizer/Cargo.toml | 1 +
datafusion/physical-optimizer/src/ensure_coop.rs | 10 +-
.../physical-optimizer/src/hash_join_buffering.rs | 103 +++
.../physical-optimizer/src/join_selection.rs | 10 +-
datafusion/physical-optimizer/src/lib.rs | 2 +
.../physical-optimizer/src/limit_pushdown.rs | 41 +
.../src/limit_pushdown_past_window.rs | 119 ++-
datafusion/physical-optimizer/src/optimizer.rs | 11 +
.../physical-optimizer/src/output_requirements.rs | 36 +-
.../physical-optimizer/src/topk_repartition.rs | 368 ++++++++
datafusion/physical-plan/src/aggregates/mod.rs | 55 +-
.../physical-plan/src/aggregates/row_hash.rs | 12 +
.../physical-plan/src/aggregates/topk/heap.rs | 4 +-
datafusion/physical-plan/src/analyze.rs | 9 +
datafusion/physical-plan/src/async_func.rs | 10 +-
datafusion/physical-plan/src/buffer.rs | 10 +-
datafusion/physical-plan/src/coalesce_batches.rs | 15 +-
.../physical-plan/src/coalesce_partitions.rs | 15 +-
datafusion/physical-plan/src/coop.rs | 10 +-
datafusion/physical-plan/src/display.rs | 34 +-
datafusion/physical-plan/src/empty.rs | 14 +-
datafusion/physical-plan/src/execution_plan.rs | 224 ++++-
datafusion/physical-plan/src/explain.rs | 10 +-
datafusion/physical-plan/src/filter.rs | 278 +++++-
datafusion/physical-plan/src/joins/cross_join.rs | 19 +-
.../physical-plan/src/joins/hash_join/exec.rs | 103 ++-
.../src/joins/hash_join/inlist_builder.rs | 39 +-
.../physical-plan/src/joins/hash_join/stream.rs | 7 +-
.../physical-plan/src/joins/nested_loop_join.rs | 22 +-
.../src/joins/piecewise_merge_join/exec.rs | 9 +
.../src/joins/sort_merge_join/exec.rs | 31 +-
.../physical-plan/src/joins/symmetric_hash_join.rs | 21 +
datafusion/physical-plan/src/joins/utils.rs | 15 +-
datafusion/physical-plan/src/limit.rs | 45 +-
datafusion/physical-plan/src/memory.rs | 10 +-
datafusion/physical-plan/src/placeholder_row.rs | 15 +-
datafusion/physical-plan/src/projection.rs | 24 +-
datafusion/physical-plan/src/recursive_query.rs | 9 +
datafusion/physical-plan/src/repartition/mod.rs | 24 +-
datafusion/physical-plan/src/sorts/partial_sort.rs | 16 +-
datafusion/physical-plan/src/sorts/sort.rs | 97 +-
.../src/sorts/sort_preserving_merge.rs | 21 +-
datafusion/physical-plan/src/spill/spill_pool.rs | 102 ++-
datafusion/physical-plan/src/streaming.rs | 9 +
datafusion/physical-plan/src/test.rs | 25 +-
datafusion/physical-plan/src/test/exec.rs | 67 +-
datafusion/physical-plan/src/union.rs | 396 ++++++++-
datafusion/physical-plan/src/unnest.rs | 45 +-
.../src/windows/bounded_window_agg_exec.rs | 45 +-
.../physical-plan/src/windows/window_agg_exec.rs | 73 +-
datafusion/physical-plan/src/work_table.rs | 14 +-
datafusion/proto-common/Cargo.toml | 3 -
datafusion/proto/proto/datafusion.proto | 25 +
datafusion/proto/src/generated/pbjson.rs | 307 +++++++
datafusion/proto/src/generated/prost.rs | 68 ++
datafusion/proto/src/logical_plan/from_proto.rs | 20 +-
datafusion/proto/src/logical_plan/mod.rs | 103 ++-
datafusion/proto/src/logical_plan/to_proto.rs | 12 +-
datafusion/proto/src/physical_plan/mod.rs | 30 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 186 ++++
.../proto/tests/cases/roundtrip_physical_plan.rs | 100 ++-
datafusion/pruning/Cargo.toml | 2 +-
.../spark/src/function/array/array_contains.rs | 168 ++++
datafusion/spark/src/function/array/mod.rs | 15 +-
datafusion/spark/src/function/math/modulus.rs | 144 ++-
datafusion/sql/Cargo.toml | 1 -
datafusion/sql/src/expr/binary_op.rs | 3 +-
datafusion/sql/src/expr/function.rs | 2 +
datafusion/sql/src/expr/mod.rs | 117 +--
datafusion/sql/src/planner.rs | 4 +-
datafusion/sql/src/select.rs | 17 +-
datafusion/sql/src/statement.rs | 11 +-
datafusion/sql/src/unparser/expr.rs | 161 ++--
datafusion/sql/src/utils.rs | 4 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 38 +-
datafusion/sql/tests/common/mod.rs | 16 +-
datafusion/sql/tests/sql_integration.rs | 51 +-
datafusion/sqllogictest/bin/sqllogictests.rs | 85 +-
.../src/engines/datafusion_engine/runner.rs | 63 +-
datafusion/sqllogictest/src/lib.rs | 2 +
datafusion/sqllogictest/src/test_file.rs | 186 ++++
.../test_files/aggregate_skip_partial.slt | 41 +-
.../test_files/aggregates_simplify.slt | 358 ++++++++
datafusion/sqllogictest/test_files/array.slt | 169 +++-
.../sqllogictest/test_files/arrow_typeof.slt | 2 +-
datafusion/sqllogictest/test_files/binary.slt | 2 +-
datafusion/sqllogictest/test_files/clickbench.slt | 932 +++++++++++++++++++-
datafusion/sqllogictest/test_files/copy.slt | 11 +
datafusion/sqllogictest/test_files/cte.slt | 2 +-
.../sqllogictest/test_files/datetime/date_part.slt | 92 +-
.../test_files/datetime/timestamps.slt | 31 +-
.../test_files/dynamic_filter_pushdown_config.slt | 99 ++-
datafusion/sqllogictest/test_files/encoding.slt | 6 +-
datafusion/sqllogictest/test_files/explain.slt | 18 +-
datafusion/sqllogictest/test_files/expr.slt | 2 +-
.../test_files/filter_without_sort_exec.slt | 44 +
datafusion/sqllogictest/test_files/functions.slt | 54 +-
.../sqllogictest/test_files/information_schema.slt | 4 +
datafusion/sqllogictest/test_files/joins.slt | 86 ++
datafusion/sqllogictest/test_files/math.slt | 6 +-
datafusion/sqllogictest/test_files/nvl.slt | 2 +-
datafusion/sqllogictest/test_files/order.slt | 48 +
.../test_files/parquet_filter_pushdown.slt | 198 +++++
.../test_files/projection_pushdown.slt | 24 +-
.../test_files/push_down_filter_regression.slt | 17 +-
datafusion/sqllogictest/test_files/scalar.slt | 2 +-
.../sqllogictest/test_files/set_variable.slt | 33 +
.../sqllogictest/test_files/simplify_expr.slt | 23 +-
.../test_files/spark/array/array_contains.slt | 140 +++
.../test_files/spark/datetime/unix.slt | 8 +-
.../sqllogictest/test_files/spark/math/mod.slt | 29 +
.../sqllogictest/test_files/spark/math/pmod.slt | 22 +-
.../test_files/spark/string/base64.slt | 4 +-
.../test_files/string/string_literal.slt | 37 +
.../sqllogictest/test_files/string/string_view.slt | 14 +-
datafusion/sqllogictest/test_files/struct.slt | 2 +-
datafusion/sqllogictest/test_files/unnest.slt | 96 ++
datafusion/sqllogictest/test_files/update.slt | 63 +-
datafusion/sqllogictest/test_files/window.slt | 50 +-
.../sqllogictest/test_files/window_limits.slt | 7 +-
.../test_files/window_topk_pushdown.slt | 141 +++
datafusion/substrait/Cargo.toml | 2 +-
datafusion/substrait/src/extensions.rs | 8 +-
.../src/logical_plan/consumer/expr/cast.rs | 11 +-
.../logical_plan/consumer/expr/field_reference.rs | 3 +
.../src/logical_plan/consumer/expr/literal.rs | 24 +-
.../src/logical_plan/consumer/expr/mod.rs | 3 +
.../logical_plan/consumer/substrait_consumer.rs | 30 +-
.../substrait/src/logical_plan/consumer/types.rs | 20 +-
.../src/logical_plan/producer/expr/cast.rs | 16 +-
.../logical_plan/producer/expr/field_reference.rs | 16 +
.../src/logical_plan/producer/expr/literal.rs | 2 +-
.../src/logical_plan/producer/expr/mod.rs | 12 +-
.../logical_plan/producer/expr/scalar_function.rs | 1 +
.../logical_plan/producer/expr/singular_or_list.rs | 22 +-
.../src/logical_plan/producer/expr/subquery.rs | 81 +-
.../substrait/src/logical_plan/producer/plan.rs | 4 -
.../logical_plan/producer/substrait_producer.rs | 30 +-
.../substrait/src/logical_plan/producer/types.rs | 44 +-
.../substrait/src/logical_plan/producer/utils.rs | 27 +-
.../tests/cases/roundtrip_logical_plan.rs | 165 +++-
datafusion/substrait/tests/utils.rs | 1 +
dev/changelog/52.2.0.md | 47 +
dev/changelog/{49.0.2.md => 52.3.0.md} | 21 +-
docs/source/contributor-guide/api-health.md | 71 +-
.../library-user-guide/custom-table-providers.md | 30 +-
docs/source/library-user-guide/extending-sql.md | 51 +-
docs/source/library-user-guide/upgrading/52.0.0.md | 2 +-
docs/source/library-user-guide/upgrading/54.0.0.md | 124 +++
docs/source/library-user-guide/upgrading/index.rst | 1 +
docs/source/user-guide/cli/usage.md | 4 +-
docs/source/user-guide/concepts-readings-events.md | 51 +-
docs/source/user-guide/configs.md | 4 +-
docs/source/user-guide/sql/scalar_functions.md | 2 +-
uv.lock | 6 +-
347 files changed, 20408 insertions(+), 3944 deletions(-)
copy .github/workflows/{stale.yml => codeql.yml} (53%)
create mode 100644 AGENTS.md
create mode 120000 CLAUDE.md
create mode 100644 benchmarks/src/util/latency_object_store.rs
copy datafusion-cli/tests/snapshots/{cli_top_memory_consumers@no_track.snap =>
cli_top_memory_consumers_with_mem_pool_type@no_track.snap} (94%)
copy datafusion-cli/tests/snapshots/{[email protected] =>
[email protected]} (95%)
create mode 100644
datafusion-cli/tests/snapshots/[email protected]
create mode 100644 datafusion/common/benches/stats_merge.rs
create mode 100644 datafusion/common/src/utils/aggregate.rs
create mode 100644 datafusion/core/benches/topk_repartition.rs
create mode 100644
datafusion/datasource-parquet/benches/parquet_struct_filter_pushdown.rs
create mode 100644 datafusion/functions-nested/benches/array_concat.rs
create mode 100644 datafusion/functions-nested/benches/array_to_string.rs
create mode 100644
datafusion/optimizer/src/simplify_expressions/linear_aggregates.rs
create mode 100644 datafusion/physical-optimizer/src/hash_join_buffering.rs
create mode 100644 datafusion/physical-optimizer/src/topk_repartition.rs
create mode 100644 datafusion/spark/src/function/array/array_contains.rs
create mode 100644 datafusion/sqllogictest/src/test_file.rs
create mode 100644 datafusion/sqllogictest/test_files/aggregates_simplify.slt
create mode 100644
datafusion/sqllogictest/test_files/spark/array/array_contains.slt
create mode 100644 datafusion/sqllogictest/test_files/window_topk_pushdown.slt
create mode 100644 dev/changelog/52.2.0.md
copy dev/changelog/{49.0.2.md => 52.3.0.md} (51%)
create mode 100644 docs/source/library-user-guide/upgrading/54.0.0.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
