This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/proto-33bc8ff79f
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit d4aa32b460 chore(deps): bump pbjson-types from 0.8.0 to 0.9.0 in the
proto group
add 8d9b080882 [Minor] propagate distinct_count as inexact through unions
(#20846)
add 4b022c0e33 fix: do not recompute hash join exec properties if not
required (#20900)
add 6b7152387c [main] Bump to 52.3.0 and changelog (#20790) (#20849)
add 385d9dbe71 try to remove redundant alias in expression rewriter and
select (#20867)
add 8b412deaca fix: Optimize `!~ '.*'` case to `col IS NULL AND
Boolean(NULL)` instead of `Eq ""` (#20702)
add 57b275a8d3 feat: correct struct column names for `arrays_zip` return
type (#20886)
add fcb1c935c7 Fix duplicate group keys after hash aggregation spill
(#20724) (#20858)
add 422b54579f fix: Track metrics in hash joins with empty build sides
(#20810)
add 3c56e5d77f perf: Use batched row conversion for `array_has_any`,
`array_has_all` (#20588)
add b7e421325c Include .proto files in datafusion-proto-common
distribution (#20921)
add d2278a90b4 Check sqllogictests for any dangling config settings
(#17914) (#20838)
add 10d8bcba63 Add support for ListView in unnest (#20760)
add d09ff92def feat: Reduce allocations for aggregating `Statistics`
(#20768)
add 2c871b2f30 Project only accessed struct leaves in Parquet row filter
pushdown (#20854)
add 9c3c01aa54 refactor: Improve `SessionContext::parse_duration` API
(#20816)
add c74976f8be minor: Move PreparedAccessPlan to same module as
ParquetAccessPlan (#20929)
add 5db04b8503 chore(deps): bump pyjwt from 2.11.0 to 2.12.0 (#20938)
add 6d3a846483 Rewrite `SUM(expr + scalar)` --> `SUM(expr) +
scalar*COUNT(expr)` (#20749)
add 9b7d0924bc Add AGENTS.md / CLAUDE.md (#20939)
add 538a2019f8 perf: Optimize array set ops on sliced arrays (#20693)
add 1f59d327ba fix: dfbench respects DATAFUSION_RUNTIME_MEMORY_LIMIT env
var (#20631)
add ab2823475d Support `columns_sorted` in row_filters (#20497)
add 8609288ae0 Add --simulate-latency / SIMULATE_LATENCY option to
dfbench / ./bench.sh (#20954)
add b61aee77c8 Minor: make signatures of `SessionContext::register_*`
methods consistent (#20873)
add 3ece9ec2e4 test: add reproducer for Dictionary InList pushdown type
mismatch (#2… (#20960)
add c6f7145e31 Extract shared `ParquetReadPlan` for leaf column resolution
(#20913)
add 5d37babf36 chore: Remove usage of `paste` crate (#20946)
add 4166a6d60f perf: Optimize comparison on nested types (#20716)
add bd071be1dc feat: add `custom_string_literal_override` to unparser
Dialect trait (#20590)
add 26251bb37a Use exact distinct_count from statistics if exists for
`COUNT(DISTINCT column))` calculations (#20845)
add a7e09412a2 fix(spark): return input string for PATH/FILE on schemeless
URLs in `parse_url` (#20506)
add 972b890027 thin-ci (#20972)
add fa6706a8cd chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 (#20973)
add 0dfcd97a37 Replace ahash with foldhash for faster hashing in
datafusion-common (#20958)
add 4c961256da Fix decimal log precision for non-power values (#20433)
add bde7fcaf45 chore(deps): bump pbjson-types from 0.8.0 to 0.9.0 in the
proto group
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d4aa32b460)
\
N -- N -- N refs/heads/dependabot/cargo/main/proto-33bc8ff79f
(bde7fcaf45)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/audit.yml | 2 +-
.github/workflows/extended.yml | 2 +-
AGENTS.md | 34 +
CLAUDE.md | 1 +
Cargo.lock | 110 ++--
Cargo.toml | 86 +--
benchmarks/Cargo.toml | 2 +
benchmarks/README.md | 6 +-
benchmarks/bench.sh | 43 +-
benchmarks/src/clickbench.rs | 4 +-
benchmarks/src/h2o.rs | 4 +-
benchmarks/src/hj.rs | 4 +-
benchmarks/src/imdb/run.rs | 6 +-
benchmarks/src/nlj.rs | 4 +-
benchmarks/src/smj.rs | 4 +-
benchmarks/src/sort_tpch.rs | 4 +-
benchmarks/src/tpcds/run.rs | 4 +-
benchmarks/src/tpch/run.rs | 6 +-
benchmarks/src/util/latency_object_store.rs | 157 +++++
benchmarks/src/util/mod.rs | 1 +
benchmarks/src/util/options.rs | 71 ++-
datafusion/common/Cargo.toml | 7 +-
datafusion/common/benches/stats_merge.rs | 85 +++
datafusion/common/benches/with_hashes.rs | 6 +-
datafusion/common/src/error.rs | 141 +++--
datafusion/common/src/hash_utils.rs | 147 +++--
datafusion/common/src/scalar/mod.rs | 18 +-
datafusion/common/src/stats.rs | 683 ++++++++++++++++-----
datafusion/common/src/utils/aggregate.rs | 149 +++++
datafusion/common/src/utils/mod.rs | 1 +
datafusion/core/Cargo.toml | 1 -
datafusion/core/src/execution/context/mod.rs | 141 ++++-
datafusion/core/tests/dataframe/mod.rs | 2 +-
datafusion/core/tests/parquet/page_pruning.rs | 620 +++++++++++--------
datafusion/core/tests/parquet/row_group_pruning.rs | 642 ++++++++++---------
.../physical_optimizer/aggregate_statistics.rs | 153 ++++-
.../tests/physical_optimizer/filter_pushdown.rs | 2 +-
datafusion/core/tests/sql/unparser.rs | 16 +-
datafusion/datasource-parquet/src/access_plan.rs | 61 +-
datafusion/datasource-parquet/src/opener.rs | 53 +-
datafusion/datasource-parquet/src/row_filter.rs | 614 ++++++++++++++----
datafusion/datasource-parquet/src/sort.rs | 79 ++-
datafusion/expr-common/Cargo.toml | 1 -
datafusion/expr-common/src/interval_arithmetic.rs | 38 +-
datafusion/expr/Cargo.toml | 1 -
datafusion/expr/src/expr.rs | 2 +-
datafusion/expr/src/expr_rewriter/order_by.rs | 240 +++++++-
datafusion/expr/src/expr_schema.rs | 4 +-
datafusion/expr/src/logical_plan/plan.rs | 18 +-
datafusion/expr/src/test/function_stub.rs | 2 -
datafusion/expr/src/udaf.rs | 96 +++
datafusion/functions-aggregate-common/Cargo.toml | 1 -
.../src/aggregate/count_distinct/native.rs | 2 +-
datafusion/functions-aggregate-common/src/utils.rs | 2 +-
datafusion/functions-aggregate/Cargo.toml | 3 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 2 +-
datafusion/functions-aggregate/src/count.rs | 51 +-
datafusion/functions-aggregate/src/hyperloglog.rs | 11 +-
datafusion/functions-aggregate/src/macros.rs | 2 -
datafusion/functions-aggregate/src/sum.rs | 52 +-
datafusion/functions-nested/Cargo.toml | 1 -
.../functions-nested/benches/array_set_ops.rs | 109 ++++
datafusion/functions-nested/src/array_has.rs | 251 ++++++--
datafusion/functions-nested/src/arrays_zip.rs | 4 +-
datafusion/functions-nested/src/except.rs | 81 ++-
datafusion/functions-nested/src/macros.rs | 6 -
datafusion/functions-nested/src/set_ops.rs | 180 +++++-
datafusion/functions-table/Cargo.toml | 1 -
datafusion/functions-table/src/lib.rs | 34 +-
datafusion/functions-window/Cargo.toml | 1 -
datafusion/functions-window/src/cume_dist.rs | 1 +
datafusion/functions-window/src/lead_lag.rs | 2 +
datafusion/functions-window/src/macros.rs | 60 +-
datafusion/functions-window/src/nth_value.rs | 3 +
datafusion/functions-window/src/ntile.rs | 1 +
datafusion/functions-window/src/rank.rs | 3 +
datafusion/functions-window/src/row_number.rs | 1 +
datafusion/functions/src/math/log.rs | 107 ++--
.../src/simplify_expressions/linear_aggregates.rs | 229 +++++++
.../optimizer/src/simplify_expressions/mod.rs | 1 +
.../optimizer/src/simplify_expressions/regex.rs | 13 +-
.../src/simplify_expressions/simplify_exprs.rs | 210 ++++++-
datafusion/physical-expr-common/Cargo.toml | 9 +-
.../physical-expr-common/benches/compare_nested.rs | 74 +++
datafusion/physical-expr-common/src/binary_map.rs | 4 +-
.../physical-expr-common/src/binary_view_map.rs | 4 +-
datafusion/physical-expr-common/src/datum.rs | 8 +-
datafusion/physical-expr/Cargo.toml | 2 -
.../physical-expr/src/expressions/in_list.rs | 6 +-
.../physical-expr/src/expressions/negative.rs | 19 +-
datafusion/physical-plan/Cargo.toml | 1 -
.../aggregates/group_values/multi_group_by/mod.rs | 2 +-
.../src/aggregates/group_values/row.rs | 2 +-
.../group_values/single_group_by/primitive.rs | 3 +-
datafusion/physical-plan/src/aggregates/mod.rs | 5 +-
.../physical-plan/src/aggregates/row_hash.rs | 12 +
.../src/aggregates/topk/hash_table.rs | 3 +-
datafusion/physical-plan/src/execution_plan.rs | 7 +-
.../physical-plan/src/joins/hash_join/exec.rs | 15 +-
.../src/joins/hash_join/partitioned_hash_eval.rs | 84 +--
.../physical-plan/src/joins/hash_join/stream.rs | 6 +-
.../physical-plan/src/joins/symmetric_hash_join.rs | 4 +-
datafusion/physical-plan/src/joins/utils.rs | 2 +-
datafusion/physical-plan/src/repartition/mod.rs | 3 +-
datafusion/physical-plan/src/sorts/sort.rs | 2 +-
datafusion/physical-plan/src/union.rs | 313 +++++++++-
datafusion/physical-plan/src/unnest.rs | 33 +-
.../src/windows/bounded_window_agg_exec.rs | 2 +-
datafusion/proto-common/Cargo.toml | 3 -
datafusion/proto/proto/datafusion.proto | 3 -
datafusion/proto/src/generated/pbjson.rs | 63 --
datafusion/proto/src/generated/prost.rs | 6 -
datafusion/proto/src/physical_plan/from_proto.rs | 7 +-
datafusion/proto/src/physical_plan/to_proto.rs | 6 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 6 +-
datafusion/spark/src/function/url/parse_url.rs | 126 +++-
datafusion/sql/Cargo.toml | 1 -
datafusion/sql/src/expr/function.rs | 2 +
datafusion/sql/src/expr/mod.rs | 68 +-
datafusion/sql/src/select.rs | 17 +-
datafusion/sql/src/unparser/dialect.rs | 11 +
datafusion/sql/src/unparser/expr.rs | 84 ++-
datafusion/sql/src/utils.rs | 4 +-
.../src/engines/datafusion_engine/runner.rs | 63 +-
datafusion/sqllogictest/test_files/aggregate.slt | 24 +-
.../test_files/aggregates_simplify.slt | 66 +-
datafusion/sqllogictest/test_files/array.slt | 62 +-
datafusion/sqllogictest/test_files/clickbench.slt | 23 +-
datafusion/sqllogictest/test_files/decimal.slt | 20 +-
datafusion/sqllogictest/test_files/joins.slt | 43 ++
datafusion/sqllogictest/test_files/order.slt | 48 ++
.../test_files/parquet_filter_pushdown.slt | 46 ++
.../sqllogictest/test_files/set_variable.slt | 30 +
.../sqllogictest/test_files/simplify_expr.slt | 19 +-
.../test_files/spark/url/parse_url.slt | 133 ++++
.../test_files/spark/url/try_parse_url.slt | 133 ++++
datafusion/sqllogictest/test_files/unnest.slt | 96 +++
dev/changelog/{49.0.2.md => 52.3.0.md} | 21 +-
docs/source/library-user-guide/upgrading/52.0.0.md | 2 +-
docs/source/user-guide/configs.md | 2 +-
uv.lock | 6 +-
141 files changed, 5979 insertions(+), 1893 deletions(-)
create mode 100644 AGENTS.md
create mode 120000 CLAUDE.md
create mode 100644 benchmarks/src/util/latency_object_store.rs
create mode 100644 datafusion/common/benches/stats_merge.rs
create mode 100644 datafusion/common/src/utils/aggregate.rs
create mode 100644
datafusion/optimizer/src/simplify_expressions/linear_aggregates.rs
create mode 100644 datafusion/physical-expr-common/benches/compare_nested.rs
copy dev/changelog/{49.0.2.md => 52.3.0.md} (51%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
