Re: [PR] Derive `Clone` for more ExecutionPlans [datafusion]

2024-11-02 Thread via GitHub
alamb merged PR #13203: URL: https://github.com/apache/datafusion/pull/13203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Revert "Improve push down filter of join (#13184)" [datafusion]

2024-11-02 Thread via GitHub
alamb commented on PR #13229: URL: https://github.com/apache/datafusion/pull/13229#issuecomment-2452946251 Thanks -- merging quickly to get main to a releasable state. I think the plan to reopen the PR and keep working on it is great. -- This is an automated message from the Apach

Re: [PR] Improve push down filter of join [datafusion]

2024-11-02 Thread via GitHub
alamb commented on PR #13184: URL: https://github.com/apache/datafusion/pull/13184#issuecomment-2452946234 For anyone following along, this PR appears to have had some correctness issues so @eejbyfeldt reverted it https://github.com/apache/datafusion/pull/13229 @JasonLi-cn are you w

Re: [PR] Derive `Clone` for more ExecutionPlans [datafusion]

2024-11-02 Thread via GitHub
alamb commented on PR #13203: URL: https://github.com/apache/datafusion/pull/13203#issuecomment-2452946361 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Switch to iterative `DynNode` and `ConcreteTreeNode` processing [datafusion]

2024-11-02 Thread via GitHub
peter-toth commented on PR #13177: URL: https://github.com/apache/datafusion/pull/13177#issuecomment-2452940289 Thank you @blaginin for improving the PR and pointing me to those conversations. Unfortunately I missed those and wasn't aware that there have been an attempt to add stack growth

Re: [I] PR: Improve push down filter of join #13184 introduces correctness issues [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #13211: PR: Improve push down filter of join #13184 introduces correctness issues URL: https://github.com/apache/datafusion/issues/13211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Revert "Improve push down filter of join (#13184)" [datafusion]

2024-11-02 Thread via GitHub
alamb merged PR #13229: URL: https://github.com/apache/datafusion/pull/13229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Improve push down filter of join [datafusion]

2024-11-02 Thread via GitHub
Dandandan commented on PR #13184: URL: https://github.com/apache/datafusion/pull/13184#issuecomment-2452952489 I think for this pushdown to be correct, the join type can be changed. I think this can be done in two phases: * if there is a filter for a left join on a right side (or le

Re: [PR] improve support for T-SQL EXECUTE statements [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
lovasoa commented on PR #1490: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1490#issuecomment-2452910298 @alamb , what do you think? Can we merge this like that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Minor: make LeftJoinData into a struct in CrossJoinExec [datafusion]

2024-11-02 Thread via GitHub
findepi commented on code in PR #13227: URL: https://github.com/apache/datafusion/pull/13227#discussion_r1826521890 ## datafusion/physical-plan/src/joins/cross_join.rs: ## @@ -47,7 +47,15 @@ use async_trait::async_trait; use futures::{ready, Stream, StreamExt, TryStreamExt};

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
Rachelint commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826554952 ## datafusion/physical-plan/src/aggregates/group_values/mod.rs: ## @@ -143,8 +148,12 @@ pub fn new_group_values(schema: SchemaRef) -> Result> { }

Re: [PR] Minor: make LeftJoinData into a struct in CrossJoinExec [datafusion]

2024-11-02 Thread via GitHub
drauschenbach commented on code in PR #13227: URL: https://github.com/apache/datafusion/pull/13227#discussion_r1826582516 ## datafusion/physical-plan/src/joins/cross_join.rs: ## @@ -47,7 +47,15 @@ use async_trait::async_trait; use futures::{ready, Stream, StreamExt, TryStreamEx

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
2010YOUY01 commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826582961 ## datafusion/physical-plan/src/aggregates/group_values/column.rs: ## @@ -35,29 +37,113 @@ use datafusion_common::{not_impl_err, DataFusionError, Result}; use

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
2010YOUY01 commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826582961 ## datafusion/physical-plan/src/aggregates/group_values/column.rs: ## @@ -35,29 +37,113 @@ use datafusion_common::{not_impl_err, DataFusionError, Result}; use

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826569866 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
2010YOUY01 commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826584391 ## datafusion/physical-plan/src/aggregates/group_values/column.rs: ## @@ -35,29 +37,113 @@ use datafusion_common::{not_impl_err, DataFusionError, Result}; use

[PR] Update README for running benchmarks in k8s [datafusion-ray]

2024-11-02 Thread via GitHub
andygrove opened a new pull request, #39: URL: https://github.com/apache/datafusion-ray/pull/39 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
Rachelint commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826586152 ## datafusion/physical-plan/src/aggregates/group_values/column.rs: ## @@ -35,29 +37,113 @@ use datafusion_common::{not_impl_err, DataFusionError, Result}; use

Re: [PR] chore: Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader [datafusion-comet]

2024-11-02 Thread via GitHub
viirya commented on PR #1047: URL: https://github.com/apache/datafusion-comet/pull/1047#issuecomment-2452902504 Thanks @kazuyukitanimura -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader [datafusion-comet]

2024-11-02 Thread via GitHub
viirya closed issue #1048: Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader URL: https://github.com/apache/datafusion-comet/issues/1048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] DataFrame parse_sql_expr does not handle aliases [datafusion]

2024-11-02 Thread via GitHub
milenkovicm commented on issue #12518: URL: https://github.com/apache/datafusion/issues/12518#issuecomment-2452907458 My bad, I thought it's released -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] chore: Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader [datafusion-comet]

2024-11-02 Thread via GitHub
viirya merged PR #1047: URL: https://github.com/apache/datafusion-comet/pull/1047 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] Improve push down filter of join [datafusion]

2024-11-02 Thread via GitHub
Dandandan commented on PR #13184: URL: https://github.com/apache/datafusion/pull/13184#issuecomment-2453038663 I tracked this here https://github.com/apache/datafusion/issues/13232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
tbar4 commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453039406 > This is a random place to give this feedback, but eventually, I would like to be able to start the executor and scheduler processes from the Python bindings as well. This was

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
milenkovicm commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453049990 > This is a random place to give this feedback, but eventually, I would like to be able to start the executor and scheduler processes from the Python bindings as well. Th

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
tbar4 commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453052573 > > This is a random place to give this feedback, but eventually, I would like to be able to start the executor and scheduler processes from the Python bindings as well. This w

Re: [PR] Improve push down filter of join [datafusion]

2024-11-02 Thread via GitHub
Dandandan commented on PR #13184: URL: https://github.com/apache/datafusion/pull/13184#issuecomment-2452970127 > I think for this pushdown to be correct, the join type can be changed. > > I think this can be done in two phases: > > 1. if there is a filter for a left join on a ri

[PR] WIP: Add source release scripts [datafusion-ray]

2024-11-02 Thread via GitHub
andygrove opened a new pull request, #40: URL: https://github.com/apache/datafusion-ray/pull/40 This PR adds source release scripts, adapted from datafusion-python. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Switch to iterative `DynNode` and `ConcreteTreeNode` processing [datafusion]

2024-11-02 Thread via GitHub
blaginin commented on PR #13177: URL: https://github.com/apache/datafusion/pull/13177#issuecomment-2453061327 And one more question on a slightly related topic. Regardless of the option we choose here, we’ll still have the same "triangles" on the profiling charts, where we don’t optimize th

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
Rachelint commented on code in PR #12996: URL: https://github.com/apache/datafusion/pull/12996#discussion_r1826610308 ## datafusion/physical-plan/src/aggregates/group_values/column.rs: ## @@ -35,29 +37,113 @@ use datafusion_common::{not_impl_err, DataFusionError, Result}; use

Re: [PR] improve support for T-SQL EXECUTE statements [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
lovasoa commented on code in PR #1490: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1490#discussion_r1826520194 ## tests/sqlparser_common.rs: ## @@ -9730,6 +9734,41 @@ fn parse_call() { ); } +#[test] +fn parse_execute_stored_procedure() { +let expecte

Re: [PR] Implement predicate pruning for `like` expressions [datafusion]

2024-11-02 Thread via GitHub
findepi commented on PR #12978: URL: https://github.com/apache/datafusion/pull/12978#issuecomment-2452910830 Given LIKE can get desugared to something else during planning (eg https://github.com/apache/datafusion/pull/13061), maybe it would be great to support this functionality with end-t

Re: [PR] rewrite prefix and constant cases of like to startswith and equality [datafusion]

2024-11-02 Thread via GitHub
findepi commented on PR #13061: URL: https://github.com/apache/datafusion/pull/13061#issuecomment-2452910622 do we have equivalent of https://github.com/apache/datafusion/pull/12978 for starts_with? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
tbar4 commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453242893 @milenkovicm @andygrove I can't create tasks, so I am just cutting new feature requests. As for the following: > can we make standalone optional feature in pytho

Re: [PR] Deprecate invoke and invoke_no_args in favor of invoke_batch [datafusion]

2024-11-02 Thread via GitHub
findepi commented on PR #13174: URL: https://github.com/apache/datafusion/pull/13174#issuecomment-2453139285 https://github.com/apache/datafusion/pull/13179 deprecated functions in ScalarUDF, this PR deprecates equivalent functions in ScalarUDFImpl trait. @comphead please take a look

[I] Expose Ballista Scheduler and Executor in Python [datafusion-ballista]

2024-11-02 Thread via GitHub
tbar4 opened a new issue, #1107: URL: https://github.com/apache/datafusion-ballista/issues/1107 **Is your feature request related to a problem or challenge? Please describe what you are trying to do.** This feature request is following up on #1100 and #1091. Currently, if you want to cre

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
tbar4 commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453224544 @milenkovicm @andygrove New feature request added #1107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]

2024-11-02 Thread via GitHub
jayzhan211 commented on PR #12853: URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2453277957 I think this is ready to merge. Additional improvements can be made in a follow-up -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]

2024-11-02 Thread via GitHub
jayzhan211 merged PR #12853: URL: https://github.com/apache/datafusion/pull/12853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] TOP before ALL/DISTINCT [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
iffyio commented on code in PR #1495: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1495#discussion_r1826904031 ## src/dialect/mod.rs: ## @@ -590,6 +590,12 @@ pub trait Dialect: Debug + Any { fn supports_try_convert(&self) -> bool { false } + +

Re: [PR] Implement predicate pruning for `like` expressions [datafusion]

2024-11-02 Thread via GitHub
adriangb commented on PR #12978: URL: https://github.com/apache/datafusion/pull/12978#issuecomment-2453103852 > Given LIKE can get desugared to something else during planning (eg #13061), maybe it would be great to support this functionality with end-to-end tests. I feel like we need

[PR] README How to develop section now also works on Apple M1 [datafusion-python]

2024-11-02 Thread via GitHub
drauschenbach opened a new pull request, #940: URL: https://github.com/apache/datafusion-python/pull/940 # Which issue does this PR close? Closes #939. # Rationale for this change README previously contained instructions that only worked on Linux. # What changes are i

Re: [PR] rewrite prefix and constant cases of like to startswith and equality [datafusion]

2024-11-02 Thread via GitHub
adriangb commented on PR #13061: URL: https://github.com/apache/datafusion/pull/13061#issuecomment-2453104004 > do we have equivalent of #12978 for starts_with? No but I think it will have a lot of code sharing with #12978 and should be relatively easy to implement. After we merge tha

Re: [PR] Implement predicate pruning for `like` expressions [datafusion]

2024-11-02 Thread via GitHub
adriangb commented on code in PR #12978: URL: https://github.com/apache/datafusion/pull/12978#discussion_r1826715489 ## datafusion/core/src/physical_optimizer/pruning.rs: ## @@ -1610,6 +1629,124 @@ fn build_statistics_expr( Ok(statistics_expr) } +fn build_like_match( +

Re: [PR] Implement predicate pruning for `like` expressions [datafusion]

2024-11-02 Thread via GitHub
adriangb commented on code in PR #12978: URL: https://github.com/apache/datafusion/pull/12978#discussion_r1826718589 ## datafusion/core/src/physical_optimizer/pruning.rs: ## @@ -1610,6 +1629,126 @@ fn build_statistics_expr( Ok(statistics_expr) } +fn build_like_match( +

Re: [PR] fix q35 [datafusion-benchmarks]

2024-11-02 Thread via GitHub
onursatici commented on code in PR #17: URL: https://github.com/apache/datafusion-benchmarks/pull/17#discussion_r1826652831 ## tpcds/queries-spark/q35.sql: ## @@ -7,19 +7,19 @@ select cd_marital_status, cd_dep_count, count(*) cnt1, + min(cd_dep_count), max(cd_dep_c

Re: [PR] Switch to iterative `DynNode` and `ConcreteTreeNode` processing [datafusion]

2024-11-02 Thread via GitHub
peter-toth commented on PR #13177: URL: https://github.com/apache/datafusion/pull/13177#issuecomment-2453107732 Are you refering to visiting / transforming a node's children in parallel on multiple threads? Unfortunately, I'm not sure either if this idea was discussed earlier. I reme

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
milenkovicm commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453111513 @tbar4 would you please create task to expose scheduler and executor in python so we can track it? I'll check if everything is there on rust side once I finish #110

Re: [I] Remove call_ip in the SchedulerServer [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #1558: URL: https://github.com/apache/datafusion/issues/1558#issuecomment-2453114324 Thanks @drauschenbach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] [Python] Raise DataFusionError instead of plain Exceptions [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #494: URL: https://github.com/apache/datafusion/issues/494#issuecomment-2453114721 Seems like this was solved a while ago -- please reopen / file a new ticket if there is additional work to do. Thanks @drauschenbach for flagging this. -- This is an automated mess

Re: [I] [Python] Raise DataFusionError instead of plain Exceptions [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #494: [Python] Raise DataFusionError instead of plain Exceptions URL: https://github.com/apache/datafusion/issues/494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Remove call_ip in the SchedulerServer [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #1558: Remove call_ip in the SchedulerServer URL: https://github.com/apache/datafusion/issues/1558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] `ProjectionPushDown` creates invalid `Aggregate` plans [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #3285: `ProjectionPushDown` creates invalid `Aggregate` plans URL: https://github.com/apache/datafusion/issues/3285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] No field named when table name is quoted in FROM clause [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #2147: URL: https://github.com/apache/datafusion/issues/2147#issuecomment-2453115796 Seems to work now. Thank you @drauschenbach for flagging this ```sql > create table t1(id int) as values (1); 0 row(s) fetched. Elapsed 0.013 seconds. > select

Re: [I] Add support for GROUP BY Struct (Unsupported data type in hasher: Struct) [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #1952: URL: https://github.com/apache/datafusion/issues/1952#issuecomment-2453115478 THanks to @drauschenbach for flagging this. I double checked and it works now ```sql DataFusion CLI v42.1.0 > create table t as values ({a: 1, b:2}), ({a:2, b:3}), ({a:

Re: [I] Add support for GROUP BY Struct (Unsupported data type in hasher: Struct) [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #1952: Add support for GROUP BY Struct (Unsupported data type in hasher: Struct) URL: https://github.com/apache/datafusion/issues/1952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] No field named when table name is quoted in FROM clause [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #2147: No field named when table name is quoted in FROM clause URL: https://github.com/apache/datafusion/issues/2147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] invalid character_octat_length in information schema for tables created with values list [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #1271: invalid character_octat_length in information schema for tables created with values list URL: https://github.com/apache/datafusion/issues/1271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Documentation for running benchmarks with simd support does not work for me [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #1577: Documentation for running benchmarks with simd support does not work for me URL: https://github.com/apache/datafusion/issues/1577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Documentation for running benchmarks with simd support does not work for me [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #1577: URL: https://github.com/apache/datafusion/issues/1577#issuecomment-2453114989 Thanks @drauschenbach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Bug for CAST from Decimal(12, 2) to Utf8 in ballista_scheduler [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #1233: Bug for CAST from Decimal(12, 2) to Utf8 in ballista_scheduler URL: https://github.com/apache/datafusion/issues/1233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Information Schema Shows Tables from Other Catalogs [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #4408: Information Schema Shows Tables from Other Catalogs URL: https://github.com/apache/datafusion/issues/4408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] When the `to_timestamp_millis` function is used and the output format is csv, it will panic [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #4947: When the `to_timestamp_millis` function is used and the output format is csv, it will panic URL: https://github.com/apache/datafusion/issues/4947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] When the `to_timestamp_millis` function is used and the output format is csv, it will panic [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #4947: URL: https://github.com/apache/datafusion/issues/4947#issuecomment-2453116260 ```sql (venv-310) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs$ datafusion-cli --format csv DataFusion CLI v42.1.0 > SELECT to_timestamp_millis(1926632005177685347);

Re: [I] Panicked at InList does not yet support nested columns. [datafusion]

2024-11-02 Thread via GitHub
alamb closed issue #3778: Panicked at InList does not yet support nested columns. URL: https://github.com/apache/datafusion/issues/3778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Panicked at InList does not yet support nested columns. [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #3778: URL: https://github.com/apache/datafusion/issues/3778#issuecomment-2453115953 Thanks for checking @drauschenbach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] converted LexOrderingRef to &LexOrdering [datafusion]

2024-11-02 Thread via GitHub
jatin510 opened a new pull request, #13233: URL: https://github.com/apache/datafusion/pull/13233 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/13220. ## Rationale for this change ## What changes are included in this

Re: [I] Information Schema Shows Tables from Other Catalogs [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #4408: URL: https://github.com/apache/datafusion/issues/4408#issuecomment-2453116063 Thanks again to @drauschenbach for highlighting this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] converted LexOrderingRef to &LexOrdering [datafusion]

2024-11-02 Thread via GitHub
jatin510 commented on PR #13233: URL: https://github.com/apache/datafusion/pull/13233#issuecomment-2453116676 @alamb am i going in right directions ? as this PR changes lots of files ? 😅 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] `ProjectionPushDown` creates invalid `Aggregate` plans [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #3285: URL: https://github.com/apache/datafusion/issues/3285#issuecomment-2453115900 Thanks for flagging this @eejbyfeldt -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] When the `to_timestamp_millis` function is used and the output format is csv, it will panic [datafusion]

2024-11-02 Thread via GitHub
alamb commented on issue #4947: URL: https://github.com/apache/datafusion/issues/4947#issuecomment-2453116341 Thanks (again) to @drauschenbach for pointing this out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] README How to develop section now also works on Apple M1 [datafusion-python]

2024-11-02 Thread via GitHub
drauschenbach commented on PR #940: URL: https://github.com/apache/datafusion-python/pull/940#issuecomment-2453118397 ## Verification on an Apple M1 ```sh $ micromamba env create -f ./conda/environments/datafusion-dev.yaml -n datafusion-dev ... Transaction finished To

[I] The "How to develop" README instructions for Conda don't work on an Apple M1 [datafusion-python]

2024-11-02 Thread via GitHub
drauschenbach opened a new issue, #939: URL: https://github.com/apache/datafusion-python/issues/939 **Describe the bug** The development instructions, as written, are unnecessarily specific to Linux. **To Reproduce** On an Apple M1: ```sh $ micromamba env create -f ./c

Re: [PR] Update README for running benchmarks in k8s [datafusion-ray]

2024-11-02 Thread via GitHub
andygrove merged PR #39: URL: https://github.com/apache/datafusion-ray/pull/39 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

Re: [PR] Update README for running benchmarks in k8s [datafusion-ray]

2024-11-02 Thread via GitHub
andygrove commented on PR #39: URL: https://github.com/apache/datafusion-ray/pull/39#issuecomment-2453233342 Thanks for the review @edmondop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Add support for SHOW DATABASES/SCHEMAS/TABLES/VIEWS in Hive [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
yoavcloud commented on PR #1487: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1487#issuecomment-2453315843 @iffyio @alamb why isn't this merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[PR] build: Add build package workflow [datafusion-comet]

2024-11-02 Thread via GitHub
wangyum opened a new pull request, #1049: URL: https://github.com/apache/datafusion-comet/pull/1049 ## What changes are included in this PR? This PR adds build package workflow. Because some developers' development machines and running machines have different operating systems. After

Re: [I] Support timestamp literals with precision specifier [datafusion]

2024-11-02 Thread via GitHub
findepi commented on issue #7249: URL: https://github.com/apache/datafusion/issues/7249#issuecomment-2452989590 > Postgres supports an optional precision specifier in timestamp literals (e.g. `timestamp (3) '2021-01-01 00:00:00.123'` . I don't think this should be necessary. From

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826569866 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] Support vectorized append and compare for multi group by [datafusion]

2024-11-02 Thread via GitHub
Rachelint commented on PR #12996: URL: https://github.com/apache/datafusion/pull/12996#issuecomment-2452991852 > Is it possible to somehow unify GroupValuesColumn and VectorizedGroupValuesColumn ? 🤔I think It can unify simply, `VectorizedGroupValuesColumn::scalarized_intern` is simi

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826569866 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
milenkovicm commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2452989217 @andygrove your opinion please, would it make sense to rename `pyballista` to `datafusion_distributed` to align names with `datafusion_ray` ? -- This is an automated me

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
milenkovicm commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2452987277 great stuff @tbar4, two questions, it would be extra mile if we could do it: - can we make `standalone` optional feature in python as well ? - does py3o/python s

[PR] Support timestamp(n) type [datafusion]

2024-11-02 Thread via GitHub
findepi opened a new pull request, #13231: URL: https://github.com/apache/datafusion/pull/13231 Relates to https://github.com/apache/datafusion/discussions/13230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826569866 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826570840 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] Support timestamp(n) SQL type [datafusion]

2024-11-02 Thread via GitHub
caicancai commented on code in PR #13231: URL: https://github.com/apache/datafusion/pull/13231#discussion_r1826570840 ## datafusion/sql/src/planner.rs: ## @@ -466,7 +467,14 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { // Timestamp Without Time zone

Re: [PR] improve support for T-SQL EXECUTE statements [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
lovasoa commented on code in PR #1490: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1490#discussion_r1826574252 ## tests/sqlparser_common.rs: ## @@ -9730,6 +9734,41 @@ fn parse_call() { ); } +#[test] +fn parse_execute_stored_procedure() { +let expecte

Re: [PR] improve support for T-SQL EXECUTE statements [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
lovasoa commented on code in PR #1490: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1490#discussion_r1826574252 ## tests/sqlparser_common.rs: ## @@ -9730,6 +9734,41 @@ fn parse_call() { ); } +#[test] +fn parse_execute_stored_procedure() { +let expecte

[PR] Revert "Improve push down filter of join (#13184)" [datafusion]

2024-11-02 Thread via GitHub
eejbyfeldt opened a new pull request, #13229: URL: https://github.com/apache/datafusion/pull/13229 This reverts commit 7ae1ccb4971474d376e5bed1d7116fcaf23f906f. ## Which issue does this PR close? Closes #13211 ## Rationale for this change The pr #13184 introduc

Re: [I] PR: Improve push down filter of join #13184 introduces correctness issues [datafusion]

2024-11-02 Thread via GitHub
eejbyfeldt commented on issue #13211: URL: https://github.com/apache/datafusion/issues/13211#issuecomment-2452920773 Created https://github.com/apache/datafusion/pull/13229 for reverting -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] improve support for T-SQL EXECUTE statements [datafusion-sqlparser-rs]

2024-11-02 Thread via GitHub
iffyio commented on code in PR #1490: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1490#discussion_r1826563723 ## tests/sqlparser_common.rs: ## @@ -9730,6 +9734,41 @@ fn parse_call() { ); } +#[test] +fn parse_execute_stored_procedure() { +let expected

Re: [PR] Switch to iterative `DynNode` and `ConcreteTreeNode` processing [datafusion]

2024-11-02 Thread via GitHub
blaginin commented on PR #13177: URL: https://github.com/apache/datafusion/pull/13177#issuecomment-2453031208 > It looks like the 2 implementations are comparable. Maybe the recursive approach + stacker is a bit faster when it comes to large trees (879 vs 915 ms). That's true! I als

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
andygrove commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453033296 > @andygrove your opinion please, would it make sense to rename `pyballista` to `datafusion_distributed` to align name with `datafusion_ray` ? I think that we should

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-02 Thread via GitHub
andygrove commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2453033652 This is a random place to give this feedback, but eventually, I would like to be able to start the executor and scheduler processes from the Python bindings as well. This w

Re: [PR] Upgrade to Datafusion 42 [datafusion-ballista]

2024-11-02 Thread via GitHub
andygrove closed pull request #1059: Upgrade to Datafusion 42 URL: https://github.com/apache/datafusion-ballista/pull/1059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Upgrade to Datafusion 42 [datafusion-ballista]

2024-11-02 Thread via GitHub
andygrove commented on PR #1059: URL: https://github.com/apache/datafusion-ballista/pull/1059#issuecomment-2453033825 This upgrade has now happened in another PR. Thanks for starting this @palaska -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Improve push down filter of join [datafusion]

2024-11-02 Thread via GitHub
Dandandan commented on PR #13184: URL: https://github.com/apache/datafusion/pull/13184#issuecomment-2453034852 Hm actually, the transformation is already supported by `EliminateOuterJoin`, the only thing that's necessary is adding support for more expressions like `abs` like used in the exa