andygrove commented on code in PR #2493:
URL:
https://github.com/apache/datafusion-ballista/pull/2493#discussion_r4116315346
##########
ballista/core/src/planner.rs:
##########
@@ -293,6 +304,28 @@ mod test {
Ok(())
}
+ #[tokio::test]
+ async fn should_not_detect_table_scanned_in_subquery() -> Result<()> {
+ let ctx = context();
+ ctx.sql("CREATE TABLE big (name VARCHAR)")
+ .await?
+ .show()
+ .await?;
+ // Unoptimized, so the subquery is still an expression rather than the
+ // join the optimizer would decorrelate it into.
+ let plan = ctx
+ .state()
+ .create_logical_plan(
+ "SELECT table_name FROM information_schema.tables \
+ WHERE table_name IN (SELECT name FROM big)",
+ )
+ .await?;
+
+ assert!(!super::scans_only_local_tables(&plan));
Review Comment:
Agreed that the client's `information_schema` is the one to trust, and we
shouldn't answer it from the scheduler. I checked what happens today when a
plan like this is distributed. The client can't serialize the
`information_schema` scan, so it fails with `Error serializing custom table ...
LogicalExtensionCodec is not provided` before it reaches the scheduler. On
`main` this query already fails that way, because the optimizer turns the
subquery into a join and `LocalRun` then sees `big`.
For the query in the test, I went with refusing it rather than running it
locally. Running it on the client would also scan `big` there, without the
cluster. Queries that read only `information_schema` still run on the client as
before. A query that mixes it with other tables now fails with an error that
says why. The check is `scans_only_information_schema` now, and it looks inside
subqueries too. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
