andygrove opened a new issue, #1582: URL: https://github.com/apache/datafusion-ballista/issues/1582
**Describe the bug**
I am running in k8s and see that queries run entirely in the client pod.
There is no activity in executors.
I can work around this with ` df._to_internal_df().collect()`.
Claude claims that this is a recent regression caused by the Jupyter PR.
```
What commit 6091b525 (feat: jupyter notebook support, Apr 9) broke:
Before that commit, DistributedDataFrame used a metaclass
(RedefiningDataFrameMeta) that automatically wrapped execution methods to route
through _to_internal_df():
OVERRIDDEN_EXECUTION_METHODS = [
"show", "count", "collect", "collect_partitioned",
"write_json", "to_arrow_table", "to_pandas", ...
]
The Jupyter PR changed the metaclass to plain metaclass=type, silently
removing all that wrapping. So collect(), show(), to_pandas() etc. now run
locally. Only write_csv() and write_parquet() were manually
re-added as overrides.
The tests in test_context.py still call df.collect() directly — they were
written expecting the old behavior and presumably now pass only because they're
running against a local/standalone cluster where
"local" and "cluster" are the same process.
```
**To Reproduce**
**Expected behavior**
A clear and concise description of what you expected to happen.
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
