milenkovicm opened a new pull request, #2748:
URL: https://github.com/apache/datafusion-ballista/pull/2748
# Which issue does this PR close?
Closes #.
Related to #2618.
# Rationale for this change
Since #2588 the Python client plugs into DataFusion Python natively. #2619
added Python UDFs, and #2637 added `BallistaSessionConfig`. Parts of the
README, the user guide and the example notebooks still described the old
wrapper-based client. For example, the quickstart said Python UDFs were not
implemented.
Python UDFs run only on a scheduler and executors started from Python
(`BallistaScheduler`, `BallistaExecutor`, `setup_test_cluster`), because only
those processes use the Python-aware codecs. Those processes had no S3 support:
the Rust binaries install the S3 object store registry and the `s3.*` config
extension, but pyballista did not. As a result, Python UDFs could not be used
on S3 data.
# What changes are included in this PR?
**S3 support for clusters started from Python**
- `BallistaScheduler`, `BallistaExecutor` and `setup_test_cluster` now
install `session_config_with_s3_support`, `session_state_with_s3_support` and
`runtime_env_with_s3_support`, the same as the `ballista-scheduler` and
`ballista-executor` binaries.
- pyballista enables the `build-binary` feature of `ballista-core`, which
provides `ballista_core::object_store`.
- New `test_s3.py`: starts RustFS (`rustfs/rustfs:1.0.0`, the image the Rust
example tests use since MinIO withdrew its images) with testcontainers. The
executors write parquet to S3, and the test reads it back and runs a Python UDF
over it. The test is skipped when Docker is not available.
**Documentation and examples**
- `quickstart.md`:
- Adds installation, the Ballista settings (`BallistaSessionConfig`, `SET
ballista.*`), Python UDFs and their cluster requirements, and starting the
scheduler and executor from Python.
- Notes that the scheduler address is only used when the first query runs.
- Replaces the outdated `explain` output.
- `jupyter.md`: adds the magic commands, `explain_visual`,
`collect_with_progress` and the `ballista[jupyter]` extra. Until now these were
only documented in the README.
- `README.md`: fixes `col(id)` in the DataFrame example. Documents the
scheduler and executor arguments, and links to the user guide instead of
repeating the Jupyter docs.
- `upgrading/56.0.0.md`: removes two claims that don't hold:
- `explain_visual(..., analyze=True)` was said to show runtime statistics;
it only renders the logical plan.
- `ballista.*` settings were said to be listed in
`information_schema.df_settings`; querying `information_schema` through a
Ballista context fails.
- Notebooks:
- use `ctx.session_id()`, since it is now a method;
- drop `print(df.explain())`, which printed `None`;
- connect to the test cluster port instead of a hard-coded one.
**`ballista.jupyter`**
- `%register` supports `json` and `avro`, in addition to `parquet` and `csv`.
- The help text no longer lists the `--no-display` and `--limit` options,
which never worked.
- The `%ballista status` message, the `explain_visual` and
`collect_with_progress` docstrings, and the `%ballista tables` output outside
IPython are corrected.
**Packaging**
- Removes the TestPyPI index for `datafusion`, because 55.0.0 is now on
PyPI, and re-locks `uv.lock`.
- Adds `testcontainers` to the `dev` group, the `test` extra and
`requirements.txt`.
- Adds Python 3.11 to 3.14 classifiers, and removes outdated comments in
`Cargo.toml`.
# Are there any user-facing changes?
Yes:
- A cluster started from Python can read and write `s3://` data, configured
with `SET s3.*` as for the Rust binaries, so Python UDFs can be used on S3 data.
- `%register` accepts `json` and `avro` files.
- The Python client documentation is updated.
There are no API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
