andygrove opened a new issue, #2746:
URL: https://github.com/apache/datafusion-ballista/issues/2746
**Is your feature request related to a problem or challenge?**
With the default `rest-api` feature, `start_grpc_service` merges the REST
routes into the same axum router as the scheduler's gRPC services, so they are
served on the scheduler's gRPC port (`bind_host:bind_port`). The API has no
authentication, and `disable_rest_api` defaults to `false`. Besides read-only
routes such as `/api/jobs`, `/api/executors` and the job plans, it serves
`PATCH /api/job/{job_id}`, which cancels a job.
Every executor has to reach that port to register and report task status, so
the port is normally open to the whole cluster network, and often to more than
that in Kubernetes. Anything that can reach it can list every job with its
plan, and cancel any job, over plain HTTP. Apache DataFusion Comet hit this in
review of apache/datafusion-comet#7420 and now sets `disable_rest_api: true` in
its scheduler.
**Describe the solution you'd like**
Some options, which could be combined:
1. Serve the REST API on its own listener, with its own `rest_api_bind_host`
/ `rest_api_port`. The bind host would default to `127.0.0.1`, so exposing it
is an explicit choice and it can be firewalled separately from the gRPC port
that executors need.
2. Make the job-cancel route opt-in, for example `rest_api_allow_cancel`, so
the default API is read-only.
3. Default `disable_rest_api` to `true` for the library's
`SchedulerConfig::default()`, keeping it enabled in the `ballista-scheduler`
binary through its CLI default, so embedders opt in.
4. Allow an authentication hook (for example a tower layer supplied through
`SchedulerConfig`) for deployments that do expose it.
Option 1 alone keeps today's behavior for anyone running the scheduler
locally, and removes the exposure on the port executors use.
**Describe alternatives you've considered**
Documenting that `rest-api` should be disabled, or the port firewalled, in
untrusted networks. That doesn't help the common case, because executors need
the same port.
**Additional context**
Route setup: `ballista/scheduler/src/scheduler_process.rs`,
`start_grpc_service`. Routes: `ballista/scheduler/src/api/routes.rs`. The
executor serves only `/healthz` and `/readyz`, so this concerns only the
scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
