villebro commented on code in PR #2577: URL: https://github.com/apache/datafusion-ballista/pull/2577#discussion_r4249318443
########## docs/source/user-guide/deployment/kubernetes.md: ########## @@ -392,6 +400,217 @@ Use the following command to set up port-forwarding. kubectl port-forward service/ballista-scheduler 50050:50050 ``` +## Serving Results to Clients Outside the Cluster + +Port forwarding the scheduler is enough to submit queries, but by default clients fetch query +results directly from the executors, and executor pods are not reachable from outside the cluster. +Deploy a result service that clients can reach, and have the scheduler point them at it. See +[Fetching Query Results](../scheduler.md#fetching-query-results) for how it works. + +Save the following as `result-service.yaml`: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: ballista-result-service +spec: + ports: + - port: 50055 + name: flight + # Plaintext HTTP/2, so a Gateway API route speaks gRPC to the pods. + appProtocol: kubernetes.io/h2c + selector: + app: ballista-result-service +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ballista-result-service +spec: + # Replicas are interchangeable: each forwards any fetch to the executor that + # holds the partition and keeps no state, so scale with result traffic. + replicas: 2 + # Start a replacement before stopping an old pod, so rollouts keep capacity. + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + selector: + matchLabels: + app: ballista-result-service + template: + metadata: + labels: + app: ballista-result-service + ballista-cluster: ballista + spec: + # Prefer placing replicas on different nodes, so they don't share a + # node's network bandwidth. + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app: ballista-result-service + # Leaves 10s of headroom after the preStop sleep (5s) and the drain + # bound (45s). + terminationGracePeriodSeconds: 60 + containers: + - name: ballista-result-service + image: <your-repo>/datafusion-ballista-result-service:latest + args: + - "--bind-port=50055" + # After SIGTERM, stop accepting connections and give in-flight + # result streams up to 45s to finish. + - "--graceful-shutdown-timeout-seconds=45" + lifecycle: + # Kubernetes removes a terminating pod from the Service's endpoints + # in parallel with shutting it down, and kube-proxy and gateways + # learn of the removal asynchronously, so new connections can still + # arrive for a few seconds. Sleeping keeps the pod accepting them + # until routing catches up; SIGTERM follows once the hook returns. + preStop: + sleep: + seconds: 5 + ports: + - containerPort: 50055 + name: flight + livenessProbe: + httpGet: + path: /healthz + port: 50055 + failureThreshold: 3 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /readyz + port: 50055 + failureThreshold: 3 + periodSeconds: 5 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: ballista-result-service +spec: + # Node drains evict at most one replica at a time. + minAvailable: 1 + selector: + matchLabels: + app: ballista-result-service +``` + +When a pod is replaced or moved to another node, Kubernetes removes it from the Service's +endpoints at the same time as it starts shutting the pod down. kube-proxy and any gateway pick up +that removal asynchronously, so for a few seconds new connections can still be routed to the pod. +The `preStop` sleep keeps the pod accepting connections during that window. On SIGTERM, which +follows the sleep, it stops accepting connections and lets in-flight result streams finish, for up +to `--graceful-shutdown-timeout-seconds` (10 by default), before exiting. The rolling-update +strategy and the PodDisruptionBudget keep at least one replica serving throughout. + +Then add `--advertise-flight-endpoint` to the scheduler container's `args` in `cluster.yaml`. Which +address to advertise depends on how clients reach the cluster. + +### Through port forwarding + +For a client on your own machine, forward a local port to the result service as well as to the +scheduler, and advertise the local address: + +```yaml +args: ["--bind-port=50050", "--advertise-flight-endpoint=127.0.0.1:50055"] +``` + +```bash +kubectl apply -f result-service.yaml +kubectl apply -f cluster.yaml +kubectl port-forward service/ballista-scheduler 50050:50050 & +kubectl port-forward service/ballista-result-service 50055:50055 & +``` + +`kubectl port-forward` sends all traffic to a single pod, so use this for testing only. + +### Through a TLS-terminating gateway + +In production, expose the result service through the [Gateway API] with a Gateway that terminates Review Comment: Agreed. daf5a257 now opens the gateway section with that: the result service doesn't authenticate fetches and connects to executors in plaintext, so the gateway should be reachable only by trusted clients, for example through an internal load balancer or the gateway's access restrictions. It also says TLS covers only the connections from clients to the gateway, and that `ballista.client.use_tls` needs a client built with a tonic TLS feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
