milenkovicm commented on PR #1568:
URL:
https://github.com/apache/datafusion-ballista/pull/1568#issuecomment-4308240156
> > thanks for your contribution @akshaychitneni
>
> >
>
> > FWIW, I believe its possible to get cluster state notifications in
current implementation:
>
> >
>
> > ```rust
>
> > let cluster = BallistaCluster::new_from_config(&config).await?;
>
> >
>
> > // this gets you a stream of `ClusterStateEvent`
>
> > cluster.cluster_state().cluster_state_events()
>
> > ```
>
> >
>
> > where `ClusterStateEvent` is defined as
>
> >
>
> > ```rust
>
> > #[derive(Debug, Clone, PartialEq)]
>
> > pub enum ClusterStateEvent {
>
> > /// An executor has been registered with the cluster.
>
> > RegisteredExecutor {
>
> > /// ID of the registered executor.
>
> > executor_id: String,
>
> > },
>
> > /// An executor has been removed from the cluster.
>
> > RemovedExecutor {
>
> > /// ID of the removed executor.
>
> > executor_id: String,
>
> > },
>
> > }
>
> > ```
>
> >
>
> > Having it as external stream gives possibility to run on different
thread than scheduler, having a listener on configuration might have better API.
>
> >
>
> > Let me know what you think
>
>
>
> Thanks for pointing out. I think the listener approach might be helpful
when running in Kubernetes-like environments, as callbacks run inline in the
scheduler's event loop, avoiding silent event loss scenarios that can occur
with a separate consumer thread.
Having listeners on scheduler loop may keep it busy handling scheduler
related tasks, like heart bits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
