andygrove opened a new issue, #2720: URL: https://github.com/apache/datafusion-ballista/issues/2720
**Describe the bug** The chaos test harness can hand two executors overlapping ports, so a cluster fails to start with `timed out waiting for 2 executors to register` before the scenario runs. `TestCluster::spawn_executor` (`chaos-testing/src/cluster.rs`) calls `free_ports::<2>()` once per executor. `free_ports` keeps the two ports of one call distinct, but drops its listeners before returning, and the executor child only binds them some milliseconds later. When the next executor is spawned in that window, its own `free_ports` call can get one of the ports the previous executor has not bound yet. Seen in CI on #2643 (`killing_every_executor_terminates_the_job::case_1_aqe_off`, run https://github.com/apache/datafusion-ballista/actions/runs/41018367092): - executor-0: flight port `43945`, gRPC port `35773` - executor-1: flight port `35773`, gRPC port `41943` executor-1 registered, then its Flight server failed with `AddrInUse` on `35773` because executor-0's gRPC server had bound it first. It stopped, the scheduler removed it, and `await_executors(2)` timed out: ``` ERROR ballista_executor::executor_process] Could not bind built-in arrow flight server. INFO ballista_executor::executor_process] "executor services stopped with reason Err(IoError(Os { code: 98, kind: AddrInUse, message: \"Address already in use\" }))" ``` The comment on `free_ports` says it is "still racy against the rest of the machine", but this race is between executors of the same cluster, not other processes. **To Reproduce** Intermittent. Run the `ballista-chaos` `ha` tests repeatedly on a busy runner, for example `cargo test -p ballista-chaos --test ha`. **Expected behavior** Executors in one cluster always get distinct ports, so cluster startup never fails with `AddrInUse`. **Additional context** A few options: - Reserve every executor's ports in one `free_ports` call before spawning any of them, so all ports of a cluster are distinct (a restarted executor would still need its own call). - Retry `spawn_executor` with fresh ports when the child exits with `AddrInUse` during startup. - Have executors bind `:0` and report the ports they got, which removes the race entirely but needs an executor-side hook. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
