How is the pre-emption configured ?
On 21/4/20 1:41 am, Ilya Karpov wrote:
Hi, all,
recently I’ve noticed strange behaviour of YARN Fair Scheduler: 2 jobs
(i.e. two simultaneously started oozie launchers) started in a queue
with a small weight, and was not able to launch spark jobs while there
were plenty resources in other queues.
In details:
- hadoop(2.6, cdh 5.12) yarn with fair scheduler
- the queue(say, *small_queue*) with small weight starts 2 oozie
launcher jobs
- oozie launcher jobs occupies all small_queue capacity (even exceed
it by 1 core), and both ready to submit spark job (in the same queue =
small_queue)
- there is about 1/4 of free cluster resources in other queues (much
more then spark jobs require)
Expected behaviour: free resources of other queues will be given too
oozie launchers (from small_queue) to start their spark jobs
Actual behaviour: spark jobs were never started
Does anybody have an idea what prevented spark jobs from launch?
