[
https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972661#comment-16972661
]
Thomas Graves commented on SPARK-29762:
---------------------------------------
The code that gets the task requirements is generic and is reused by more then
just TASKs. Also there is code that loops over the task requirements and assume
all task requirements are there, but you can't do that if you are relying on a
default value because they won't be present. At some point you either have to
look at all executor requirements and then build up the task requirements that
include the default ones, or you just have to change those loops to be over the
executor requirements.
I'm not saying its not possible, my comment was just saying it isn't as
straight forward as I originally thought.
I don't think you have the issue you are talking about because you have to have
the task requirements be based on the executor resource requests (which is part
of complication). as long as you do that with the stage level scheduling I
think its ok. ResourceProfile ExecutorRequest = 1 cpu, 1 GPU, user defined
taskRequest = null, then translate into taskRequest = 1 cpu, 1 GPU.
> GPU Scheduling - default task resource amount to 1
> --------------------------------------------------
>
> Key: SPARK-29762
> URL: https://issues.apache.org/jira/browse/SPARK-29762
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
>
> Default the task level resource configs (for gpu/fpga, etc) to 1. So if the
> user specifies the executor resource then to make it more user friendly lets
> have the task resource config default to 1. This is ok right now since we
> require resources to have an address. It also matches what we do for the
> spark.task.cpus configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
