[
https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971798#comment-16971798
]
Thomas Graves commented on SPARK-29762:
---------------------------------------
this is actually more complex then you might think because the resource configs
are just configs. So you have spark.executor.resource.gpu.amount for instance,
the corresponding task config would be spark.task.resource.gpu.amount. Where
gpu could be any resource. The way the code is written now if it just grabs
all the resources and iterates over them in various places and assume you have
specified a task requirement for each executor resource.
If you remove that assumption you now have to be careful about what you are
iterating over and really you have to use the resources from the executor
configs, not the task configs. But you still have to read the task configs and
if a resource isn't there then default it to 1.
> GPU Scheduling - default task resource amount to 1
> --------------------------------------------------
>
> Key: SPARK-29762
> URL: https://issues.apache.org/jira/browse/SPARK-29762
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
>
> Default the task level resource configs (for gpu/fpga, etc) to 1. So if the
> user specifies the executor resource then to make it more user friendly lets
> have the task resource config default to 1. This is ok right now since we
> require resources to have an address. It also matches what we do for the
> spark.task.cpus configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
