Asif Khatri created LIVY-978:
--------------------------------
Summary: Update the application tag generation logic in Livy
Key: LIVY-978
URL: https://issues.apache.org/jira/browse/LIVY-978
Project: Livy
Issue Type: Bug
Components: Server
Affects Versions: 0.7.0
Reporter: Asif Khatri
Assignee: Asif Khatri
The following issue occurs:
{code:java}
ERROR org.apache.livy.utils.SparkYarnApp: Error whiling refreshing YARN state
java.lang.IllegalStateException: No YARN application is found with tag
livy-batch-2-per2hlwa in 120 seconds. This may be because 1) spark-submit fail
to submit application to YARN; or 2) YARN cluster doesn't have enough resources
to start the application in time. Please check Livy log and YARN log to know
the details.
{code}
if the below configuration parameter is set in YARN:
{code:java}
yarn.resourcemanager.application-tag-based-placement.force-lowercase=false
{code}
We can fix this in SparkYarnApp by updating app tag generation logic to
generate appTag always in lowercase characters. SparkYarnApp code can be changed
from
{code:java}
val appTag = s"livy-session-$id-${Random.alphanumeric.take(8).mkString}"
{code}
to
{code:java}
val appTag =
s"livy-session-$id-${Random.alphanumeric.take(8).mkString}".toLowerCase()
{code}
at
[https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L92]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
