[
https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820715#comment-17820715
]
Eyal Allweil commented on LIVY-863:
-----------------------------------
I don't have any suggestions, but I saw your question in this issue's PR and
that looks like a good idea.
I modified some [similar
code|https://github.com/apache/datafu/blob/main/datafu-spark/src/main/resources/pyspark_utils/bridge_utils.py#L47]
in [Apache DataFu|https://github.com/apache/datafu] to do a bogus import like
you suggested ({_}org.fake.fake{_}). Our tests pass with this change, so it
appears that non-existing imports are ignored.
Does that help?
> Missing JVM class imports for Spark3
> ------------------------------------
>
> Key: LIVY-863
> URL: https://issues.apache.org/jira/browse/LIVY-863
> Project: Livy
> Issue Type: Bug
> Components: REPL
> Reporter: Moritz
> Priority: Major
> Fix For: 0.9.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway
> imports some java classes for the user which have diverged from the ones in
> Spark 3. Some imports are missing which lead to methods like
> `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
> File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py",
> line 386, in explain
>
> print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(),
> explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
