This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new f311677 [ZEPPELIN-4441]. Update interpreter overview page
f311677 is described below
commit f311677126baa55266c524c9ff8859ca84ebf564
Author: Jeff Zhang <zjf...@apache.org>
AuthorDate: Sun Nov 10 15:22:23 2019 +0800
[ZEPPELIN-4441]. Update interpreter overview page
### What is this PR for?
This PR is to polish the interpreter overview page.
### What type of PR is it?
[ Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4441
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zjf...@apache.org>
Closes #3541 from zjffdu/ZEPPELIN-4441 and squashes the following commits:
c360fec52 [Jeff Zhang] [ZEPPELIN-4441]. Update interpreter overview page
---
.../zeppelin/img/screenshots/conf_interpreter.png | Bin 39388 -> 54051 bytes
.../img/screenshots/interpreter_binding.png | Bin 240869 -> 0 bytes
docs/usage/interpreter/overview.md | 103 +++++++++------------
3 files changed, 45 insertions(+), 58 deletions(-)
diff --git a/docs/assets/themes/zeppelin/img/screenshots/conf_interpreter.png
b/docs/assets/themes/zeppelin/img/screenshots/conf_interpreter.png
index 156c357..bc15298 100644
Binary files a/docs/assets/themes/zeppelin/img/screenshots/conf_interpreter.png
and b/docs/assets/themes/zeppelin/img/screenshots/conf_interpreter.png differ
diff --git
a/docs/assets/themes/zeppelin/img/screenshots/interpreter_binding.png
b/docs/assets/themes/zeppelin/img/screenshots/interpreter_binding.png
deleted file mode 100644
index 536783c..0000000
Binary files
a/docs/assets/themes/zeppelin/img/screenshots/interpreter_binding.png and
/dev/null differ
diff --git a/docs/usage/interpreter/overview.md
b/docs/usage/interpreter/overview.md
index ef2eda9..4fbbf37 100644
--- a/docs/usage/interpreter/overview.md
+++ b/docs/usage/interpreter/overview.md
@@ -27,25 +27,30 @@ limitations under the License.
In this section, we will explain the role of interpreters, interpreter groups
and interpreter settings in Zeppelin.
The concept of Zeppelin interpreters allows any language or data-processing
backend to be plugged into Zeppelin.
-Currently, Zeppelin supports many interpreters such as Scala (with Apache
Spark), Python (with Apache Spark), Spark SQL, JDBC, Markdown, Shell and so on.
+Currently, Zeppelin supports many interpreters such as Scala (with Apache
Spark), Python (with Apache Spark), Spark SQL, Hive, JDBC, Markdown, Shell and
so on.
-## What are Zeppelin interpreters?
+## What are Zeppelin Interpreters ?
A Zeppelin interpreter is a plug-in which enables Zeppelin users to use a
specific language/data-processing-backend. For example, to use Scala code in
Zeppelin, you would use the `%spark` interpreter.
When you click the ```+Create``` button on the interpreter page, the
interpreter drop-down list box will show all the available interpreters on your
server.
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_create.png"
width="280px">
-## What are the interpreter settings?
+You can create multiple interpreters for the same engine with different
interpreter setting. e.g. You can create `spark2` for Spark 2.x and create
`spark1` for Spark 1.x.
+
+For each paragraph you write in Zeppelin, you need to specify its interpreter
first via `%interpreter_group.interpreter_name`. e.g. `%spark.pyspark`,
`%spark.r`
+
+## What are the Interpreter Settings?
The interpreter settings are the configuration of a given interpreter on the
Zeppelin server. For example, certain properties need to be set for the Apache
Hive JDBC interpreter to connect to the Hive server.
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_setting.png"
width="500px">
-Properties are exported as environment variables on the system if the property
name consists of upper-case characters, numbers or underscores ([A-Z_0-9]).
Otherwise, the property is set as a JVM property.
+Properties are exported as environment variables on the system if the property
name consists of upper-case characters, numbers or underscores ([A-Z_0-9]).
Otherwise, the property is set as a common interpreter property.
+e.g. You can define `SPARK_HOME` and `HADOOP_CONF_DIR` in spark's interpreter
setting, they are be passed to Spark interpreter process as environment
variable which is used by Spark.
-You may use parameters from the context of the interpreter by adding
#{contextParameterName} in the value. The parameter can be of the following
types: string, number, boolean.
+You may use parameters from the context of the interpreter by adding
#{contextParameterName} in the interpreter property value. The parameter can be
of the following types: string, number, boolean.
-###### Context parameters
+### Context Parameters
<table class="table-configuration">
<tr>
<th>Name</th>
@@ -69,20 +74,14 @@ You may use parameters from the context of the interpreter
by adding #{contextPa
</tr>
</table>
-If the context parameter is null, then it is replaced by an empty string.
+If the context parameter is null, then it is replaced by an empty string. The
following screenshot is one example where we make the user name as the property
value of `default.user`.
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_setting_with_context_parameters.png"
width="800px">
-<br>
-Each notebook can be bound to multiple Interpreter Settings using the setting
icon in the upper right corner of the notebook.
-
-<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_binding.png"
width="800px">
-
-
-## What are interpreter groups?
-Every interpreter belongs to an **Interpreter Group**. Interpreter Groups are
units of interpreters that can be started/stopped together.
-By default, every interpreter belongs to a separate group, but the group might
contain more interpreters. For example, the Spark interpreter group includes
Spark support, pySpark, Spark SQL and the dependency loader.
+## What are Interpreter Groups ?
+Every interpreter belongs to an **Interpreter Group**. Interpreter Groups are
units of interpreters that run in one single JVM process and can be
started/stopped together.
+By default, every interpreter belongs to a separate group, but the group might
contain more interpreters. For example, the Spark interpreter group includes
Scala Spark, PySpark, IPySpark, SparkR and Spark SQL.
Technically, Zeppelin interpreters from the same group run within the same
JVM. For more information about this, please consult [the documentation on
writing interpreters](../development/writing_zeppelin_interpreter.html).
@@ -91,65 +90,44 @@ Each interpreter belongs to a single group and is
registered together. All relev
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_setting_spark.png"
width="500px">
-## Interpreter binding mode
+## Interpreter Binding Mode
-In the Interpreter Settings, one can choose one of the 'shared', 'scoped', or
'isolated' interpreter binding modes.
-In 'shared' mode, every notebook bound to the Interpreter Setting will share a
single Interpreter instance. In 'scoped' mode, each notebook will create a new
interpreter instance in the same interpreter process. In 'isolated' mode, each
notebook will create new a interpreter process.
+In the Interpreter Settings, one can choose one of the `shared`, `scoped`, or
`isolated` interpreter binding modes.
+In `shared` mode, every note/user using this interpreter will share a single
interpreter instance.
+`scoped` and `isolated` mode can be used under 2 dimensions: `per user` or
`per note`.
+e.g. In `scoped per note` mode, each note will create a new interpreter
instance in the same interpreter process. In `isolated per note` mode, each
note will create a new interpreter process.
For more information, please consult [Interpreter Binding
Mode](./interpreter_binding_mode.html).
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_persession.png"
width="400px">
-
-
-## Connecting to the existing remote interpreter
-
-Zeppelin users can start interpreter threads embedded in their service. This
provides users with the flexibility of starting interpreters on a remote host.
To start an interpreter along with your service you have to create an instance
of ``RemoteInterpreterServer`` and start it as follows:
-
-```java
-RemoteInterpreterServer interpreter=new RemoteInterpreterServer(3678);
-// Here, 3678 is the port on which interpreter will listen.
-interpreter.start();
-
-```
-
-The above code will start an interpreter thread inside your process. Once the
interpreter has started, you can configure Zeppelin to connect to
RemoteInterpreter by checking the **Connect to existing process** checkbox and
then provide the **Host** and **Port** on which interpreter process is
listening, as shown in the image below:
-
-<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/existing_interpreter.png"
width="450px">
-
-## Precode
-
-Snippet of code (language of interpreter) that executes after initialization
of the interpreter depends on [Binding mode](#interpreter-binding-mode). To
configure, add a parameter with the class of the interpreter
(`zeppelin.<ClassName>.precode`) except JDBCInterpreter ([JDBC
precode](../../interpreter/jdbc.html#usage-precode)).
-
-<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_precode.png"
width="800px">
-
## Interpreter Lifecycle Management
-Before 0.8.0, Zeppelin didn't have lifecycle management for interpreters.
Users had to shut down interpreters explicitly via the UI. Starting from 0.8.0,
Zeppelin provides a new interface
-`LifecycleManager` to control the lifecycle of interpreters. For now, there
are two implementations: `NullLifecycleManager` and `TimeoutLifecycleManager`,
which is the default.
+Before 0.8.0, Zeppelin doesn't have lifecycle management for interpreters.
Users had to shut down interpreters explicitly via the UI. Starting from 0.8.0,
Zeppelin provides a new interface
+`LifecycleManager` to control the lifecycle of interpreters. For now, there
are two implementations: `NullLifecycleManager` and `TimeoutLifecycleManager`
which is the default.
`NullLifecycleManager` will do nothing, i.e., the user needs to control the
lifecycle of interpreter by themselves as before. `TimeoutLifecycleManager`
will shut down interpreters after an interpreter remains idle for a while. By
default, the idle threshold is 1 hour.
Users can change this threshold via the
`zeppelin.interpreter.lifecyclemanager.timeout.threshold` setting.
`TimeoutLifecycleManager` is the default lifecycle manager, and users can
change it via `zeppelin.interpreter.lifecyclemanager.class`.
-## Inline Generic ConfInterpreter
+## Inline Generic Configuration
-Zeppelin's interpreter setting is shared by all users and notes, if you want
to have different settings, you have to create a new interpreter, e.g. you can
create `spark_jar1` for running Spark with dependency jar1 and `spark_jar2` for
running Spark with dependency jar2.
-This approach works, but is not particularly convenient. `ConfInterpreter` can
provide more fine-grained control on interpreter settings and more flexibility.
+Zeppelin's interpreter setting is shared by all users and notes, if you want
to have different settings, you have to create a new interpreter, e.g. you can
create `spark_jar1` for running Spark with dependency `jar1` and `spark_jar2`
for running Spark with dependency `jar2`.
+This approach works, but is not convenient. Inline generic configuration can
provide more fine-grained control on interpreter settings and more flexibility.
-`ConfInterpreter` is a generic interpreter that can be used by any
interpreter. The input format should be the property file format.
-It can be used to make custom settings for any interpreter. However,
`ConfInterpreter` needs to be run before that interpreter process is launched.
When that interpreter process is launched is determined by the interpreter mode
setting.
-So users need to understand the [interpreter mode
setting](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and
be aware of when the interpreter process is launched. E.g., if we set the Spark
interpreter setting as isolated per note, then, under this setting, each note
will launch one interpreter process.
-In this scenario, users need to put `ConfInterpreter` as the first paragraph
as in the below example. Otherwise, the customized setting cannot be applied
(actually it would report ERROR).
-<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/conf_interpreter.png"
width="700px">
+`ConfInterpreter` is a generic interpreter that can be used by any
interpreter. You can use it just like defining a java property file.
+It can be used to make custom settings for any interpreter. However,
`ConfInterpreter` needs to run before that interpreter process is launched.
When that interpreter process is launched is determined by the interpreter
binding mode setting.
+So users need to understand the [interpreter binding mode
setting](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and
be aware of when the interpreter process is launched. E.g., if we set the Spark
interpreter setting as isolated per note, then under this setting, each note
will launch one interpreter process.
+In this scenario, users need to put `ConfInterpreter` as the first paragraph
as in the below example. Otherwise, the customized setting cannot be applied
(actually it would report `ERROR`).
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/conf_interpreter.png"
width="600px">
-## Interpreter Process Recovery
-Before 0.8.0, shutting down Zeppelin also meant to shutdown all the running
interpreter processes. Usually, an administrator will shutdown the Zeppelin
server for maintenance or upgrades, but would not want to shut down the running
interpreter processes.
-In such cases, interpreter process recovery is necessary. Starting from 0.8.0,
users can enable interpreter process recovery via the setting
`zeppelin.recovery.storage.class` as
-`org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other
implementations if available in the future. By default it is
`org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage`,
- which means recovery is not enabled. Enabling recovery means shutting down
Zeppelin would not terminate interpreter processes, and when Zeppelin is
restarted, it would try to reconnect to the existing running interpreter
processes. If you want to kill all the interpreter processes after terminating
Zeppelin even when recovery is enabled, you can run `bin/stop-interpreter.sh`
+## Precode
+
+Snippet of code (language of interpreter) that executes after initialization
of the interpreter depends on [Binding mode](#interpreter-binding-mode). To
configure, add a parameter with the class of the interpreter
(`zeppelin.<ClassName>.precode`) except JDBCInterpreter ([JDBC
precode](../../interpreter/jdbc.html#usage-precode)).
+
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/interpreter_precode.png"
width="800px">
## Credential Injection
@@ -162,7 +140,16 @@ Credentials from the credential manager can be injected
into Notebooks. Credenti
<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/credential_entry.png"
width="500px">
**Credential Injection Example**
-```
+
+```scala
val password = "{password.SOME_CREDENTIAL_ENTITY}"
+
val username = "{user.SOME_CREDENTIAL_ENTITY}"
```
+
+## Interpreter Process Recovery (Experimental)
+
+Before 0.8.0, shutting down Zeppelin also meant to shutdown all the running
interpreter processes. Usually, an administrator will shutdown the Zeppelin
server for maintenance or upgrades, but would not want to shut down the running
interpreter processes.
+In such cases, interpreter process recovery is necessary. Starting from 0.8.0,
users can enable interpreter process recovery via the setting
`zeppelin.recovery.storage.class` as
+`org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other
implementations if available in the future. By default it is
`org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage`,
+ which means recovery is not enabled. Enabling recovery means shutting down
Zeppelin would not terminate interpreter processes, and when Zeppelin is
restarted, it would try to reconnect to the existing running interpreter
processes. If you want to kill all the interpreter processes after terminating
Zeppelin even when recovery is enabled, you can run `bin/stop-interpreter.sh`
