This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 0566354 [ZEPPELIN-4375]. Add more flink tutorial notes
0566354 is described below
commit 056635409a39ba7916a4865cc75c4d712dda21a6
Author: Jeff Zhang <[email protected]>
AuthorDate: Thu Oct 17 19:33:51 2019 +0800
[ZEPPELIN-4375]. Add more flink tutorial notes
### What is this PR for?
This PR add 2 tutorial note for flink，one for batch，another for streaming.
Besides that, I also update the flink documentation for some improvement.
### What type of PR is it?
[ Improvement | Documentation ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4375
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <[email protected]>
Closes #3481 from zjffdu/ZEPPELIN-4375 and squashes the following commits:
d1ee659b0 [Jeff Zhang] address comment
1e8db163a [Jeff Zhang] [ZEPPELIN-4375]. Add more flink tutorial notes
---
docs/interpreter/flink.md | 35 +-
.../Flink Batch Tutorial_2EN1E1ATY.zpln | 602 +++++++++++++++++++++
.../Flink Stream Tutorial_2ER62Y5VJ.zpln | 427 +++++++++++++++
3 files changed, 1061 insertions(+), 3 deletions(-)
diff --git a/docs/interpreter/flink.md b/docs/interpreter/flink.md
index ad71487..3fc75c9 100644
--- a/docs/interpreter/flink.md
+++ b/docs/interpreter/flink.md
@@ -78,7 +78,7 @@ You can also set other flink properties which are not listed
in the table. For a
<tr>
<td>flink.execution.mode</td>
<td>local</td>
- <td>Execution mode of flink, e.g. local/yarn/remote</td>
+ <td>Execution mode of flink, e.g. local | yarn | remote</td>
</tr>
<tr>
<td>flink.execution.remote.host</td>
@@ -148,13 +148,18 @@ You can also set other flink properties which are not
listed in the table. For a
<tr>
<td>zeppelin.flink.planner</td>
<td>blink</td>
- <td>planner or flink table api, blink or flink</td>
+ <td>planner of flink table api, <em>blink</em> or <em>flink</em></td>
</tr>
<tr>
<td>zeppelin.pyflink.python</td>
<td>python</td>
<td>python executable for pyflink</td>
</tr>
+ <tr>
+ <td>HADOOP_CONF_DIR</td>
+ <td></td>
+ <td>location of hadoop conf, this is must be set if running in yarn
mode</td>
+ </tr>
</table>
@@ -166,6 +171,25 @@ Zeppelin will create 4 variables to represent flink's
entrypoint:
* `stenv` (StreamTableEnvironment)
* `btenv` (BatchTableEnvironment)
+## Flink Planner
+
+Starting from Flink 1.9, there're 2 planners supported by Flink's table api:
`flink` & `blink`.
+* If you want to use DataSet api, then please use flink planner (specify
`zeppelin.flink.planner` to `flink`).
+* In other cases, we would always recommend you to use `blink` planner which
is also the default value of `zeppelin.flink.planner`.
+
+## How to use Hive
+
+In order to use Hive in Flink, you have to do several setting.
+* Set `zeppelin.flink.enableHive` to `true`
+* Copy necessary dependencies to flink's lib folder, check this
[link](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/hive/#depedencies)
for more details
+ * flink-connector-hive_{scala_version}-{flink.version}.jar
+ * flink-hadoop-compatibility_{scala_version}-{flink.version}.jar
+ * flink-shaded-hadoop-2-uber-{hadoop.version}-{flink-shaded.version}.jar
+ * hive-exec-2.x.jar (for Hive 1.x, you need to copy hive-exec-1.x.jar,
hive-metastore-1.x.jar, libfb303-0.9.2.jar and libthrift-0.9.2.jar)
+* Specify `HIVE_CONF_DIR` either in flink interpreter setting or
`zeppelin-env.sh`
+* Specify `zeppelin.flink.hive.version`, by default it is 2.3.4. If you are
using Hive 1.2.x, then you need to set it as `1.2.2`
+
+After these settings, you will be able to query hive table via either table
api `%flink` or batch sql `%flink.bsql`
## ZeppelinContext
Zeppelin automatically injects `ZeppelinContext` as variable `z` in your
Scala/Python environment. `ZeppelinContext` provides some additional functions
and utilities.
@@ -173,8 +197,13 @@ See
[Zeppelin-Context](../usage/other_features/zeppelin_context.html) for more d
## IPython support
-By default, zeppelin would use IPython in `pyflink` when IPython is available,
Otherwise it would fall back to the original PyFlink implementation.
+By default, zeppelin would use IPython in `%flink.pyflink` when IPython is
available, Otherwise it would fall back to the original python implementation.
If you don't want to use IPython, then you can set
`zeppelin.pyflink.useIPython` as `false` in interpreter setting. For the
IPython features, you can refer doc
[Python Interpreter](python.html)
+## Tutorial Notes
+
+Zeppelin is shipped with several Flink tutorial notes which may be helpful for
you.
+
+
diff --git a/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln
b/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln
new file mode 100644
index 0000000..018089e
--- /dev/null
+++ b/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln
@@ -0,0 +1,602 @@
+{
+ "paragraphs": [
+ {
+ "title": "Introduction",
+ "text": "%md\n\nThis is a tutorial note for Flink batch scenario (`To be
noticed, you need to use flink 1.9 or afterwards`), . You can run flink scala
api via `%flink` and run flink batch sql via `%flink.bsql`. \nThis note use
flink\u0027s DataSet api to demonstrate flink\u0027s batch capablity. DataSet
is only supported by flink planner, so here we have to specify the planner
`zeppelin.flink.planner` as `flink`, otherwise it would use blink planner by
default which doesn\u0027t su [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:15:54.910",
+ "config": {
+ "tableHide": false,
+ "editorSetting": {
+ "language": "markdown",
+ "editOnDblClick": true,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/markdown",
+ "fontSize": 9.0,
+ "editorHide": true,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "HTML",
+ "data": "\u003cdiv
class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial note for
Flink batch scenario (\u003ccode\u003eTo be noticed, you need to use flink 1.9
or afterwards\u003c/code\u003e), . You can run flink scala api via
\u003ccode\u003e%flink\u003c/code\u003e and run flink batch sql via
\u003ccode\u003e%flink.bsql\u003c/code\u003e.\u003cbr/\u003eThis note use
flink\u0026rsquo;s DataSet api to demonstrate flink\u0026rsquo;s batch
capablity. DataSet is onl [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641095_-188362229",
+ "id": "paragraph_1547794482637_957545547",
+ "dateCreated": "2019-09-26 17:20:41.095",
+ "dateStarted": "2019-10-08 15:15:54.910",
+ "dateFinished": "2019-10-08 15:15:54.923",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Configure Flink Interpreter",
+ "text": "%flink.conf\n\nFLINK_HOME \u003cFLINK_INSTALLATION\u003e\n#
DataSet is only supported in flink planner, so here we use flink planner. By
default it is blink planner\nzeppelin.flink.planner flink\n\n\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:37:39.948",
+ "config": {
+ "editorSetting": {
+ "language": "text",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/text",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": []
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641097_1856810430",
+ "id": "paragraph_1546565092490_1952685806",
+ "dateCreated": "2019-09-26 17:20:41.097",
+ "dateStarted": "2019-10-11 10:00:42.031",
+ "dateFinished": "2019-10-11 10:00:42.052",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%sh\n\ncd /tmp\nwget
https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\n";,
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:16:45.127",
+ "config": {
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "colWidth": 12.0,
+ "fontSize": 9.0,
+ "enabled": true,
+ "results": {},
+ "editorSetting": {
+ "language": "sh",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "editorMode": "ace/mode/sh"
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "--2019-10-08 15:16:46--
https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\nResolving
s3.amazonaws.com (s3.amazonaws.com)... 54.231.120.2\nConnecting to
s3.amazonaws.com (s3.amazonaws.com)|54.231.120.2|:443... connected.\nHTTP
request sent, awaiting response... 200 OK\nLength: 461474 (451K)
[application/octet-stream]\nSaving to: \u0027bank.csv\u0027\n\n 0K
.......... .......... .......... .......... .......... 11% 91.0K 4s\n 50K
.......... ...... [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489915993_-1377803690",
+ "id": "paragraph_1569489915993_-1377803690",
+ "dateCreated": "2019-09-26 17:25:15.993",
+ "dateStarted": "2019-10-08 15:16:45.132",
+ "dateFinished": "2019-10-08 15:16:50.387",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Load Bank Data",
+ "text": "%flink\n\nval bankText \u003d
benv.readTextFile(\"/tmp/bank.csv\")\nval bank \u003d bankText.map(s
\u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d
\"\\\"age\\\"\").map(\n s \u003d\u003e (s(0).toInt,\n
s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n
s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\",
\"\").toInt\n )\n )\n\nbtenv.registerDataSet(\"bank\", bank,
\u0027age, \u0027j [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:01:44.897",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 111.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "\u001b[1m\u001b[34mbankText\u001b[0m:
\u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[String]\u001b[0m \u003d
org.apache.flink.api.scala.DataSet@683ccfd\n\u001b[1m\u001b[34mbank\u001b[0m:
\u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[(Int, String, String,
String, Int)]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@35e6c08f\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641099_-2030350664",
+ "id": "paragraph_1546584347815_1533642635",
+ "dateCreated": "2019-09-26 17:20:41.099",
+ "dateStarted": "2019-10-11 10:01:44.904",
+ "dateFinished": "2019-10-11 10:02:12.333",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\ndescribe bank\n\n\n\n\n\n\n\n\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:07:51.800",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "table": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ }
+ },
+ "commonSetting": {}
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data":
"Column\tType\nOptional[age]\tOptional[INT]\nOptional[job]\tOptional[STRING]\nOptional[marital]\tOptional[STRING]\nOptional[education]\tOptional[STRING]\nOptional[balance]\tOptional[INT]\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491358206_1385745589",
+ "id": "paragraph_1569491358206_1385745589",
+ "dateCreated": "2019-09-26 17:49:18.206",
+ "dateStarted": "2019-10-11 10:07:51.808",
+ "dateFinished": "2019-10-11 10:07:52.076",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v\nfrom bank \nwhere age
\u003c 30 \ngroup by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:07:53.645",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ }
+ },
+ "commonSetting": {}
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data":
"age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n"
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491511585_-677666348",
+ "id": "paragraph_1569491511585_-677666348",
+ "dateCreated": "2019-09-26 17:51:51.585",
+ "dateStarted": "2019-10-11 10:07:53.654",
+ "dateFinished": "2019-10-11 10:08:10.439",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere
age \u003c ${maxAge\u003d30} \ngroup by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:08:13.910",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "pieChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "age",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "values": [
+ {
+ "name": "v",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {
+ "maxAge": "40"
+ },
+ "forms": {
+ "maxAge": {
+ "type": "TextBox",
+ "name": "maxAge",
+ "defaultValue": "30",
+ "hidden": false
+ }
+ }
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data":
"age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n35\t180\n36\t188\n37\t161\n38\t159\n39\t130\n"
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641100_115468969",
+ "id": "paragraph_1546592465802_-1181051373",
+ "dateCreated": "2019-09-26 17:20:41.100",
+ "dateStarted": "2019-10-11 10:08:13.923",
+ "dateFinished": "2019-10-11 10:08:14.876",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere
marital\u003d\u0027${marital\u003dsingle,single|divorced|married}\u0027 \ngroup
by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:37:59.493",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ },
+ "lineChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ },
+ "stackedAreaChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "age",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "values": [
+ {
+ "name": "v",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {
+ "marital": "married"
+ },
+ "forms": {
+ "marital": {
+ "type": "Select",
+ "options": [
+ {
+ "value": "single"
+ },
+ {
+ "value": "divorced"
+ },
+ {
+ "value": "married"
+ }
+ ],
+ "name": "marital",
+ "defaultValue": "single",
+ "hidden": false
+ }
+ }
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data":
"age\tv\n23\t3\n24\t11\n25\t11\n26\t18\n27\t26\n28\t23\n29\t37\n30\t56\n31\t104\n32\t105\n33\t103\n34\t142\n35\t109\n36\t117\n37\t100\n38\t99\n39\t88\n40\t105\n41\t97\n42\t91\n43\t79\n44\t68\n45\t76\n46\t82\n47\t78\n48\t91\n49\t87\n50\t74\n51\t63\n52\t66\n53\t75\n54\t56\n55\t68\n56\t50\n57\t78\n58\t67\n59\t56\n60\t36\n61\t15\n62\t5\n63\t7\n64\t6\n65\t4\n66\t7\n67\t5\n68\t1\n69\t5\n70\t5\n71\t5\n72\t4\n73\t6\n74\t2\n75\t3\n76\t1\n77\t5\n78\t2\n79\t3\n80\t6\n81\t1\n83\t
[...]
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641100_-1622522691",
+ "id": "paragraph_1546592478596_-1766740165",
+ "dateCreated": "2019-09-26 17:20:41.100",
+ "dateStarted": "2019-10-11 10:08:16.423",
+ "dateFinished": "2019-10-11 10:08:17.295",
+ "status": "FINISHED"
+ },
+ {
+ "text": "\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-09-26 17:54:48.292",
+ "config": {
+ "colWidth": 12.0,
+ "fontSize": 9.0,
+ "enabled": true,
+ "results": {},
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "editorMode": "ace/mode/scala"
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641101_-1182720873",
+ "id": "paragraph_1553093710610_-1734599499",
+ "dateCreated": "2019-09-26 17:20:41.101",
+ "status": "READY"
+ }
+ ],
+ "name": "Flink Batch Tutorial",
+ "id": "2EN1E1ATY",
+ "defaultInterpreterGroup": "spark",
+ "version": "0.9.0-SNAPSHOT",
+ "permissions": {},
+ "noteParams": {},
+ "noteForms": {},
+ "angularObjects": {},
+ "config": {
+ "isZeppelinNotebookCronEnable": false
+ },
+ "info": {},
+ "path": "/Flink Batch Tutorial"
+}
\ No newline at end of file
diff --git a/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln
b/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln
new file mode 100644
index 0000000..b8f3658
--- /dev/null
+++ b/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln
@@ -0,0 +1,427 @@
+{
+ "paragraphs": [
+ {
+ "title": "Introduction",
+ "text": "%md\n\nThis is a tutorial note for Flink Streaming application.
You can run flink scala api via `%flink` and run flink stream sql via
`%flink.ssql`. We provide 2 examples in this tutorial:\n\n* Classical word
count example via Flink streaming\n* We simulate a web log stream source, and
then query and visualize this streaming data in Zeppelin.\n\n\nFor now, the
capability of supporting flink streaming job is very limited in Zeppelin. For
example, canceling job is not suppor [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:18:50.038",
+ "config": {
+ "tableHide": false,
+ "editorSetting": {
+ "language": "markdown",
+ "editOnDblClick": true,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/markdown",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "HTML",
+ "data": "\u003cdiv
class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial note for
Flink Streaming application. You can run flink scala api via
\u003ccode\u003e%flink\u003c/code\u003e and run flink stream sql via
\u003ccode\u003e%flink.ssql\u003c/code\u003e. We provide 2 examples in this
tutorial:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eClassical word count
example via Flink streaming\u003c/li\u003e\n \u003cli\u003eWe simulate a web
log stream source, and [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705460_-1023073277",
+ "id": "paragraph_1548052720723_1943177100",
+ "dateCreated": "2019-09-26 17:55:05.460",
+ "dateStarted": "2019-10-08 15:18:47.711",
+ "dateFinished": "2019-10-08 15:18:47.724",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Configure Flink Interpreter",
+ "text": "%flink.conf\n\nFLINK_HOME \u003cFLINK_INSTALLATION\u003e\n# Use
blink planner for flink streaming scenario\nzeppelin.flink.planner blink\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:38:09.734",
+ "config": {
+ "editorSetting": {
+ "language": "text",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/text",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": []
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_327101264",
+ "id": "paragraph_1546571299955_275296580",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:21:48.153",
+ "dateFinished": "2019-10-08 15:21:48.165",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Stream WordCount",
+ "text": "%flink\n\nval data \u003d senv.fromElements(\"hello world\",
\"hello flink\", \"hello hadoop\")\ndata.flatMap(line \u003d\u003e
line.split(\"\\\\s\"))\n .map(w \u003d\u003e (w, 1))\n .keyBy(0)\n .sum(1)\n
.print\n\nsenv.execute()\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:07.936",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "\u001b[1m\u001b[34mdata\u001b[0m:
\u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.DataStream[String]\u001b[0m
\u003d
org.apache.flink.streaming.api.scala.DataStream@5a099566\n\u001b[1m\u001b[34mres0\u001b[0m:
\u001b[1m\u001b[32morg.apache.flink.streaming.api.datastream.DataStreamSink[(String,
Int)]\u001b[0m \u003d
org.apache.flink.streaming.api.datastream.DataStreamSink@58805cef\n(hello,1)\n(world,1)\n(hello,2)\n(flink,1)\n(hello,3)\n(hadoop,1)\n\u001b[1m\u00
[...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_377981606",
+ "id": "paragraph_1546571324670_-435705916",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:07.941",
+ "dateFinished": "2019-10-08 15:19:23.511",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Register a Stream DataSource to simulate web log",
+ "text": "%flink\n\nimport
org.apache.flink.streaming.api.functions.source.SourceFunction\nimport
org.apache.flink.table.api.TableEnvironment\nimport
org.apache.flink.streaming.api.TimeCharacteristic\nimport
org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport
java.util.Collections\nimport
scala.collection.JavaConversions._\n\nsenv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)\nsenv.enableCheckpointing(1000)\n\nval
data \u003d senv.addSource(new SourceFunct [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:31.503",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "import
org.apache.flink.streaming.api.functions.source.SourceFunction\nimport
org.apache.flink.table.api.TableEnvironment\nimport
org.apache.flink.streaming.api.TimeCharacteristic\nimport
org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport
java.util.Collections\nimport
scala.collection.JavaConversions._\n\u001b[1m\u001b[34mres4\u001b[0m:
\u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.StreamExecutionEnvironment\u001b[0m
\u003d org.apache.flink. [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_-1986381705",
+ "id": "paragraph_1546571333074_1869171983",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:31.507",
+ "dateFinished": "2019-10-08 15:19:34.040",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Total Page View",
+ "text": "%flink.ssql(type\u003dsingle, parallelism\u003d1,
refreshInterval\u003d3000, template\u003d\u003ch1\u003e{1}\u003c/h1\u003e until
\u003ch2\u003e{0}\u003c/h2\u003e, enableSavePoint\u003dfalse,
runWithSavePoint\u003dfalse)\n\nselect max(rowtime), count(1) from log\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:41.998",
+ "config": {
+ "savepointPath": "file:/tmp/save_point/savepoint-13e681-f552013d6184",
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 141.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_616808409",
+ "id": "paragraph_1546571459644_596843735",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:42.003",
+ "dateFinished": "2019-09-26 17:57:24.042",
+ "status": "ABORT"
+ },
+ {
+ "title": "Page View by Page",
+ "text": "%flink.ssql(type\u003dretract, refreshInterval\u003d2000,
parallelism\u003d1, enableSavePoint\u003dfalse,
runWithSavePoint\u003dfalse)\n\nselect url, count(1) as pv from log group by
url",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:20:02.708",
+ "config": {
+ "savepointPath":
"file:/flink/save_point/savepoint-e4f781-996290516ef1",
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 198.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "url": "string",
+ "pv": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "xLabelStatus": "default",
+ "rotate": {
+ "degree": "-45"
+ }
+ }
+ },
+ "commonSetting": {},
+ "keys": [],
+ "groups": [],
+ "values": []
+ },
+ "helium": {}
+ },
+ "1": {
+ "graph": {
+ "mode": "table",
+ "height": 86.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_1478852202",
+ "id": "paragraph_1546571485092_-716357716",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "dateStarted": "2019-09-26 17:56:45.502",
+ "dateFinished": "2019-09-26 17:57:24.042",
+ "status": "ABORT"
+ },
+ {
+ "title": "Page View Per Page for each 5 Seconds",
+ "text": "%flink.ssql(type\u003dts, parallelism\u003d1,
refreshInterval\u003d2000, enableSavePoint\u003dfalse,
runWithSavePoint\u003dfalse, threshold\u003d60000)\n\nselect
TUMBLE_START(rowtime, INTERVAL \u00275\u0027 SECOND) as start_time, url,
count(1) as pv from log group by TUMBLE(rowtime, INTERVAL \u00275\u0027
SECOND), url",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:38:03.669",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "lineChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "start_time": "string",
+ "url": "string",
+ "pv": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash":
"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
filter for
columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
pagination for better
navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable
a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "lineChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "rotate"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "start_time",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [
+ {
+ "name": "url",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "pv",
+ "index": 2.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data": "start_time\turl\tpv\n2018-01-01
00:00:00.0\thome\t9\n2018-01-01 00:00:00.0\tsearch\t17\n2018-01-01
00:00:00.0\tproduct\t24\n2018-01-01 00:00:05.0\tsearch\t17\n2018-01-01
00:00:05.0\thome\t8\n2018-01-01 00:00:05.0\tproduct\t25\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_-861601454",
+ "id": "paragraph_1546571542468_1571709353",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "status": "READY"
+ },
+ {
+ "text": "%flink.ssql\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-09-26 17:55:05.462",
+ "config": {},
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_-1279435256",
+ "id": "paragraph_1546571603746_-749250139",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "status": "READY"
+ }
+ ],
+ "name": "Flink Stream Tutorial",
+ "id": "2ER62Y5VJ",
+ "defaultInterpreterGroup": "spark",
+ "version": "0.9.0-SNAPSHOT",
+ "permissions": {},
+ "noteParams": {},
+ "noteForms": {},
+ "angularObjects": {},
+ "config": {
+ "isZeppelinNotebookCronEnable": false
+ },
+ "info": {},
+ "path": "/Flink Stream Tutorial"
+}
\ No newline at end of file
