This is an automated email from the ASF dual-hosted git repository.
jongyoul pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 602554e434 [ZEPPELIN-5788] Remove beam interpreter (#4435)
602554e434 is described below
commit 602554e4340d3201fe3087b5a76d29af67f4375b
Author: Lee Byonghun <dlqu...@gmail.com>
AuthorDate: Sun Jul 31 20:23:14 2022 +0900
[ZEPPELIN-5788] Remove beam interpreter (#4435)
* [ZEPPELIN-5788] Remove beam interpreter
* [ZEPPELIN-5788] remove beam profile when publish to maven
---
.github/workflows/core.yml | 2 +-
.github/workflows/frontend.yml | 2 +-
beam/README.md | 25 --
beam/pom.xml | 257 ---------------------
.../org/apache/zeppelin/beam/BeamInterpreter.java | 33 ---
beam/src/main/resources/interpreter-setting.json | 13 --
.../apache/zeppelin/beam/BeamInterpreterTest.java | 98 --------
conf/interpreter-list | 1 -
dev/create_release.sh | 2 +-
dev/publish_release.sh | 4 +-
docs/_includes/themes/zeppelin/_navigation.html | 1 -
docs/index.md | 1 -
docs/interpreter/beam.md | 116 ----------
docs/usage/interpreter/installation.md | 5 -
docs/usage/other_features/zeppelin_context.md | 3 +-
pom.xml | 1 -
.../docker/zeppelin-interpreter/conda_packages.txt | 1 -
zeppelin-distribution/src/bin_license/LICENSE | 1 -
18 files changed, 6 insertions(+), 560 deletions(-)
diff --git a/.github/workflows/core.yml b/.github/workflows/core.yml
index cc653b37ae..919adf36ab 100644
--- a/.github/workflows/core.yml
+++ b/.github/workflows/core.yml
@@ -82,7 +82,7 @@ jobs:
interpreter-test-non-core:
runs-on: ubuntu-20.04
env:
- INTERPRETERS:
'beam,hbase,jdbc,file,flink-cmd,cassandra,elasticsearch,bigquery,alluxio,livy,groovy,sap,java,neo4j,submarine,sparql,mongodb,influxdb,ksql'
+ INTERPRETERS:
'hbase,jdbc,file,flink-cmd,cassandra,elasticsearch,bigquery,alluxio,livy,groovy,sap,java,neo4j,submarine,sparql,mongodb,influxdb,ksql'
steps:
- name: Checkout
uses: actions/checkout@v2
diff --git a/.github/workflows/frontend.yml b/.github/workflows/frontend.yml
index 98f47375bd..6faac91d4d 100644
--- a/.github/workflows/frontend.yml
+++ b/.github/workflows/frontend.yml
@@ -19,7 +19,7 @@ env:
SPARK_PRINT_LAUNCH_COMMAND: "true"
SPARK_LOCAL_IP: 127.0.0.1
ZEPPELIN_LOCAL_IP: 127.0.0.1
- INTERPRETERS:
'!beam,!hbase,!jdbc,!file,!flink,!cassandra,!elasticsearch,!bigquery,!alluxio,!livy,!groovy,!sap,!java,!neo4j,!submarine,!sparql,!mongodb'
+ INTERPRETERS:
'!hbase,!jdbc,!file,!flink,!cassandra,!elasticsearch,!bigquery,!alluxio,!livy,!groovy,!sap,!java,!neo4j,!submarine,!sparql,!mongodb'
jobs:
run-e2e-tests-in-zeppelin-web:
diff --git a/beam/README.md b/beam/README.md
deleted file mode 100644
index de39815eeb..0000000000
--- a/beam/README.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# Overview
-Beam interpreter for Apache Zeppelin
-
-# Architecture
-Current interpreter implementation supports the static repl. It compiles the
code in memory, execute it and redirect the output to zeppelin.
-
-## Building the Beam Interpreter
-You have to first build the Beam interpreter by enable the **beam** profile as
follows:
-
-```
-./mvnw clean package -Pbeam -DskipTests -Pscala-2.10
-```
-
-### Notice
-- Flink runner comes with binary compiled for scala 2.10. So, currently we
support only Scala 2.10
-
-### Technical overview
-
- * Upon starting an interpreter, an instance of `JavaCompiler` is created.
-
- * When the user runs commands with beam, the `JavaParser` go through the code
to get a class that contains the main method.
-
- * Then it replaces the class name with random class name to avoid overriding
while compilation. it creates new out & err stream to get the data in new
stream instead of the console, to redirect output to zeppelin.
-
- * If there is any error during compilation, it can catch and redirect to
zeppelin.
diff --git a/beam/pom.xml b/beam/pom.xml
deleted file mode 100644
index 58cfdcb30e..0000000000
--- a/beam/pom.xml
+++ /dev/null
@@ -1,257 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
-
- <parent>
- <artifactId>zeppelin-interpreter-parent</artifactId>
- <groupId>org.apache.zeppelin</groupId>
- <version>0.11.0-SNAPSHOT</version>
- <relativePath>../zeppelin-interpreter-parent/pom.xml</relativePath>
- </parent>
-
- <artifactId>zeppelin-beam</artifactId>
- <packaging>jar</packaging>
- <name>Zeppelin: Beam interpreter</name>
-
- <properties>
- <interpreter.name>beam</interpreter.name>
-
- <beam.hadoop.version>3.2.3</beam.hadoop.version>
- <beam.spark.version>1.6.2</beam.spark.version>
- <beam.beam.version>2.0.0</beam.beam.version>
- <beam.scala.binary.version>2.10</beam.scala.binary.version>
-
- <!-- library versions -->
- <netty.version>4.1.42.Final</netty.version>
- <servlet.api.version>3.1.0</servlet.api.version>
- <avro.version>1.8.1</avro.version> <!-- should match beam dependency -->
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>io.netty</groupId>
- <artifactId>netty-all</artifactId>
- <version>${netty.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.spark</groupId>
- <artifactId>spark-core_${beam.scala.binary.version}</artifactId>
- <version>${beam.spark.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- <exclusion>
- <artifactId>netty-all</artifactId>
- <groupId>io.netty</groupId>
- </exclusion>
- <exclusion>
- <artifactId>akka-actor_${beam.scala.binary.version}</artifactId>
- <groupId>org.spark-project.akka</groupId>
- </exclusion>
- <exclusion>
- <artifactId>akka-remote_${beam.scala.binary.version}</artifactId>
- <groupId>org.spark-project.akka</groupId>
- </exclusion>
- <exclusion>
- <artifactId>akka-slf4j_${beam.scala.binary.version}</artifactId>
- <groupId>org.spark-project.akka</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.spark</groupId>
- <artifactId>spark-streaming_${beam.scala.binary.version}</artifactId>
- <version>${beam.spark.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- <version>${beam.hadoop.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${beam.hadoop.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.zeppelin</groupId>
- <artifactId>zeppelin-java</artifactId>
- <version>${project.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${beam.hadoop.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${beam.hadoop.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-annotations</artifactId>
- <version>${beam.hadoop.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-common</artifactId>
- <version>${beam.hadoop.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-common</artifactId>
- <version>${beam.hadoop.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>com.thoughtworks.qdox</groupId>
- <artifactId>qdox</artifactId>
- <version>2.0-M3</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-parent</artifactId>
- <version>${beam.beam.version}</version>
- <type>pom</type>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-core-java</artifactId>
- <version>${beam.beam.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>google-http-client-jackson2</artifactId>
- <groupId>com.google.http-client</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-direct-java</artifactId>
- <version>${beam.beam.version}</version>
- </dependency>
-
- <dependency>
- <groupId>javax.servlet</groupId>
- <artifactId>javax.servlet-api</artifactId>
- <version>${servlet.api.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
- <version>${beam.beam.version}</version>
- <exclusions>
- <exclusion>
- <artifactId>google-http-client-jackson2</artifactId>
- <groupId>com.google.http-client</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-spark</artifactId>
- <version>${beam.beam.version}</version>
- <type>jar</type>
- </dependency>
-
- <dependency>
- <groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-flink_${beam.scala.binary.version}</artifactId>
- <version>${beam.beam.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.commons</groupId>
- <artifactId>commons-exec</artifactId>
- <version>${commons.exec.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.avro</groupId>
- <artifactId>avro</artifactId>
- <version>${avro.version}</version>
- </dependency>
-
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <artifactId>maven-enforcer-plugin</artifactId>
- </plugin>
- <plugin>
- <artifactId>maven-resources-plugin</artifactId>
- </plugin>
- <plugin>
- <artifactId>maven-shade-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- <configuration>
- <skip>false</skip>
- </configuration>
- </plugin>
- </plugins>
- </build>
-</project>
diff --git a/beam/src/main/java/org/apache/zeppelin/beam/BeamInterpreter.java
b/beam/src/main/java/org/apache/zeppelin/beam/BeamInterpreter.java
deleted file mode 100644
index 06ab33b772..0000000000
--- a/beam/src/main/java/org/apache/zeppelin/beam/BeamInterpreter.java
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zeppelin.beam;
-
-import org.apache.zeppelin.java.JavaInterpreter;
-
-import java.util.Properties;
-
-/**
- * Beam interpreter
- */
-public class BeamInterpreter extends JavaInterpreter {
-
- public BeamInterpreter(Properties property) {
- super(property);
- }
-
-}
diff --git a/beam/src/main/resources/interpreter-setting.json
b/beam/src/main/resources/interpreter-setting.json
deleted file mode 100644
index 189e08b5d8..0000000000
--- a/beam/src/main/resources/interpreter-setting.json
+++ /dev/null
@@ -1,13 +0,0 @@
-[
- {
- "group": "beam",
- "name": "beam",
- "className": "org.apache.zeppelin.beam.BeamInterpreter",
- "defaultInterpreter": true,
- "properties": {
- },
- "editor": {
- "editOnDblClick": false
- }
- }
-]
diff --git a/beam/src/test/org/apache/zeppelin/beam/BeamInterpreterTest.java
b/beam/src/test/org/apache/zeppelin/beam/BeamInterpreterTest.java
deleted file mode 100644
index d1c56ee06e..0000000000
--- a/beam/src/test/org/apache/zeppelin/beam/BeamInterpreterTest.java
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zeppelin.beam;
-
-import org.apache.zeppelin.interpreter.InterpreterContext;
-import org.apache.zeppelin.interpreter.InterpreterResult;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-import java.io.PrintWriter;
-import java.io.StringWriter;
-import java.util.Properties;
-
-import static org.junit.Assert.assertEquals;
-
-/**
- * BeamInterpreterTest
- */
-public class BeamInterpreterTest {
-
- private static BeamInterpreter beam;
- private static InterpreterContext context;
-
- @BeforeClass
- public static void setUp() {
- Properties p = new Properties();
- beam = new BeamInterpreter(p);
- beam.open();
- context = InterpreterContext.builder().build();
- }
-
- @AfterClass
- public static void tearDown() {
- beam.close();
- }
-
- @Test
- public void testStaticRepl() {
-
- StringWriter writer = new StringWriter();
- PrintWriter out = new PrintWriter(writer);
- out.println("public class HelloWorld {");
- out.println(" public static void main(String args[]) {");
- out.println(" System.out.println(\"This is in another java file\");");
- out.println(" }");
- out.println("}");
- out.close();
-
- InterpreterResult res = beam.interpret(writer.toString(), context);
-
- assertEquals(InterpreterResult.Code.SUCCESS, res.code());
- }
-
- @Test
- public void testStaticReplWithoutMain() {
-
- StringBuffer sourceCode = new StringBuffer();
- sourceCode.append("package org.mdkt;\n");
- sourceCode.append("public class HelloClass {\n");
- sourceCode.append(" public String hello() { return \"hello\"; }");
- sourceCode.append("}");
- InterpreterResult res = beam.interpret(sourceCode.toString(), context);
- assertEquals(InterpreterResult.Code.ERROR, res.code());
- }
-
- @Test
- public void testStaticReplWithSyntaxError() {
-
- StringWriter writer = new StringWriter();
- PrintWriter out = new PrintWriter(writer);
- out.println("public class HelloWorld {");
- out.println(" public static void main(String args[]) {");
- out.println(" System.out.prin(\"This is in another java file\");");
- out.println(" }");
- out.println("}");
- out.close();
- InterpreterResult res = beam.interpret(writer.toString(), context);
-
- assertEquals(InterpreterResult.Code.ERROR, res.code());
- }
-
-}
diff --git a/conf/interpreter-list b/conf/interpreter-list
index 95b01746df..68a207e8c6 100644
--- a/conf/interpreter-list
+++ b/conf/interpreter-list
@@ -19,7 +19,6 @@
alluxio org.apache.zeppelin:zeppelin-alluxio:0.10.0
Alluxio interpreter
angular org.apache.zeppelin:zeppelin-angular:0.10.0 HTML
and AngularJS view rendering
-beam org.apache.zeppelin:zeppelin-beam:0.10.0 Beam
interpreter
bigquery org.apache.zeppelin:zeppelin-bigquery:0.10.0
BigQuery interpreter
cassandra org.apache.zeppelin:zeppelin-cassandra:0.10.0
Cassandra interpreter
elasticsearch org.apache.zeppelin:zeppelin-elasticsearch:0.10.0
Elasticsearch interpreter
diff --git a/dev/create_release.sh b/dev/create_release.sh
index 4f3ce17f84..3c429e2a00 100755
--- a/dev/create_release.sh
+++ b/dev/create_release.sh
@@ -98,7 +98,7 @@ function make_binary_release() {
git_clone
make_source_package
-make_binary_release netinst "-Pweb-angular -Phadoop-2.6 -pl
!beam,!hbase,!jdbc,!file,!flink,!cassandra,!elasticsearch,!bigquery,!alluxio,!livy,!groovy,!sap,!java,!neo4j,!submarine,!sparql,!mongodb,!ksql
-am"
+make_binary_release netinst "-Pweb-angular -Phadoop-2.6 -pl
!hbase,!jdbc,!file,!flink,!cassandra,!elasticsearch,!bigquery,!alluxio,!livy,!groovy,!sap,!java,!neo4j,!submarine,!sparql,!mongodb,!ksql
-am"
make_binary_release all "-Pweb-angular -Phadoop-2.6"
# remove non release files and dirs
diff --git a/dev/publish_release.sh b/dev/publish_release.sh
index 4231b73ebd..a08dd7f68a 100755
--- a/dev/publish_release.sh
+++ b/dev/publish_release.sh
@@ -91,11 +91,11 @@ function publish_snapshot_to_maven() {
echo "<password>$ASF_PASSWORD</password>" >> $tmp_settings
echo "</server></servers></settings>" >> $tmp_settings
- ./mvnw --settings $tmp_settings -Dmaven.repo.local="${tmp_repo}" -Pbeam
-DskipTests \
+ ./mvnw --settings $tmp_settings -Dmaven.repo.local="${tmp_repo}" -DskipTests
\
$PUBLISH_PROFILES -Drat.skip=true deploy
"${BASEDIR}/change_scala_version.sh" 2.11
- ./mvnw -Pscala-2.11 --settings $tmp_settings
-Dmaven.repo.local="${tmp_repo}" -Pbeam -DskipTests \
+ ./mvnw -Pscala-2.11 --settings $tmp_settings
-Dmaven.repo.local="${tmp_repo}" -DskipTests \
$PUBLISH_PROFILES -Drat.skip=true clean deploy
rm $tmp_settings
diff --git a/docs/_includes/themes/zeppelin/_navigation.html
b/docs/_includes/themes/zeppelin/_navigation.html
index cbef812dc2..7cbb234881 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -139,7 +139,6 @@
<li><a href="{{BASE_PATH}}/interpreter/r.html">R</a></li>
<li role="separator" class="divider"></li>
<li><a
href="{{BASE_PATH}}/interpreter/alluxio.html">Alluxio</a></li>
- <li><a href="{{BASE_PATH}}/interpreter/beam.html">Beam</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/bigquery.html">BigQuery</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/cassandra.html">Cassandra</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/elasticsearch.html">Elasticsearch</a></li>
diff --git a/docs/index.md b/docs/index.md
index 45d8922ddf..59dd3afa09 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -135,7 +135,6 @@ limitations under the License.
#### Available Interpreters
* [Alluxio](./interpreter/alluxio.html)
- * [Beam](./interpreter/beam.html)
* [BigQuery](./interpreter/bigquery.html)
* [Cassandra](./interpreter/cassandra.html)
* [Elasticsearch](./interpreter/elasticsearch.html)
diff --git a/docs/interpreter/beam.md b/docs/interpreter/beam.md
deleted file mode 100644
index d992b8ee5b..0000000000
--- a/docs/interpreter/beam.md
+++ /dev/null
@@ -1,116 +0,0 @@
----
-layout: page
-title: Beam interpreter in Apache Zeppelin
-description: Apache Beam is an open source, unified programming model that you
can use to create a data processing pipeline.
-group: interpreter
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-{% include JB/setup %}
-
-# Beam interpreter for Apache Zeppelin
-
-<div id="toc"></div>
-
-## Overview
-[Apache Beam](http://beam.incubator.apache.org) is an open source unified
platform for data processing pipelines. A pipeline can be build using one of
the Beam SDKs.
-The execution of the pipeline is done by different Runners. Currently, Beam
supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner.
-
-## How to use
-Basically, you can write normal Beam java code where you can determine the
Runner. You should write the main method inside a class becuase the interpreter
invoke this main to execute the pipeline. Unlike Zeppelin normal pattern, each
paragraph is considered as a separate job, there isn't any relation to any
other paragraph.
-
-The following is a demonstration of a word count example with data represented
in array of strings
-But it can read data from files by replacing
`Create.of(SENTENCES).withCoder(StringUtf8Coder.of())` with
`TextIO.Read.from("path/to/filename.txt")`
-
-```java
-%beam
-
-// most used imports
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.transforms.Create;
-import java.io.Serializable;
-import java.util.Arrays;
-import java.util.List;
-import java.util.ArrayList;
-import org.apache.beam.runners.direct.*;
-import org.apache.beam.sdk.runners.*;
-import org.apache.beam.sdk.options.*;
-import org.apache.beam.runners.flink.*;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Count;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SimpleFunction;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.options.PipelineOptions;
-
-public class MinimalWordCount {
- static List<String> s = new ArrayList<>();
-
- static final String[] SENTENCES_ARRAY = new String[] {
- "Hadoop is the Elephant King!",
- "A yellow and elegant thing.",
- "He never forgets",
- "Useful data, or lets",
- "An extraneous element cling!",
- "A wonderful king is Hadoop.",
- "The elephant plays well with Sqoop.",
- "But what helps him to thrive",
- "Are Impala, and Hive,",
- "And HDFS in the group.",
- "Hadoop is an elegant fellow.",
- "An elephant gentle and mellow.",
- "He never gets mad,",
- "Or does anything bad,",
- "Because, at his core, he is yellow",
- };
- static final List<String> SENTENCES = Arrays.asList(SENTENCES_ARRAY);
- public static void main(String[] args) {
- PipelineOptions options =
PipelineOptionsFactory.create().as(PipelineOptions.class);
- options.setRunner(FlinkRunner.class);
- Pipeline p = Pipeline.create(options);
- p.apply(Create.of(SENTENCES).withCoder(StringUtf8Coder.of()))
- .apply("ExtractWords", ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) {
- for (String word : c.element().split("[^a-zA-Z']+")) {
- if (!word.isEmpty()) {
- c.output(word);
- }
- }
- }
- }))
- .apply(Count.<String> perElement())
- .apply("FormatResults", ParDo.of(new DoFn<KV<String, Long>, String>() {
- @ProcessElement
- public void processElement(DoFn<KV<String, Long>,
String>.ProcessContext arg0)
- throws Exception {
- s.add("\n" + arg0.element().getKey() + "\t" +
arg0.element().getValue());
- }
- }));
- p.run();
- System.out.println("%table word\tcount");
- for (int i = 0; i < s.size(); i++) {
- System.out.print(s.get(i));
- }
-
- }
-}
-
-```
-
diff --git a/docs/usage/interpreter/installation.md
b/docs/usage/interpreter/installation.md
index f3eb9e26b5..d508c16f67 100644
--- a/docs/usage/interpreter/installation.md
+++ b/docs/usage/interpreter/installation.md
@@ -117,11 +117,6 @@ You can also find the below community managed interpreter
list in `conf/interpre
<td>org.apache.zeppelin:zeppelin-angular:0.10.0</td>
<td>HTML and AngularJS view rendering</td>
</tr>
- <tr>
- <td>beam</td>
- <td>org.apache.zeppelin:zeppelin-beam:0.10.0</td>
- <td>Beam interpreter</td>
- </tr>
<tr>
<td>bigquery</td>
<td>org.apache.zeppelin:zeppelin-bigquery:0.10.0</td>
diff --git a/docs/usage/other_features/zeppelin_context.md
b/docs/usage/other_features/zeppelin_context.md
index 076dbe1f07..9cd9167fe2 100644
--- a/docs/usage/other_features/zeppelin_context.md
+++ b/docs/usage/other_features/zeppelin_context.md
@@ -38,8 +38,7 @@ environments is described below.
In many programming-language interpreters (e.g. Apache Spark, Python, R) the
zeppelin-context is available
as a predefined variable `z` that can be used by directly invoking its methods.
The methods available on the `z` object are described below.
-Other interpreters based on programming languages like Apache Beam, etc. also
provide the
-predefined variable `z`.
+Other interpreters based on programming languages. also provide the predefined
variable `z`.
### Exploring Spark DataFrames
In the Apache Spark interpreter, the zeppelin-context provides a `show`
method, which,
diff --git a/pom.xml b/pom.xml
index ba63eef088..6366a8557f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -84,7 +84,6 @@
<module>neo4j</module>
<module>sap</module>
<module>java</module>
- <module>beam</module>
<module>ksql</module>
<module>sparql</module>
<module>zeppelin-common</module>
diff --git a/scripts/docker/zeppelin-interpreter/conda_packages.txt
b/scripts/docker/zeppelin-interpreter/conda_packages.txt
index 95bdfa27d9..3be8519342 100644
--- a/scripts/docker/zeppelin-interpreter/conda_packages.txt
+++ b/scripts/docker/zeppelin-interpreter/conda_packages.txt
@@ -12,7 +12,6 @@ matplotlib
ipykernel
jupyter_client
bokeh
-apache-beam
# R packages
r-evaluate
diff --git a/zeppelin-distribution/src/bin_license/LICENSE
b/zeppelin-distribution/src/bin_license/LICENSE
index 379a4f42e9..b0c9efc133 100644
--- a/zeppelin-distribution/src/bin_license/LICENSE
+++ b/zeppelin-distribution/src/bin_license/LICENSE
@@ -35,7 +35,6 @@ The following components are provided under Apache License.
(Apache 2.0) Apache HBase (http://hbase.apache.org/)
(Apache 2.0) Apache Flink (http://flink.apache.org/)
(Apache 2.0) Apache Jena (https://jena.apache.org/)
- (Apache 2.0) Apache Beam (http://beam.apache.org/)
(Apache 2.0) Apache Thrift 0.12.0 (org.apache.thrift:libthrift:0.12.0 -
http://thrift.apache.org/)
(Apache 2.0) Apache Lucene (https://lucene.apache.org/)
(Apache 2.0) Apache Zookeeper (org.apache.zookeeper:zookeeper:jar:3.4.5 -
http://zookeeper.apache.org/)
