Modified: zeppelin/site/docs/0.8.2/setup/basics/how_to_build.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/basics/how_to_build.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/basics/how_to_build.html (original) +++ zeppelin/site/docs/0.8.2/setup/basics/how_to_build.html Sun Sep 29 07:49:17 2019 @@ -286,23 +286,23 @@ limitations under the License. <p>If you haven't installed Git and Maven yet, check the <a href="#build-requirements">Build requirements</a> section and follow the step by step instructions from there.</p> <h4>1. Clone the Apache Zeppelin repository</h4> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://github.com/apache/zeppelin.git +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">git clone https://github.com/apache/zeppelin.git </code></pre></div> <h4>2. Build source</h4> <p>You can build Zeppelin with following maven command:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -DskipTests <span class="o">[</span>Options<span class="o">]</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">mvn clean package -DskipTests <span class="o">[</span>Options<span class="o">]</span> </code></pre></div> <p>If you're unsure about the options, use the same commands that creates official binary package.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="c1"># update all pom.xml to use scala 2.11</span> -./dev/change_scala_version.sh <span class="m">2</span>.11 -<span class="c1"># build zeppelin with all interpreters and include latest version of Apache spark support for local mode.</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="c"># update all pom.xml to use scala 2.11</span> +./dev/change_scala_version.sh 2.11 +<span class="c"># build zeppelin with all interpreters and include latest version of Apache spark support for local mode.</span> mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 </code></pre></div> <h4>3. Done</h4> <p>You can directly start Zeppelin by running after successful build:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/zeppelin-daemon.sh start +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">./bin/zeppelin-daemon.sh start </code></pre></div> <p>Check <a href="#build-profiles">build-profiles</a> section for further build options. If you are behind proxy, follow instructions in <a href="#proxy-setting-optional">Proxy setting</a> section.</p> @@ -320,7 +320,7 @@ If you are behind proxy, follow instruct <p>Set spark major version</p> <p>Available profiles are</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>-Pspark-2.1 +<div class="highlight"><pre><code class="text language-text" data-lang="text">-Pspark-2.1 -Pspark-2.0 -Pspark-1.6 -Pspark-1.5 @@ -338,7 +338,7 @@ If you are behind proxy, follow instruct <p>set hadoop major version</p> <p>Available profiles are</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>-Phadoop-0.23 +<div class="highlight"><pre><code class="text language-text" data-lang="text">-Phadoop-0.23 -Phadoop-1 -Phadoop-2.2 -Phadoop-2.3 @@ -352,7 +352,7 @@ If you are behind proxy, follow instruct <p>set scala version (default 2.10) Available profiles are</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>-Pscala-2.10 +<div class="highlight"><pre><code class="text language-text" data-lang="text">-Pscala-2.10 -Pscala-2.11 </code></pre></div> <h5><code>-Pr</code> (optional)</h5> @@ -369,7 +369,7 @@ Available profiles are</p> The correct Maven artifacts can be found for every version of MapR at http://doc.mapr.com</p> <p>Available profiles are</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>-Pmapr3 +<div class="highlight"><pre><code class="text language-text" data-lang="text">-Pmapr3 -Pmapr40 -Pmapr41 -Pmapr50 @@ -382,52 +382,52 @@ The correct Maven artifacts can be found <h3>Build command examples</h3> <p>Here are some examples with several options:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="c1"># build with spark-2.1, scala-2.11</span> -./dev/change_scala_version.sh <span class="m">2</span>.11 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="c"># build with spark-2.1, scala-2.11</span> +./dev/change_scala_version.sh 2.11 mvn clean package -Pspark-2.1 -Phadoop-2.4 -Pscala-2.11 -DskipTests -<span class="c1"># build with spark-2.0, scala-2.11</span> -./dev/change_scala_version.sh <span class="m">2</span>.11 +<span class="c"># build with spark-2.0, scala-2.11</span> +./dev/change_scala_version.sh 2.11 mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pscala-2.11 -DskipTests -<span class="c1"># build with spark-1.6, scala-2.10</span> +<span class="c"># build with spark-1.6, scala-2.10</span> mvn clean package -Pspark-1.6 -Phadoop-2.4 -DskipTests -<span class="c1"># spark-cassandra integration</span> -mvn clean package -Pcassandra-spark-1.5 -Dhadoop.version<span class="o">=</span><span class="m">2</span>.6.0 -Phadoop-2.6 -DskipTests -DskipTests +<span class="c"># spark-cassandra integration</span> +mvn clean package -Pcassandra-spark-1.5 -Dhadoop.version<span class="o">=</span>2.6.0 -Phadoop-2.6 -DskipTests -DskipTests -<span class="c1"># with CDH</span> -mvn clean package -Pspark-1.5 -Dhadoop.version<span class="o">=</span><span class="m">2</span>.6.0-cdh5.5.0 -Phadoop-2.6 -Pvendor-repo -DskipTests +<span class="c"># with CDH</span> +mvn clean package -Pspark-1.5 -Dhadoop.version<span class="o">=</span>2.6.0-cdh5.5.0 -Phadoop-2.6 -Pvendor-repo -DskipTests -<span class="c1"># with MapR</span> +<span class="c"># with MapR</span> mvn clean package -Pspark-1.5 -Pmapr50 -DskipTests </code></pre></div> <p>Ignite Interpreter</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -Dignite.version<span class="o">=</span><span class="m">1</span>.9.0 -DskipTests +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">mvn clean package -Dignite.version<span class="o">=</span>1.9.0 -DskipTests </code></pre></div> <p>Scalding Interpreter</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -Pscalding -DskipTests +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">mvn clean package -Pscalding -DskipTests </code></pre></div> <h3>Optional configurations</h3> <p>Here are additional configurations that could be optionally tuned using the trailing <code>-D</code> option for maven commands</p> <p>Spark package</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>spark.archive <span class="c1"># default spark-${spark.version}</span> -spark.src.download.url <span class="c1"># default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}.tgz</span> -spark.bin.download.url <span class="c1"># default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}-bin-without-hadoop.tgz</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">spark.archive <span class="c"># default spark-${spark.version}</span> +spark.src.download.url <span class="c"># default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}.tgz</span> +spark.bin.download.url <span class="c"># default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}-bin-without-hadoop.tgz</span> </code></pre></div> <p>Py4J package</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>python.py4j.version <span class="c1"># default 0.9.2</span> -pypi.repo.url <span class="c1"># default https://pypi.python.org/packages</span> -python.py4j.repo.folder <span class="c1"># default /64/5c/01e13b68e8caafece40d549f232c9b5677ad1016071a48d04cc3895acaa3</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">python.py4j.version <span class="c"># default 0.9.2</span> +pypi.repo.url <span class="c"># default https://pypi.python.org/packages</span> +python.py4j.repo.folder <span class="c"># default /64/5c/01e13b68e8caafece40d549f232c9b5677ad1016071a48d04cc3895acaa3</span> </code></pre></div> <p>final URL location for Py4J package will be produced as following:</p> <p><code>${pypi.repo.url}${python.py4j.repo.folder}py4j-${python.py4j.version}.zip</code></p> <p>Frontend Maven Plugin configurations</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>plugin.frontend.nodeDownloadRoot # default https://nodejs.org/dist/ +<div class="highlight"><pre><code class="text language-text" data-lang="text">plugin.frontend.nodeDownloadRoot # default https://nodejs.org/dist/ plugin.frontend.npmDownloadRoot # default http://registry.npmjs.org/npm/-/ plugin.frontend.yarnDownloadRoot # default https://github.com/yarnpkg/yarn/releases/download/ </code></pre></div> @@ -437,7 +437,7 @@ plugin.frontend.yarnDownloadRoot # defau <p>If you don't have requirements prepared, install it. (The installation method may vary according to your environment, example is for Ubuntu.)</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo apt-get update +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo apt-get update sudo apt-get install git sudo apt-get install openjdk-7-jdk sudo apt-get install npm @@ -446,7 +446,7 @@ sudo apt-get install r-base-dev sudo apt-get install r-cran-evaluate </code></pre></div> <h3>Install maven</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz sudo tar -zxf apache-maven-3.3.9-bin.tar.gz -C /usr/local/ sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/local/bin/mvn </code></pre></div> @@ -460,7 +460,7 @@ sudo ln -s /usr/local/apache-maven-3.3.9 <p>If you're behind the proxy, you'll need to configure maven and npm to pass through it.</p> <p>First of all, configure maven in your <code>~/.m2/settings.xml</code>.</p> -<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="nt"><settings></span> +<div class="highlight"><pre><code class="xml language-xml" data-lang="xml"><span class="nt"><settings></span> <span class="nt"><proxies></span> <span class="nt"><proxy></span> <span class="nt"><id></span>proxy-http<span class="nt"></id></span> @@ -486,18 +486,18 @@ sudo ln -s /usr/local/apache-maven-3.3.9 <span class="nt"></settings></span> </code></pre></div> <p>Then, next commands will configure npm.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>npm config <span class="nb">set</span> proxy http://localhost:3128 -npm config <span class="nb">set</span> https-proxy http://localhost:3128 -npm config <span class="nb">set</span> registry <span class="s2">"http://registry.npmjs.org/"</span> -npm config <span class="nb">set</span> strict-ssl <span class="nb">false</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">npm config <span class="nb">set </span>proxy http://localhost:3128 +npm config <span class="nb">set </span>https-proxy http://localhost:3128 +npm config <span class="nb">set </span>registry <span class="s2">"http://registry.npmjs.org/"</span> +npm config <span class="nb">set </span>strict-ssl <span class="nb">false</span> </code></pre></div> <p>Configure git as well</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git config --global http.proxy http://localhost:3128 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">git config --global http.proxy http://localhost:3128 git config --global https.proxy http://localhost:3128 git config --global url.<span class="s2">"http://"</span>.insteadOf git:// </code></pre></div> <p>To clean up, set <code>active false</code> in Maven <code>settings.xml</code> and run these commands.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>npm config rm proxy +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">npm config rm proxy npm config rm https-proxy git config --global --unset http.proxy git config --global --unset https.proxy @@ -510,10 +510,10 @@ git config --global --unset url.<span cl <h2>Package</h2> <p>To package the final distribution including the compressed archive, run:</p> -<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span></span>mvn clean package -Pbuild-distr +<div class="highlight"><pre><code class="sh language-sh" data-lang="sh">mvn clean package -Pbuild-distr </code></pre></div> <p>To build a distribution with specific profiles, run:</p> -<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span></span>mvn clean package -Pbuild-distr -Pspark-1.5 -Phadoop-2.4 +<div class="highlight"><pre><code class="sh language-sh" data-lang="sh">mvn clean package -Pbuild-distr -Pspark-1.5 -Phadoop-2.4 </code></pre></div> <p>The profiles <code>-Pspark-1.5 -Phadoop-2.4</code> can be adjusted if you wish to build to a specific spark versions. </p> @@ -522,10 +522,10 @@ git config --global --unset url.<span cl <h2>Run end-to-end tests</h2> <p>Zeppelin comes with a set of end-to-end acceptance tests driving headless selenium browser</p> -<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span></span><span class="c1"># assumes zeppelin-server running on localhost:8080 (use -Durl=.. to override)</span> +<div class="highlight"><pre><code class="sh language-sh" data-lang="sh"><span class="c"># assumes zeppelin-server running on localhost:8080 (use -Durl=.. to override)</span> mvn verify -<span class="c1"># or take care of starting/stoping zeppelin-server from packaged zeppelin-distribuion/target</span> +<span class="c"># or take care of starting/stoping zeppelin-server from packaged zeppelin-distribuion/target</span> mvn verify -P using-packaged-distr </code></pre></div> <p><a href="https://github.com/igrigorik/ga-beacon";><img src="https://ga-beacon.appspot.com/UA-45176241-4/apache/zeppelin/README.md?pixel"; alt="Analytics"></a></p>
Modified: zeppelin/site/docs/0.8.2/setup/deployment/cdh.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/cdh.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/cdh.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/cdh.html Sun Sep 29 07:49:17 2019 @@ -267,35 +267,35 @@ limitations under the License. </blockquote> <p>You can import the Docker image by pulling it from Cloudera Docker Hub.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker pull cloudera/quickstart:latest +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker pull cloudera/quickstart:latest </code></pre></div> <h3>2. Run docker</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run -it <span class="se">\</span> - -p <span class="m">80</span>:80 <span class="se">\</span> - -p <span class="m">4040</span>:4040 <span class="se">\</span> - -p <span class="m">8020</span>:8020 <span class="se">\</span> - -p <span class="m">8022</span>:8022 <span class="se">\</span> - -p <span class="m">8030</span>:8030 <span class="se">\</span> - -p <span class="m">8032</span>:8032 <span class="se">\</span> - -p <span class="m">8033</span>:8033 <span class="se">\</span> - -p <span class="m">8040</span>:8040 <span class="se">\</span> - -p <span class="m">8042</span>:8042 <span class="se">\</span> - -p <span class="m">8088</span>:8088 <span class="se">\</span> - -p <span class="m">8480</span>:8480 <span class="se">\</span> - -p <span class="m">8485</span>:8485 <span class="se">\</span> - -p <span class="m">8888</span>:8888 <span class="se">\</span> - -p <span class="m">9083</span>:9083 <span class="se">\</span> - -p <span class="m">10020</span>:10020 <span class="se">\</span> - -p <span class="m">10033</span>:10033 <span class="se">\</span> - -p <span class="m">18088</span>:18088 <span class="se">\</span> - -p <span class="m">19888</span>:19888 <span class="se">\</span> - -p <span class="m">25000</span>:25000 <span class="se">\</span> - -p <span class="m">25010</span>:25010 <span class="se">\</span> - -p <span class="m">25020</span>:25020 <span class="se">\</span> - -p <span class="m">50010</span>:50010 <span class="se">\</span> - -p <span class="m">50020</span>:50020 <span class="se">\</span> - -p <span class="m">50070</span>:50070 <span class="se">\</span> - -p <span class="m">50075</span>:50075 <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run -it <span class="se">\</span> + -p 80:80 <span class="se">\</span> + -p 4040:4040 <span class="se">\</span> + -p 8020:8020 <span class="se">\</span> + -p 8022:8022 <span class="se">\</span> + -p 8030:8030 <span class="se">\</span> + -p 8032:8032 <span class="se">\</span> + -p 8033:8033 <span class="se">\</span> + -p 8040:8040 <span class="se">\</span> + -p 8042:8042 <span class="se">\</span> + -p 8088:8088 <span class="se">\</span> + -p 8480:8480 <span class="se">\</span> + -p 8485:8485 <span class="se">\</span> + -p 8888:8888 <span class="se">\</span> + -p 9083:9083 <span class="se">\</span> + -p 10020:10020 <span class="se">\</span> + -p 10033:10033 <span class="se">\</span> + -p 18088:18088 <span class="se">\</span> + -p 19888:19888 <span class="se">\</span> + -p 25000:25000 <span class="se">\</span> + -p 25010:25010 <span class="se">\</span> + -p 25020:25020 <span class="se">\</span> + -p 50010:50010 <span class="se">\</span> + -p 50020:50020 <span class="se">\</span> + -p 50070:50070 <span class="se">\</span> + -p 50075:50075 <span class="se">\</span> -h quickstart.cloudera --privileged<span class="o">=</span><span class="nb">true</span> <span class="se">\</span> agitated_payne_backup /usr/bin/docker-quickstart<span class="p">;</span> </code></pre></div> @@ -306,9 +306,9 @@ limitations under the License. <h3>4. Configure Spark interpreter in Zeppelin</h3> <p>Set following configurations to <code>conf/zeppelin-env.sh</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">MASTER</span><span class="o">=</span>yarn-client -<span class="nb">export</span> <span class="nv">HADOOP_CONF_DIR</span><span class="o">=[</span>your_hadoop_conf_path<span class="o">]</span> -<span class="nb">export</span> <span class="nv">SPARK_HOME</span><span class="o">=[</span>your_spark_home_path<span class="o">]</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">MASTER</span><span class="o">=</span>yarn-client +<span class="nb">export </span><span class="nv">HADOOP_CONF_DIR</span><span class="o">=[</span>your_hadoop_conf_path<span class="o">]</span> +<span class="nb">export </span><span class="nv">SPARK_HOME</span><span class="o">=[</span>your_spark_home_path<span class="o">]</span> </code></pre></div> <p><code>HADOOP_CONF_DIR</code>(Hadoop configuration path) is defined in <code>/scripts/docker/spark-cluster-managers/cdh/hdfs_conf</code>.</p> Modified: zeppelin/site/docs/0.8.2/setup/deployment/docker.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/docker.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/docker.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/docker.html Sun Sep 29 07:49:17 2019 @@ -271,23 +271,23 @@ limitations under the License. <p>You need to <a href="https://docs.docker.com/engine/installation/";>install docker</a> on your machine.</p> <h3>Running docker image</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run -p <span class="m">8080</span>:8080 --rm --name zeppelin apache/zeppelin:<release-version> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run -p 8080:8080 --rm --name zeppelin apache/zeppelin:<release-version> </code></pre></div> <ul> <li>Zeppelin will run at <code>http://localhost:8080</code>.</li> </ul> <p>If you want to specify <code>logs</code> and <code>notebook</code> dir, </p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run -p <span class="m">8080</span>:8080 --rm <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run -p 8080:8080 --rm <span class="se">\</span> -v <span class="nv">$PWD</span>/logs:/logs <span class="se">\</span> -v <span class="nv">$PWD</span>/notebook:/notebook <span class="se">\</span> -e <span class="nv">ZEPPELIN_LOG_DIR</span><span class="o">=</span><span class="s1">'/logs'</span> <span class="se">\</span> -e <span class="nv">ZEPPELIN_NOTEBOOK_DIR</span><span class="o">=</span><span class="s1">'/notebook'</span> <span class="se">\</span> ---name zeppelin apache/zeppelin:<release-version> <span class="c1"># e.g '0.7.1'</span> +--name zeppelin apache/zeppelin:<release-version> <span class="c"># e.g '0.7.1'</span> </code></pre></div> <h3>Building dockerfile locally</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span> -<span class="nb">cd</span> scripts/docker/zeppelin/bin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span> +<span class="nb">cd </span>scripts/docker/zeppelin/bin docker build -t my-zeppelin:my-tag ./ </code></pre></div> Modified: zeppelin/site/docs/0.8.2/setup/deployment/flink_and_spark_cluster.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/flink_and_spark_cluster.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/flink_and_spark_cluster.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/flink_and_spark_cluster.html Sun Sep 29 07:49:17 2019 @@ -286,16 +286,16 @@ limitations under the License. <h5>git</h5> <p>From the command prompt:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo apt-get install git +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo apt-get install git </code></pre></div> <h5>openssh-server</h5> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo apt-get install openssh-server +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo apt-get install openssh-server </code></pre></div> <h5>OpenJDK 7</h5> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo apt-get install openjdk-7-jdk openjdk-7-jre-lib +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo apt-get install openjdk-7-jdk openjdk-7-jre-lib </code></pre></div> <p><em>A note for those using Ubuntu 16.04</em>: To install <code>openjdk-7</code> on Ubuntu 16.04, one must add a repository. <a href="http://askubuntu.com/questions/761127/ubuntu-16-04-and-openjdk-7";>Source</a></p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo add-apt-repository ppa:openjdk-r/ppa +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-7-jdk openjdk-7-jre-lib </code></pre></div> @@ -304,17 +304,17 @@ sudo apt-get install openjdk-7-jdk openj <p>Zeppelin requires maven version 3.x. The version available in the repositories at the time of writing is 2.x, so maven must be installed manually.</p> <p>Purge any existing versions of maven.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo apt-get purge maven maven2 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo apt-get purge maven maven2 </code></pre></div> <p>Download the maven 3.3.9 binary.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>wget <span class="s2">"http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz"</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">wget <span class="s2">"http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz"</span> </code></pre></div> <p>Unarchive the binary and move to the <code>/usr/local</code> directory.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>tar -zxvf apache-maven-3.3.9-bin.tar.gz +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">tar -zxvf apache-maven-3.3.9-bin.tar.gz sudo mv ./apache-maven-3.3.9 /usr/local </code></pre></div> <p>Create symbolic links in <code>/usr/bin</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvn +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvn </code></pre></div> <h3>Installing Zeppelin</h3> @@ -322,13 +322,13 @@ sudo mv ./apache-maven-3.3.9 /usr/local <p>From the command prompt: Clone Zeppelin.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://github.com/apache/zeppelin.git +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">git clone https://github.com/apache/zeppelin.git </code></pre></div> <p>Enter the Zeppelin root directory.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> zeppelin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd </span>zeppelin </code></pre></div> <p>Package Zeppelin.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package -DskipTests -Pspark-1.6 -Dflink.version<span class="o">=</span><span class="m">1</span>.1.3 -Pscala-2.10 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">mvn clean package -DskipTests -Pspark-1.6 -Dflink.version<span class="o">=</span>1.1.3 -Pscala-2.10 </code></pre></div> <p><code>-DskipTests</code> skips build tests- you're not developing (yet), so you don't need to do tests, the clone version <em>should</em> build.</p> @@ -353,7 +353,7 @@ As long as you didn't edit any code, </ul> <p>Start the Zeppelin daemon.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/zeppelin-daemon.sh start +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">bin/zeppelin-daemon.sh start </code></pre></div> <p>Use <code>ifconfig</code> to determine the host machine's IP address. If you are not familiar with how to do this, a fairly comprehensive post can be found <a href="http://www.cyberciti.biz/faq/how-to-find-out-the-ip-address-assigned-to-eth0-and-display-ip-only/";>here</a>.</p> @@ -364,7 +364,7 @@ As long as you didn't edit any code, <h5>Flink Test</h5> <p>Create a new notebook named "Flink Test" and copy and paste the following code.</p> -<div class="highlight"><pre><code class="language-scala" data-lang="scala"><span></span><span class="o">%</span><span class="n">flink</span> <span class="c1">// let Zeppelin know what interpreter to use.</span> +<div class="highlight"><pre><code class="scala language-scala" data-lang="scala"><span class="o">%</span><span class="n">flink</span> <span class="c1">// let Zeppelin know what interpreter to use.</span> <span class="k">val</span> <span class="n">text</span> <span class="k">=</span> <span class="n">benv</span><span class="o">.</span><span class="n">fromElements</span><span class="o">(</span><span class="s">"In the time of chimpanzees, I was a monkey"</span><span class="o">,</span> <span class="c1">// some lines of text to analyze</span> <span class="s">"Butane in my veins and I'm out to cut the junkie"</span><span class="o">,</span> @@ -401,7 +401,7 @@ As long as you didn't edit any code, <h5>Spark Test</h5> <p>Create a new notebook named "Spark Test" and copy and paste the following code.</p> -<div class="highlight"><pre><code class="language-scala" data-lang="scala"><span></span><span class="o">%</span><span class="n">spark</span> <span class="c1">// let Zeppelin know what interpreter to use.</span> +<div class="highlight"><pre><code class="scala language-scala" data-lang="scala"><span class="o">%</span><span class="n">spark</span> <span class="c1">// let Zeppelin know what interpreter to use.</span> <span class="k">val</span> <span class="n">text</span> <span class="k">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span><span class="s">"In the time of chimpanzees, I was a monkey"</span><span class="o">,</span> <span class="c1">// some lines of text to analyze</span> <span class="s">"Butane in my veins and I'm out to cut the junkie"</span><span class="o">,</span> @@ -436,7 +436,7 @@ As long as you didn't edit any code, <p>Run the code to make sure the built-in Zeppelin Flink interpreter is working properly.</p> <p>Finally, stop the Zeppelin daemon. From the command prompt run:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/zeppelin-daemon.sh stop +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">bin/zeppelin-daemon.sh stop </code></pre></div> <h3>Installing Clusters</h3> @@ -447,13 +447,13 @@ As long as you didn't edit any code, <p>Building from source is recommended where possible, for simplicity in this tutorial we will download Flink and Spark Binaries.</p> <p>To download the Flink Binary use <code>wget</code></p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>wget <span class="s2">"http://mirror.cogentco.com/pub/apache/flink/flink-1.1.3/flink-1.1.3-bin-hadoop24-scala_2.10.tgz"</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">wget <span class="s2">"http://mirror.cogentco.com/pub/apache/flink/flink-1.1.3/flink-1.1.3-bin-hadoop24-scala_2.10.tgz"</span> tar -xzvf flink-1.1.3-bin-hadoop24-scala_2.10.tgz </code></pre></div> <p>This will download Flink 1.1.3, compatible with Hadoop 2.4. You do not have to install Hadoop for this binary to work, but if you are using Hadoop, please change <code>24</code> to your appropriate version.</p> <p>Start the Flink Cluster.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>flink-1.1.3/bin/start-cluster.sh +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">flink-1.1.3/bin/start-cluster.sh </code></pre></div> <h6>Building From source</h6> @@ -462,14 +462,14 @@ tar -xzvf flink-1.1.3-bin-hadoop24-scala <p>See the <a href="https://github.com/apache/flink/blob/master/README.md";>Flink Installation guide</a> for more detailed instructions.</p> <p>Return to the directory where you have been downloading, this tutorial assumes that is <code>$HOME</code>. Clone Flink, check out release-1.1.3-rc2, and build.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$HOME</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$HOME</span> git clone https://github.com/apache/flink.git -<span class="nb">cd</span> flink +<span class="nb">cd </span>flink git checkout release-1.1.3-rc2 mvn clean install -DskipTests </code></pre></div> <p>Start the Flink Cluster in stand-alone mode</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>build-target/bin/start-cluster.sh +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">build-target/bin/start-cluster.sh </code></pre></div> <h6>Ensure the cluster is up</h6> @@ -480,11 +480,11 @@ mvn clean install -DskipTests <p>If no task managers are present, restart the Flink cluster with the following commands:</p> <p>(if binaries)</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>flink-1.1.3/bin/stop-cluster.sh +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">flink-1.1.3/bin/stop-cluster.sh flink-1.1.3/bin/start-cluster.sh </code></pre></div> <p>(if built from source)</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>build-target/bin/stop-cluster.sh +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">build-target/bin/stop-cluster.sh build-target/bin/start-cluster.sh </code></pre></div> <h5>Spark 1.6 Cluster</h5> @@ -496,7 +496,7 @@ build-target/bin/start-cluster.sh <p>Using binaries is also</p> <p>To download the Spark Binary use <code>wget</code></p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>wget <span class="s2">"http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz"</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">wget <span class="s2">"http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz"</span> tar -xzvf spark-1.6.3-bin-hadoop2.6.tgz mv spark-1.6.3-bin-hadoop2.6 spark </code></pre></div> @@ -511,21 +511,21 @@ mv spark-1.6.3-bin-hadoop2.6 spark <p>Return to the directory where you have been downloading, this tutorial assumes that is $HOME. Clone Spark, check out branch-1.6, and build. <strong>Note:</strong> Recall, we're only checking out 1.6 because it is the most recent Spark for which a Zeppelin profile exists at the time of writing. You are free to check out other version, just make sure you build Zeppelin against the correct version of Spark. However if you use Spark 2.0, the word count example will need to be changed as Spark 2.0 is not compatible with the following examples.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$HOME</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$HOME</span> </code></pre></div> <p>Clone, check out, and build Spark version 1.6.x.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://github.com/apache/spark.git -<span class="nb">cd</span> spark +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">git clone https://github.com/apache/spark.git +<span class="nb">cd </span>spark git checkout branch-1.6 mvn clean package -DskipTests </code></pre></div> <h6>Start the Spark cluster</h6> <p>Return to the <code>$HOME</code> directory.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$HOME</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$HOME</span> </code></pre></div> <p>Start the Spark cluster in stand alone mode, specifying the webui-port as some port other than 8080 (the webui-port of Zeppelin).</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>spark/sbin/start-master.sh --webui-port <span class="m">8082</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">spark/sbin/start-master.sh --webui-port 8082 </code></pre></div> <p><strong>Note:</strong> Why <code>--webui-port 8082</code>? There is a digression toward the end of this document that explains this.</p> @@ -536,10 +536,10 @@ mvn clean package -DskipTests <p>Toward the top of the page there will be a <em>URL</em>: spark://<code>yourhost</code>:7077. Note this URL, the Spark Master URI, it will be needed in subsequent steps.</p> <p>Start the slave using the URI from the Spark master WebUI:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>spark/sbin/start-slave.sh spark://yourhostname:7077 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">spark/sbin/start-slave.sh spark://yourhostname:7077 </code></pre></div> <p>Return to the root directory and start the Zeppelin daemon.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$HOME</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$HOME</span> zeppelin/bin/zeppelin-daemon.sh start </code></pre></div> Modified: zeppelin/site/docs/0.8.2/setup/deployment/spark_cluster_mode.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/spark_cluster_mode.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/spark_cluster_mode.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/spark_cluster_mode.html Sun Sep 29 07:49:17 2019 @@ -278,15 +278,15 @@ You can simply set up Spark standalone e <h3>1. Build Docker file</h3> <p>You can find docker script files under <code>scripts/docker/spark-cluster-managers</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_standalone +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_standalone docker build -t <span class="s2">"spark_standalone"</span> . </code></pre></div> <h3>2. Run docker</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run -it <span class="se">\</span> --p <span class="m">8080</span>:8080 <span class="se">\</span> --p <span class="m">7077</span>:7077 <span class="se">\</span> --p <span class="m">8888</span>:8888 <span class="se">\</span> --p <span class="m">8081</span>:8081 <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run -it <span class="se">\</span> +-p 8080:8080 <span class="se">\</span> +-p 7077:7077 <span class="se">\</span> +-p 8888:8888 <span class="se">\</span> +-p 8081:8081 <span class="se">\</span> -h sparkmaster <span class="se">\</span> --name spark_standalone <span class="se">\</span> spark_standalone bash<span class="p">;</span> @@ -306,7 +306,7 @@ spark_standalone bash<span class="p">;</ <p><img src="/docs/0.8.2/assets/themes/zeppelin/img/docs-img/spark_ui.png" /></p> <p>You can also simply verify that Spark is running well in Docker with below command.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>ps -ef <span class="p">|</span> grep spark +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">ps -ef <span class="p">|</span> grep spark </code></pre></div> <h2>Spark on YARN mode</h2> @@ -319,28 +319,28 @@ spark_standalone bash<span class="p">;</ <h3>1. Build Docker file</h3> <p>You can find docker script files under <code>scripts/docker/spark-cluster-managers</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_yarn_cluster +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_yarn_cluster docker build -t <span class="s2">"spark_yarn"</span> . </code></pre></div> <h3>2. Run docker</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run -it <span class="se">\</span> - -p <span class="m">5000</span>:5000 <span class="se">\</span> - -p <span class="m">9000</span>:9000 <span class="se">\</span> - -p <span class="m">9001</span>:9001 <span class="se">\</span> - -p <span class="m">8088</span>:8088 <span class="se">\</span> - -p <span class="m">8042</span>:8042 <span class="se">\</span> - -p <span class="m">8030</span>:8030 <span class="se">\</span> - -p <span class="m">8031</span>:8031 <span class="se">\</span> - -p <span class="m">8032</span>:8032 <span class="se">\</span> - -p <span class="m">8033</span>:8033 <span class="se">\</span> - -p <span class="m">8080</span>:8080 <span class="se">\</span> - -p <span class="m">7077</span>:7077 <span class="se">\</span> - -p <span class="m">8888</span>:8888 <span class="se">\</span> - -p <span class="m">8081</span>:8081 <span class="se">\</span> - -p <span class="m">50010</span>:50010 <span class="se">\</span> - -p <span class="m">50075</span>:50075 <span class="se">\</span> - -p <span class="m">50020</span>:50020 <span class="se">\</span> - -p <span class="m">50070</span>:50070 <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run -it <span class="se">\</span> + -p 5000:5000 <span class="se">\</span> + -p 9000:9000 <span class="se">\</span> + -p 9001:9001 <span class="se">\</span> + -p 8088:8088 <span class="se">\</span> + -p 8042:8042 <span class="se">\</span> + -p 8030:8030 <span class="se">\</span> + -p 8031:8031 <span class="se">\</span> + -p 8032:8032 <span class="se">\</span> + -p 8033:8033 <span class="se">\</span> + -p 8080:8080 <span class="se">\</span> + -p 7077:7077 <span class="se">\</span> + -p 8888:8888 <span class="se">\</span> + -p 8081:8081 <span class="se">\</span> + -p 50010:50010 <span class="se">\</span> + -p 50075:50075 <span class="se">\</span> + -p 50020:50020 <span class="se">\</span> + -p 50070:50070 <span class="se">\</span> --name spark_yarn <span class="se">\</span> -h sparkmaster <span class="se">\</span> spark_yarn bash<span class="p">;</span> @@ -350,16 +350,16 @@ docker build -t <span class="s2">"s <h3>3. Verify running Spark on YARN.</h3> <p>You can simply verify the processes of Spark and YARN are running well in Docker with below command.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>ps -ef +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">ps -ef </code></pre></div> <p>You can also check each application web UI for HDFS on <code>http://<hostname>:50070/</code>, YARN on <code>http://<hostname>:8088/cluster</code> and Spark on <code>http://<hostname>:8080/</code>.</p> <h3>4. Configure Spark interpreter in Zeppelin</h3> <p>Set following configurations to <code>conf/zeppelin-env.sh</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">MASTER</span><span class="o">=</span>yarn-client -<span class="nb">export</span> <span class="nv">HADOOP_CONF_DIR</span><span class="o">=[</span>your_hadoop_conf_path<span class="o">]</span> -<span class="nb">export</span> <span class="nv">SPARK_HOME</span><span class="o">=[</span>your_spark_home_path<span class="o">]</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">MASTER</span><span class="o">=</span>yarn-client +<span class="nb">export </span><span class="nv">HADOOP_CONF_DIR</span><span class="o">=[</span>your_hadoop_conf_path<span class="o">]</span> +<span class="nb">export </span><span class="nv">SPARK_HOME</span><span class="o">=[</span>your_spark_home_path<span class="o">]</span> </code></pre></div> <p><code>HADOOP_CONF_DIR</code>(Hadoop configuration path) is defined in <code>/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf</code>.</p> @@ -378,19 +378,19 @@ docker build -t <span class="s2">"s <p>You can simply set up <a href="http://spark.apache.org/docs/latest/running-on-mesos.html";>Spark on Mesos</a> docker environment with below steps.</p> <h3>1. Build Docker file</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_mesos +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> <span class="nv">$ZEPPELIN_HOME</span>/scripts/docker/spark-cluster-managers/spark_mesos docker build -t <span class="s2">"spark_mesos"</span> . </code></pre></div> <h3>2. Run docker</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>docker run --net<span class="o">=</span>host -it <span class="se">\</span> --p <span class="m">8080</span>:8080 <span class="se">\</span> --p <span class="m">7077</span>:7077 <span class="se">\</span> --p <span class="m">8888</span>:8888 <span class="se">\</span> --p <span class="m">8081</span>:8081 <span class="se">\</span> --p <span class="m">8082</span>:8082 <span class="se">\</span> --p <span class="m">5050</span>:5050 <span class="se">\</span> --p <span class="m">5051</span>:5051 <span class="se">\</span> --p <span class="m">4040</span>:4040 <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">docker run --net<span class="o">=</span>host -it <span class="se">\</span> +-p 8080:8080 <span class="se">\</span> +-p 7077:7077 <span class="se">\</span> +-p 8888:8888 <span class="se">\</span> +-p 8081:8081 <span class="se">\</span> +-p 8082:8082 <span class="se">\</span> +-p 5050:5050 <span class="se">\</span> +-p 5051:5051 <span class="se">\</span> +-p 4040:4040 <span class="se">\</span> -h sparkmaster <span class="se">\</span> --name spark_mesos <span class="se">\</span> spark_mesos bash<span class="p">;</span> @@ -400,14 +400,14 @@ spark_mesos bash<span class="p">;</span> <h3>3. Verify running Spark on Mesos.</h3> <p>You can simply verify the processes of Spark and Mesos are running well in Docker with below command.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>ps -ef +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">ps -ef </code></pre></div> <p>You can also check each application web UI for Mesos on <code>http://<hostname>:5050/cluster</code> and Spark on <code>http://<hostname>:8080/</code>.</p> <h3>4. Configure Spark interpreter in Zeppelin</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">MASTER</span><span class="o">=</span>mesos://127.0.1.1:5050 -<span class="nb">export</span> <span class="nv">MESOS_NATIVE_JAVA_LIBRARY</span><span class="o">=[</span>PATH OF libmesos.so<span class="o">]</span> -<span class="nb">export</span> <span class="nv">SPARK_HOME</span><span class="o">=[</span>PATH OF SPARK HOME<span class="o">]</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">MASTER</span><span class="o">=</span>mesos://127.0.1.1:5050 +<span class="nb">export </span><span class="nv">MESOS_NATIVE_JAVA_LIBRARY</span><span class="o">=[</span>PATH OF libmesos.so<span class="o">]</span> +<span class="nb">export </span><span class="nv">SPARK_HOME</span><span class="o">=[</span>PATH OF SPARK HOME<span class="o">]</span> </code></pre></div> <p>Don't forget to set Spark <code>master</code> as <code>mesos://127.0.1.1:5050</code> in Zeppelin <strong>Interpreters</strong> setting page like below.</p> @@ -424,7 +424,7 @@ spark_mesos bash<span class="p">;</span> <ul> <li>If you have problem with hostname, use <code>--add-host</code> option when executing <code>dockerrun</code></li> </ul> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>## use `--add-host=moby:127.0.0.1` option to resolve +<div class="highlight"><pre><code class="text language-text" data-lang="text">## use `--add-host=moby:127.0.0.1` option to resolve ## since docker container couldn't resolve `moby` : java.net.UnknownHostException: moby: moby: Name or service not known @@ -436,7 +436,7 @@ spark_mesos bash<span class="p">;</span> <ul> <li>If you have problem with mesos master, try <code>mesos://127.0.0.1</code> instead of <code>mesos://127.0.1.1</code></li> </ul> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>I0103 20:17:22.329269 340 sched.cpp:330] New master detected at master@127.0.1.1:5050 +<div class="highlight"><pre><code class="text language-text" data-lang="text">I0103 20:17:22.329269 340 sched.cpp:330] New master detected at master@127.0.1.1:5050 I0103 20:17:22.330749 340 sched.cpp:341] No credentials provided. Attempting to register without authentication W0103 20:17:22.333531 340 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050' W0103 20:17:24.040252 339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050' Modified: zeppelin/site/docs/0.8.2/setup/deployment/virtual_machine.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/virtual_machine.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/virtual_machine.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/virtual_machine.html Sun Sep 29 07:49:17 2019 @@ -279,7 +279,7 @@ For SparkR users, this script includes s <ol> <li>Download and Install Vagrant: <a href="http://www.vagrantup.com/downloads.html";>Vagrant Downloads</a></li> <li><p>Install Ansible: <a href="http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip";>Ansible Python pip install</a></p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>sudo easy_install pip +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">sudo easy_install pip sudo pip install ansible ansible --version </code></pre></div> @@ -291,12 +291,12 @@ ansible --version <p>Thats it ! You can now run <code>vagrant ssh</code> and this will place you into the guest machines terminal prompt.</p> <p>If you don't wish to build Zeppelin from scratch, run the z-manager installer script while running in the guest VM:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>curl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh <span class="p">|</span> bash +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">curl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh <span class="p">|</span> bash </code></pre></div> <h2>Building Zeppelin</h2> <p>You can now </p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone git://git.apache.org/zeppelin.git +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">git clone git://git.apache.org/zeppelin.git </code></pre></div> <p>into a directory on your host machine, or directly in your virtual machine.</p> @@ -337,7 +337,7 @@ ansible --version <h2>How to build & run Zeppelin</h2> <p>This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. The following build steps will also include Python and R support via PySpark and SparkR:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /zeppelin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> /zeppelin mvn clean package -Pspark-1.6 -Phadoop-2.4 -DskipTests ./bin/zeppelin-daemon.sh start </code></pre></div> @@ -350,7 +350,7 @@ mvn clean package -Pspark-1.6 -Phadoop-2 <p>If you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host.</p> <p>Comment out the <code>forward_port</code> line, and uncomment the <code>private_network</code> line in Vagrantfile. The subnet that works best for your local network will vary so adjust <code>192.168.*.*</code> accordingly.</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>#config.vm.network "forwarded_port", guest: 8080, host: 8080 +<div class="highlight"><pre><code class="text language-text" data-lang="text">#config.vm.network "forwarded_port", guest: 8080, host: 8080 config.vm.network "private_network", ip: "192.168.51.52" </code></pre></div> <p><code>vagrant halt</code> followed by <code>vagrant up</code> will restart the guest machine bound to the IP address of <code>192.168.51.52</code>. @@ -361,46 +361,46 @@ This approach usually is typically requi <h3>Python Extras</h3> <p>With Zeppelin running, <strong>Numpy</strong>, <strong>SciPy</strong>, <strong>Pandas</strong> and <strong>Matplotlib</strong> will be available. Create a pyspark notebook, and try the below code.</p> -<div class="highlight"><pre><code class="language-python" data-lang="python"><span></span><span class="o">%</span><span class="n">pyspark</span> +<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="o">%</span><span class="n">pyspark</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="nn">scipy</span> <span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">import</span> <span class="nn">matplotlib</span> -<span class="k">print</span> <span class="s2">"numpy "</span> <span class="o">+</span> <span class="n">numpy</span><span class="o">.</span><span class="n">__version__</span> -<span class="k">print</span> <span class="s2">"scipy "</span> <span class="o">+</span> <span class="n">scipy</span><span class="o">.</span><span class="n">__version__</span> -<span class="k">print</span> <span class="s2">"pandas "</span> <span class="o">+</span> <span class="n">pandas</span><span class="o">.</span><span class="n">__version__</span> -<span class="k">print</span> <span class="s2">"matplotlib "</span> <span class="o">+</span> <span class="n">matplotlib</span><span class="o">.</span><span class="n">__version__</span> +<span class="k">print</span> <span class="s">"numpy "</span> <span class="o">+</span> <span class="n">numpy</span><span class="o">.</span><span class="n">__version__</span> +<span class="k">print</span> <span class="s">"scipy "</span> <span class="o">+</span> <span class="n">scipy</span><span class="o">.</span><span class="n">__version__</span> +<span class="k">print</span> <span class="s">"pandas "</span> <span class="o">+</span> <span class="n">pandas</span><span class="o">.</span><span class="n">__version__</span> +<span class="k">print</span> <span class="s">"matplotlib "</span> <span class="o">+</span> <span class="n">matplotlib</span><span class="o">.</span><span class="n">__version__</span> </code></pre></div> <p>To Test plotting using Matplotlib into a rendered <code>%html</code> SVG image, try</p> -<div class="highlight"><pre><code class="language-python" data-lang="python"><span></span><span class="o">%</span><span class="n">pyspark</span> +<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="o">%</span><span class="n">pyspark</span> <span class="kn">import</span> <span class="nn">matplotlib</span> -<span class="n">matplotlib</span><span class="o">.</span><span class="n">use</span><span class="p">(</span><span class="s1">'Agg'</span><span class="p">)</span> <span class="c1"># turn off interactive charting so this works for server side SVG rendering</span> +<span class="n">matplotlib</span><span class="o">.</span><span class="n">use</span><span class="p">(</span><span class="s">'Agg'</span><span class="p">)</span> <span class="c"># turn off interactive charting so this works for server side SVG rendering</span> <span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="kn">import</span> <span class="nn">StringIO</span> -<span class="c1"># clear out any previous plots on this note</span> +<span class="c"># clear out any previous plots on this note</span> <span class="n">plt</span><span class="o">.</span><span class="n">clf</span><span class="p">()</span> <span class="k">def</span> <span class="nf">show</span><span class="p">(</span><span class="n">p</span><span class="p">):</span> <span class="n">img</span> <span class="o">=</span> <span class="n">StringIO</span><span class="o">.</span><span class="n">StringIO</span><span class="p">()</span> - <span class="n">p</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s1">'svg'</span><span class="p">)</span> + <span class="n">p</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s">'svg'</span><span class="p">)</span> <span class="n">img</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> - <span class="k">print</span> <span class="s2">"%html <div style='width:600px'>"</span> <span class="o">+</span> <span class="n">img</span><span class="o">.</span><span class="n">buf</span> <span class="o">+</span> <span class="s2">"</div>"</span> + <span class="k">print</span> <span class="s">"%html <div style='width:600px'>"</span> <span class="o">+</span> <span class="n">img</span><span class="o">.</span><span class="n">buf</span> <span class="o">+</span> <span class="s">"</div>"</span> -<span class="c1"># Example data</span> -<span class="n">people</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Tom'</span><span class="p">,</span> <span class="s1">'Dick'</span><span class="p">,</span> <span class="s1">'Harry'</span><span class="p">,</span> <span class="s1">'Slim'</span><span class="p">,</span> <span class="s1">'Jim'</span><span class="p">)</span> +<span class="c"># Example data</span> +<span class="n">people</span> <span class="o">=</span> <span class="p">(</span><span class="s">'Tom'</span><span class="p">,</span> <span class="s">'Dick'</span><span class="p">,</span> <span class="s">'Harry'</span><span class="p">,</span> <span class="s">'Slim'</span><span class="p">,</span> <span class="s">'Jim'</span><span class="p">)</span> <span class="n">y_pos</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">people</span><span class="p">))</span> <span class="n">performance</span> <span class="o">=</span> <span class="mi">3</span> <span class="o">+</span> <span class="mi">10</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">people</span><span class="p">))</span> <span class="n">error</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">people</span><span class="p">))</span> -<span class="n">plt</span><span class="o">.</span><span class="n">barh</span><span class="p">(</span><span class="n">y_pos</span><span class="p">,</span> <span class="n">performance</span><span class="p">,</span> <span class="n">xerr</span><span class="o">=</span><span class="n">error</span><span class="p">,</span> <span class="n">align</span><span class="o">=</span><span class="s1">'center'</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.4</span><span class="p">)</span> +<span class="n">plt</span><span class="o">.</span><span class="n">barh</span><span class="p">(</span><span class="n">y_pos</span><span class="p">,</span> <span class="n">performance</span><span class="p">,</span> <span class="n">xerr</span><span class="o">=</span><span class="n">error</span><span class="p">,</span> <span class="n">align</span><span class="o">=</span><span class="s">'center'</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.4</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">yticks</span><span class="p">(</span><span class="n">y_pos</span><span class="p">,</span> <span class="n">people</span><span class="p">)</span> -<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'Performance'</span><span class="p">)</span> -<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s1">'How fast do you want to go today?'</span><span class="p">)</span> +<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Performance'</span><span class="p">)</span> +<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s">'How fast do you want to go today?'</span><span class="p">)</span> <span class="n">show</span><span class="p">(</span><span class="n">plt</span><span class="p">)</span> </code></pre></div> Modified: zeppelin/site/docs/0.8.2/setup/deployment/yarn_install.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/deployment/yarn_install.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/deployment/yarn_install.html (original) +++ zeppelin/site/docs/0.8.2/setup/deployment/yarn_install.html Sun Sep 29 07:49:17 2019 @@ -267,12 +267,12 @@ limitations under the License. <p>This step is optional, however its nice to run Zeppelin under its own user. In case you do not like to use Zeppelin (hope not) the user could be deleted along with all the packages that were installed for Zeppelin, Zeppelin binary itself and associated directories.</p> <p>Create a zeppelin user and switch to zeppelin user or if zeppelin user is already created then login as zeppelin.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>useradd zeppelin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">useradd zeppelin su - zeppelin whoami </code></pre></div> <p>Assuming a zeppelin user is created then running whoami command must return</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>zeppelin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">zeppelin </code></pre></div> <p>Its assumed in the rest of the document that zeppelin user is indeed created and below installation instructions are performed as zeppelin user.</p> @@ -291,13 +291,13 @@ whoami <h4>Hadoop client</h4> <p>Zeppelin can work with multiple versions & distributions of Hadoop. A complete list is available <a href="https://github.com/apache/zeppelin#build";>here</a>. This document assumes Hadoop 2.7.x client libraries including configuration files are installed on Zeppelin node. It also assumes /etc/hadoop/conf contains various Hadoop configuration files. The location of Hadoop configuration files may vary, hence use appropriate location.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>hadoop version -Hadoop <span class="m">2</span>.7.1.2.3.1.0-2574 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">hadoop version +Hadoop 2.7.1.2.3.1.0-2574 Subversion g...@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0 -Compiled by jenkins on <span class="m">2015</span>-07-25T22:36Z -Compiled with protoc <span class="m">2</span>.5.0 -From <span class="nb">source</span> with checksum 54f9bbb4492f92975e84e390599b881d -This <span class="nb">command</span> was run using /usr/hdp/2.3.1.0-2574/hadoop/lib/hadoop-common-2.7.1.2.3.1.0-2574.jar +Compiled by jenkins on 2015-07-25T22:36Z +Compiled with protoc 2.5.0 +From <span class="nb">source </span>with checksum 54f9bbb4492f92975e84e390599b881d +This <span class="nb">command </span>was run using /usr/hdp/2.3.1.0-2574/hadoop/lib/hadoop-common-2.7.1.2.3.1.0-2574.jar </code></pre></div> <h4>Spark</h4> @@ -320,28 +320,28 @@ This document assumes that Zeppelin is l <h2>Zeppelin Configuration</h2> <p>Zeppelin configuration needs to be modified to connect to YARN cluster. Create a copy of zeppelin environment shell script.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>cp /home/zeppelin/zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/zeppelin/conf/zeppelin-env.sh +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">cp /home/zeppelin/zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/zeppelin/conf/zeppelin-env.sh </code></pre></div> <p>Set the following properties</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/usr/java/jdk1.7.0_79"</span> -<span class="nb">export</span> <span class="nv">HADOOP_CONF_DIR</span><span class="o">=</span><span class="s2">"/etc/hadoop/conf"</span> -<span class="nb">export</span> <span class="nv">ZEPPELIN_JAVA_OPTS</span><span class="o">=</span><span class="s2">"-Dhdp.version=2.3.1.0-2574"</span> -<span class="nb">export</span> <span class="nv">SPARK_HOME</span><span class="o">=</span><span class="s2">"/usr/lib/spark"</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/usr/java/jdk1.7.0_79"</span> +<span class="nb">export </span><span class="nv">HADOOP_CONF_DIR</span><span class="o">=</span><span class="s2">"/etc/hadoop/conf"</span> +<span class="nb">export </span><span class="nv">ZEPPELIN_JAVA_OPTS</span><span class="o">=</span><span class="s2">"-Dhdp.version=2.3.1.0-2574"</span> +<span class="nb">export </span><span class="nv">SPARK_HOME</span><span class="o">=</span><span class="s2">"/usr/lib/spark"</span> </code></pre></div> <p>As /etc/hadoop/conf contains various configurations of YARN cluster, Zeppelin can now submit Spark/Hive jobs on YARN cluster form its web interface. The value of hdp.version is set to 2.3.1.0-2574. This can be obtained by running the following command</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>hdp-select status hadoop-client <span class="p">|</span> sed <span class="s1">'s/hadoop-client - \(.*\)/\1/'</span> -<span class="c1"># It returned 2.3.1.0-2574</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">hdp-select status hadoop-client <span class="p">|</span> sed <span class="s1">'s/hadoop-client - \(.*\)/\1/'</span> +<span class="c"># It returned 2.3.1.0-2574</span> </code></pre></div> <h2>Start/Stop</h2> <h3>Start Zeppelin</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> /home/zeppelin/zeppelin +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nb">cd</span> /home/zeppelin/zeppelin bin/zeppelin-daemon.sh start </code></pre></div> <p>After successful start, visit http://[zeppelin-server-host-name]:8080 with your web browser.</p> <h3>Stop Zeppelin</h3> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/zeppelin-daemon.sh stop +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">bin/zeppelin-daemon.sh stop </code></pre></div> <h2>Interpreter</h2> @@ -388,15 +388,15 @@ Click on Save button. Once these configu <h2>Debug</h2> <p>Zeppelin does not emit any kind of error messages on web interface when notebook/paragraph is run. If a paragraph fails it only displays ERROR. The reason for failure needs to be looked into log files which is present in logs directory under zeppelin installation base directory. Zeppelin creates a log file for each kind of interpreter.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span>$ <span class="nb">pwd</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span><span class="nv">$ </span><span class="nb">pwd</span> /home/zeppelin/zeppelin/logs -<span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span>$ ls -l -total <span class="m">844</span> --rw-rw-r-- <span class="m">1</span> zeppelin zeppelin <span class="m">14648</span> Aug <span class="m">3</span> <span class="m">14</span>:45 zeppelin-interpreter-hive-zeppelin-zeppelin-3529.log --rw-rw-r-- <span class="m">1</span> zeppelin zeppelin <span class="m">625050</span> Aug <span class="m">3</span> <span class="m">16</span>:05 zeppelin-interpreter-spark-zeppelin-zeppelin-3529.log --rw-rw-r-- <span class="m">1</span> zeppelin zeppelin <span class="m">200394</span> Aug <span class="m">3</span> <span class="m">21</span>:15 zeppelin-zeppelin-zeppelin-3529.log --rw-rw-r-- <span class="m">1</span> zeppelin zeppelin <span class="m">16162</span> Aug <span class="m">3</span> <span class="m">14</span>:03 zeppelin-zeppelin-zeppelin-3529.out -<span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span>$ +<span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span><span class="nv">$ </span>ls -l +total 844 +-rw-rw-r-- 1 zeppelin zeppelin 14648 Aug 3 14:45 zeppelin-interpreter-hive-zeppelin-zeppelin-3529.log +-rw-rw-r-- 1 zeppelin zeppelin 625050 Aug 3 16:05 zeppelin-interpreter-spark-zeppelin-zeppelin-3529.log +-rw-rw-r-- 1 zeppelin zeppelin 200394 Aug 3 21:15 zeppelin-zeppelin-zeppelin-3529.log +-rw-rw-r-- 1 zeppelin zeppelin 16162 Aug 3 14:03 zeppelin-zeppelin-zeppelin-3529.out +<span class="o">[</span>zeppelin@zeppelin-3529 logs<span class="o">]</span><span class="err">$</span> </code></pre></div> </div> </div> Modified: zeppelin/site/docs/0.8.2/setup/operation/configuration.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/operation/configuration.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/operation/configuration.html (original) +++ zeppelin/site/docs/0.8.2/setup/operation/configuration.html Sun Sep 29 07:49:17 2019 @@ -627,7 +627,7 @@ limitations under the License. <p>A condensed example can be found in the top answer to this <a href="http://stackoverflow.com/questions/4008837/configure-ssl-on-jetty";>StackOverflow post</a>.</p> <p>The keystore holds the private key and certificate on the server end. The trustore holds the trusted client certificates. Be sure that the path and password for these two stores are correctly configured in the password fields below. They can be obfuscated using the Jetty password tool. After Maven pulls in all the dependency to build Zeppelin, one of the Jetty jars contain the Password tool. Invoke this command from the Zeppelin home build directory with the appropriate version, user, and password.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>java -cp ./zeppelin-server/target/lib/jetty-all-server-<version>.jar <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">java -cp ./zeppelin-server/target/lib/jetty-all-server-<version>.jar <span class="se">\</span> org.eclipse.jetty.util.security.Password <user> <password> </code></pre></div> <p>If you are using a self-signed, a certificate signed by an untrusted CA, or if client authentication is enabled, then the client must have a browser create exceptions for both the normal HTTPS port and WebSocket port. This can by done by trying to establish an HTTPS connection to both ports in a browser (e.g. if the ports are 443 and 8443, then visit https://127.0.0.1:443 and https://127.0.0.1:8443). This step can be skipped if the server certificate is signed by a trusted CA and client auth is disabled.</p> @@ -635,7 +635,7 @@ org.eclipse.jetty.util.security.Password <h3>Configuring server side SSL</h3> <p>The following properties needs to be updated in the <code>zeppelin-site.xml</code> in order to enable server side SSL.</p> -<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="nt"><property></span> +<div class="highlight"><pre><code class="xml language-xml" data-lang="xml"><span class="nt"><property></span> <span class="nt"><name></span>zeppelin.server.ssl.port<span class="nt"></name></span> <span class="nt"><value></span>8443<span class="nt"></value></span> <span class="nt"><description></span>Server ssl port. (used when ssl property is set to true)<span class="nt"></description></span> @@ -674,7 +674,7 @@ org.eclipse.jetty.util.security.Password <h3>Enabling client side certificate authentication</h3> <p>The following properties needs to be updated in the <code>zeppelin-site.xml</code> in order to enable client side certificate authentication.</p> -<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="nt"><property></span> +<div class="highlight"><pre><code class="xml language-xml" data-lang="xml"><span class="nt"><property></span> <span class="nt"><name></span>zeppelin.server.ssl.port<span class="nt"></name></span> <span class="nt"><value></span>8443<span class="nt"></value></span> <span class="nt"><description></span>Server ssl port. (used when ssl property is set to true)<span class="nt"></description></span> @@ -711,7 +711,7 @@ org.eclipse.jetty.util.security.Password <p>Please notice that passwords will be stored in <em>plain text</em> by default. To encrypt the passwords, use the ZEPPELIN<em>CREDENTIALS</em>ENCRYPT_KEY config variable. This will encrypt passwords using the AES-128 algorithm.</p> <p>You can generate an appropriate encryption key any way you'd like - for instance, by using the openssl tool:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>openssl enc -aes-128-cbc -k secret -P -md sha1 +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">openssl enc -aes-128-cbc -k secret -P -md sha1 </code></pre></div> <p><em>Important</em>: storing your encryption key in a configuration file is <em>not advised</em>. Depending on your environment security needs, you may want to consider utilizing a credentials server, storing the ZEPPELIN<em>CREDENTIALS</em>ENCRYPT_KEY as an OS env variable, or any other approach that would not colocate the encryption key and the encrypted content (the credentials.json file).</p> @@ -722,17 +722,17 @@ org.eclipse.jetty.util.security.Password <p>The Password tool documentation can be found <a href="http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html";>here</a>.</p> <p>After using the tool:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>java -cp <span class="nv">$ZEPPELIN_HOME</span>/zeppelin-server/target/lib/jetty-util-9.2.15.v20160210.jar <span class="se">\</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash">java -cp <span class="nv">$ZEPPELIN_HOME</span>/zeppelin-server/target/lib/jetty-util-9.2.15.v20160210.jar <span class="se">\</span> org.eclipse.jetty.util.security.Password <span class="se">\</span> password -<span class="m">2016</span>-12-15 <span class="m">10</span>:46:47.931:INFO::main: Logging initialized @101ms +2016-12-15 10:46:47.931:INFO::main: Logging initialized @101ms password OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v MD5:5f4dcc3b5aa765d61d8327deb882cf99 </code></pre></div> <p>update your configuration with the obfuscated password :</p> -<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span></span><span class="nt"><property></span> +<div class="highlight"><pre><code class="xml language-xml" data-lang="xml"><span class="nt"><property></span> <span class="nt"><name></span>zeppelin.ssl.keystore.password<span class="nt"></name></span> <span class="nt"><value></span>OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v<span class="nt"></value></span> <span class="nt"><description></span>Keystore password. Can be obfuscated by the Jetty Password tool<span class="nt"></description></span> Modified: zeppelin/site/docs/0.8.2/setup/security/authentication_nginx.html URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.2/setup/security/authentication_nginx.html?rev=1867697&r1=1867696&r2=1867697&view=diff ============================================================================== --- zeppelin/site/docs/0.8.2/setup/security/authentication_nginx.html (original) +++ zeppelin/site/docs/0.8.2/setup/security/authentication_nginx.html Sun Sep 29 07:49:17 2019 @@ -277,7 +277,7 @@ Here are instructions how to accomplish <li><p>Install NGINX server on your server instance</p> <p>You can install NGINX server with same box where zeppelin installed or separate box where it is dedicated to serve as proxy server.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>$ apt-get install nginx +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nv">$ </span>apt-get install nginx </code></pre></div> <blockquote> <p><strong>NOTE :</strong> On pre 1.3.13 version of NGINX, Proxy for Websocket may not fully works. Please use latest version of NGINX. See: <a href="https://www.nginx.com/blog/websocket-nginx/";>NGINX documentation</a>.</p> @@ -285,11 +285,11 @@ Here are instructions how to accomplish <li><p>Setup init script in NGINX</p> <p>In most cases, NGINX configuration located under <code>/etc/nginx/sites-available</code>. Create your own configuration or add your existing configuration at <code>/etc/nginx/sites-available</code>.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>$ <span class="nb">cd</span> /etc/nginx/sites-available -$ touch my-zeppelin-auth-setting +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cd</span> /etc/nginx/sites-available +<span class="nv">$ </span>touch my-zeppelin-auth-setting </code></pre></div> <p>Now add this script into <code>my-zeppelin-auth-setting</code> file. You can comment out <code>optional</code> lines If you want serve Zeppelin under regular HTTP 80 Port.</p> -<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>upstream zeppelin { +<div class="highlight"><pre><code class="text language-text" data-lang="text">upstream zeppelin { server [YOUR-ZEPPELIN-SERVER-IP]:[YOUR-ZEPPELIN-SERVER-PORT]; # For security, It is highly recommended to make this address/port as non-public accessible } @@ -327,20 +327,20 @@ server { } </code></pre></div> <p>Then make a symbolic link to this file from <code>/etc/nginx/sites-enabled/</code> to enable configuration above when NGINX reloads.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>$ ln -s /etc/nginx/sites-enabled/my-zeppelin-auth-setting /etc/nginx/sites-available/my-zeppelin-auth-setting +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nv">$ </span>ln -s /etc/nginx/sites-enabled/my-zeppelin-auth-setting /etc/nginx/sites-available/my-zeppelin-auth-setting </code></pre></div></li> <li><p>Setup user credential into <code>.htpasswd</code> file and restart server</p> <p>Now you need to setup <code>.htpasswd</code> file to serve list of authenticated user credentials for NGINX server.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>$ <span class="nb">cd</span> /etc/nginx -$ htpasswd -c htpasswd <span class="o">[</span>YOUR-ID<span class="o">]</span> +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cd</span> /etc/nginx +<span class="nv">$ </span>htpasswd -c htpasswd <span class="o">[</span>YOUR-ID<span class="o">]</span> NEW passwd: <span class="o">[</span>YOUR-PASSWORD<span class="o">]</span> RE-type new passwd: <span class="o">[</span>YOUR-PASSWORD-AGAIN<span class="o">]</span> </code></pre></div> <p>Or you can use your own apache <code>.htpasswd</code> files in other location for setting up property: <code>auth_basic_user_file</code></p> <p>Restart NGINX server.</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>$ service nginx restart +<div class="highlight"><pre><code class="bash language-bash" data-lang="bash"><span class="nv">$ </span>service nginx restart </code></pre></div> <p>Then check HTTP Basic Authentication works in browser. If you can see regular basic auth popup and then able to login with credential you entered into <code>.htpasswd</code> you are good to go.</p></li> <li><p>More security consideration</p></li>
