svn commit: r28111 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_13_20_01-f1a99ad-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Sat Jul 14 03:15:52 2018 New Revision: 28111 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_13_20_01-f1a99ad docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23984][K8S][TEST] Added Integration Tests for PySpark on Kubernetes

2018-07-13 Thread mcheah
Repository: spark Updated Branches: refs/heads/master a75571b46 -> f1a99ad58 [SPARK-23984][K8S][TEST] Added Integration Tests for PySpark on Kubernetes ## What changes were proposed in this pull request? I added integration tests for PySpark ( + checking JVM options + RemoteFileTest) which w

svn commit: r28110 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_13_16_01-a75571b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Fri Jul 13 23:15:37 2018 New Revision: 28110 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_13_16_01-a75571b docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23831][SQL] Add org.apache.derby to IsolatedClientLoader

2018-07-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 3b6005b8a -> a75571b46 [SPARK-23831][SQL] Add org.apache.derby to IsolatedClientLoader ## What changes were proposed in this pull request? Add `org.apache.derby` to `IsolatedClientLoader`, otherwise it may throw an exception: ```scala ...

svn commit: r28099 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_13_12_01-3b6005b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Fri Jul 13 19:16:14 2018 New Revision: 28099 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_13_12_01-3b6005b docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23528][ML] Add numIter to ClusteringSummary

2018-07-13 Thread holden
Repository: spark Updated Branches: refs/heads/master 3bcb1b481 -> 3b6005b8a [SPARK-23528][ML] Add numIter to ClusteringSummary ## What changes were proposed in this pull request? Added the number of iterations in `ClusteringSummary`. This is an helpful information in evaluating how to event

svn commit: r28096 - in /dev/spark/2.3.3-SNAPSHOT-2018_07_13_10_01-9cf375f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Fri Jul 13 17:16:08 2018 New Revision: 28096 Log: Apache Spark 2.3.3-SNAPSHOT-2018_07_13_10_01-9cf375f docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: Revert "[SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods"

2018-07-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master c1b62e420 -> 3bcb1b481 Revert "[SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods" This reverts commit c1b62e420a43aa7da36733ccdbec057d87ac1b43. Project: http://git-wip-us.apache.org/repos/asf/spark/repo C

spark git commit: [SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods

2018-07-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master dfd7ac988 -> c1b62e420 [SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods ## What changes were proposed in this pull request? Improve Avro unit test: 1. use QueryTest/SharedSQLContext/SQLTestUtils, instead o

spark git commit: [SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work

2018-07-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.3 32429256f -> 9cf375f5b [SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work ## What changes were proposed in this pull request? When we use a reference from Dataset in filter or sort, which was not used in t

spark git commit: [SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work

2018-07-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 0f24c6f8a -> dfd7ac988 [SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work ## What changes were proposed in this pull request? When we use a reference from Dataset in filter or sort, which was not used in the p

svn commit: r28095 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_13_08_01-0f24c6f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Fri Jul 13 15:16:42 2018 New Revision: 28095 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_13_08_01-0f24c6f docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24713] AppMatser of spark streaming kafka OOM if there are hund…

2018-07-13 Thread koeninger
Repository: spark Updated Branches: refs/heads/master 0ce11d0e3 -> 0f24c6f8a [SPARK-24713] AppMatser of spark streaming kafka OOM if there are hund… We have hundreds of kafka topics need to be consumed in one application. The application master will throw OOM exception after hanging for nea

svn commit: r28085 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_13_00_02-0ce11d0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-13 Thread pwendell
Author: pwendell Date: Fri Jul 13 07:16:33 2018 New Revision: 28085 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_13_00_02-0ce11d0 docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---