spark git commit: [SPARK-24840][SQL] do not use dummy filter to switch codegen on/of

2018-07-18 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 1a4fda886 -> d05a926e7 [SPARK-24840][SQL] do not use dummy filter to switch codegen on/of ## What changes were proposed in this pull request? It's a little tricky and fragile to use a dummy filter to switch codegen on/off. For now we shou

svn commit: r28211 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_18_20_02-1a4fda8-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Thu Jul 19 03:17:21 2018 New Revision: 28211 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_18_20_02-1a4fda8 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [INFRA] Close stale PR

2018-07-18 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master cd5d93c0e -> 1a4fda886 [INFRA] Close stale PR Closes #17422 Closes #17619 Closes #18034 Closes #18229 Closes #18268 Closes #17973 Closes #18125 Closes #18918 Closes #19274 Closes #19456 Closes #19510 Closes #19420 Closes #20090 Closes #2017

spark git commit: [SPARK-24854][SQL] Gathering all Avro options into the AvroOptions class

2018-07-18 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 753f11516 -> cd5d93c0e [SPARK-24854][SQL] Gathering all Avro options into the AvroOptions class ## What changes were proposed in this pull request? In the PR, I propose to put all `Avro` options in new class `AvroOptions` in the same way

spark git commit: [SPARK-21261][DOCS][SQL] SQL Regex document fix

2018-07-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master d404e54e6 -> 753f11516 [SPARK-21261][DOCS][SQL] SQL Regex document fix ## What changes were proposed in this pull request? Fix regexes in spark-sql command examples. This takes over https://github.com/apache/spark/pull/18477 ## How was th

svn commit: r28210 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_18_16_02-d404e54-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Wed Jul 18 23:16:08 2018 New Revision: 28210 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_18_16_02-d404e54 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24129][K8S] Add option to pass --build-arg's to docker-image-tool.sh

2018-07-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master cd203e0df -> d404e54e6 [SPARK-24129][K8S] Add option to pass --build-arg's to docker-image-tool.sh ## What changes were proposed in this pull request? Adding `-b arg` option to take `--build-arg` parameters to pass into the docker command

svn commit: r28207 - in /dev/spark/2.3.3-SNAPSHOT-2018_07_18_14_01-7be70e2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Wed Jul 18 21:15:46 2018 New Revision: 28207 Log: Apache Spark 2.3.3-SNAPSHOT-2018_07_18_14_01-7be70e2 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24163][SPARK-24164][SQL] Support column list as the pivot column in Pivot

2018-07-18 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 1272b2034 -> cd203e0df [SPARK-24163][SPARK-24164][SQL] Support column list as the pivot column in Pivot ## What changes were proposed in this pull request? 1. Extend the Parser to enable parsing a column list as the pivot column. 2. Extend

svn commit: r28206 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_18_12_01-c8bee93-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Wed Jul 18 19:16:42 2018 New Revision: 28206 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_18_12_01-c8bee93 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22151] PYTHONPATH not picked up from the spark.yarn.appMaste…

2018-07-18 Thread tgraves
Repository: spark Updated Branches: refs/heads/master c8bee932c -> 1272b2034 [SPARK-22151] PYTHONPATH not picked up from the spark.yarn.appMaste… …rEnv properly Running in yarn cluster mode and trying to set pythonpath via spark.yarn.appMasterEnv.PYTHONPATH doesn't work. the yarn Client

spark git commit: [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap

2018-07-18 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-2.2 17db57213 -> 144426cff [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap ## What changes were proposed in this pull request? When speculation is enabled, TaskSetManager#markPartitionCompleted should write successful task

spark git commit: [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap

2018-07-18 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-2.3 e31b4766b -> 7be70e29d [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap ## What changes were proposed in this pull request? When speculation is enabled, TaskSetManager#markPartitionCompleted should write successful task

spark git commit: [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap

2018-07-18 Thread tgraves
Repository: spark Updated Branches: refs/heads/master fc0c8c971 -> c8bee932c [SPARK-24677][CORE] Avoid NoSuchElementException from MedianHeap ## What changes were proposed in this pull request? When speculation is enabled, TaskSetManager#markPartitionCompleted should write successful task dura

spark git commit: [SPARK-24825][K8S][TEST] Kubernetes integration tests build the whole reactor

2018-07-18 Thread mcheah
Repository: spark Updated Branches: refs/heads/master ebe9e2848 -> fc0c8c971 [SPARK-24825][K8S][TEST] Kubernetes integration tests build the whole reactor ## What changes were proposed in this pull request? Make the integration test script build all modules. In order to not run all the non-K

svn commit: r28200 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_18_08_01-ebe9e28-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Wed Jul 18 15:17:11 2018 New Revision: 28200 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_18_08_01-ebe9e28 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24628][DOC] Typos of the example code in docs/mllib-data-types.md

2018-07-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 002300dd4 -> ebe9e2848 [SPARK-24628][DOC] Typos of the example code in docs/mllib-data-types.md ## What changes were proposed in this pull request? The example wants to create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0)), but the l

spark git commit: [SPARK-24804] There are duplicate words in the test title in the DatasetSuite

2018-07-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2694dd2bf -> 002300dd4 [SPARK-24804] There are duplicate words in the test title in the DatasetSuite ## What changes were proposed in this pull request? In DatasetSuite.scala, in the 1299 line, test("SPARK-19896: cannot have circular refere

spark git commit: [MINOR][CORE] Add test cases for RDD.cartesian

2018-07-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 34cb3b54e -> 2694dd2bf [MINOR][CORE] Add test cases for RDD.cartesian ## What changes were proposed in this pull request? While looking through the codebase, it appeared that the scala code for RDD.cartesian does not have any tests for co

spark git commit: [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-java and Scala 2.12 build.

2018-07-18 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 3b59d326c -> 34cb3b54e [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-java and Scala 2.12 build. ## What changes were proposed in this pull request? This pr fixes lint-java and Scala 2.12 build. lint-java: ``` [ERROR] src/test/reso

svn commit: r28193 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_18_00_01-3b59d32-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-18 Thread pwendell
Author: pwendell Date: Wed Jul 18 07:17:12 2018 New Revision: 28193 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_18_00_01-3b59d32 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---