spark git commit: [SPARK-24478][SQL] Move projection and filter push down to physical conversion

Repository: spark Updated Branches: refs/heads/master 270a9a3ca -> 22daeba59 [SPARK-24478][SQL] Move projection and filter push down to physical conversion ## What changes were proposed in this pull request? This removes the v2 optimizer rule for push-down and instead pushes filters and requ

svn commit: r27472 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_14_16_01-270a9a3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jun 14 23:15:36 2018 New Revision: 27472 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_14_16_01-270a9a3 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

[1/2] spark git commit: [SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle

Repository: spark Updated Branches: refs/heads/master 18cb0c079 -> 270a9a3ca http://git-wip-us.apache.org/repos/asf/spark/blob/270a9a3c/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala --

[2/2] spark git commit: [SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle

[SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle ## What changes were proposed in this pull request? Previously, the scheduler backend was maintaining state in many places, not only for reading state but also writing to it. For example, state had to b

spark git commit: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution where no main class is required.

Repository: spark Updated Branches: refs/heads/master b8f27ae3b -> 18cb0c079 [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution where no main class is required. ## What changes were proposed in this pull request? With [PR 20925](https://github.com/apache/spark/pull/20925) now it's not p

spark git commit: [SPARK-24543][SQL] Support any type as DDL string for from_json's schema

Repository: spark Updated Branches: refs/heads/master d3eed8fd6 -> b8f27ae3b [SPARK-24543][SQL] Support any type as DDL string for from_json's schema ## What changes were proposed in this pull request? In the PR, I propose to support any DataType represented as DDL string for the from_json f

spark git commit: [SPARK-24563][PYTHON] Catch TypeError when testing existence of HiveConf when creating pysp…

Repository: spark Updated Branches: refs/heads/master fdadc4be0 -> d3eed8fd6 [SPARK-24563][PYTHON] Catch TypeError when testing existence of HiveConf when creating pysp… …ark shell ## What changes were proposed in this pull request? This PR catches TypeError when testing existence of Hi

svn commit: r27464 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_14_12_01-fdadc4b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jun 14 19:16:10 2018 New Revision: 27464 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_14_12_01-fdadc4b docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r27460 - in /dev/spark/2.3.2-SNAPSHOT-2018_06_14_10_01-e6bf325-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jun 14 17:15:51 2018 New Revision: 27460 Log: Apache Spark 2.3.2-SNAPSHOT-2018_06_14_10_01-e6bf325 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys

Repository: spark Updated Branches: refs/heads/branch-2.3 a2f65eb79 -> e6bf325de [SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys `EnsureRequirement` in its `reorder` method currently assumes that the same key appears only once in the join condition. This of

spark git commit: [SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys

Repository: spark Updated Branches: refs/heads/master 534065efe -> fdadc4be0 [SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys ## What changes were proposed in this pull request? `EnsureRequirement` in its `reorder` method currently assumes that the same key

svn commit: r27446 - in /dev/spark/2.3.2-SNAPSHOT-2018_06_14_02_02-a2f65eb-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jun 14 09:17:36 2018 New Revision: 27446 Log: Apache Spark 2.3.2-SNAPSHOT-2018_06_14_02_02-a2f65eb docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r27443 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_14_00_01-534065e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jun 14 07:17:21 2018 New Revision: 27443 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_14_00_01-534065e docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---