spark git commit: [SPARK-24924][SQL][FOLLOW-UP] Add mapping for built-in Avro data source

2018-07-27 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master e8752095a -> c6a3db2fb [SPARK-24924][SQL][FOLLOW-UP] Add mapping for built-in Avro data source ## What changes were proposed in this pull request? Add one more test case for `com.databricks.spark.avro`. ## How was this patch tested? N/A A

spark git commit: [SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF

2018-07-27 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 6424b146c -> e8752095a [SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF ## What changes were proposed in this pull request? This PR add supports for using mixed Python UDF and Scalar Pandas UDF, in the follow

svn commit: r28402 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_27_20_01-6424b14-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Sat Jul 28 03:16:04 2018 New Revision: 28402 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_27_20_01-6424b14 docs [This commit notification would consist of 1470 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [MINOR] Update docs for functions.scala to make it clear not all the built-in functions are defined there

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 34ebcc6b5 -> 6424b146c [MINOR] Update docs for functions.scala to make it clear not all the built-in functions are defined there The title summarizes the change. Author: Reynold Xin Closes #21318 from rxin/functions. Project: http://g

svn commit: r28398 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_27_16_01-34ebcc6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Fri Jul 27 23:15:57 2018 New Revision: 28398 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_27_16_01-34ebcc6 docs [This commit notification would consist of 1470 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [MINOR] Improve documentation for HiveStringType's

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 10f1f1965 -> 34ebcc6b5 [MINOR] Improve documentation for HiveStringType's The diff should be self-explanatory. Author: Reynold Xin Closes #21897 from rxin/hivestringtypedoc. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Co

spark git commit: [SPARK-21274][SQL] Implement EXCEPT ALL clause.

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 5828f41a5 -> 10f1f1965 [SPARK-21274][SQL] Implement EXCEPT ALL clause. ## What changes were proposed in this pull request? Implements EXCEPT ALL clause through query rewrites using existing operators in Spark. In this PR, an internal UDTF

svn commit: r28396 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_27_12_02-5828f41-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Fri Jul 27 19:16:20 2018 New Revision: 28396 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_27_12_02-5828f41 docs [This commit notification would consist of 1470 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-13343] speculative tasks that didn't commit shouldn't be marked as success

2018-07-27 Thread tgraves
Repository: spark Updated Branches: refs/heads/master ee5a5a092 -> 5828f41a5 [SPARK-13343] speculative tasks that didn't commit shouldn't be marked as success **Description** Currently Speculative tasks that didn't commit can show up as success (depending on timing of commit). This is a bit

spark git commit: [SPARK-21960][STREAMING] Spark Streaming Dynamic Allocation should respect spark.executor.instances

2018-07-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0a0f68bae -> ee5a5a092 [SPARK-21960][STREAMING] Spark Streaming Dynamic Allocation should respect spark.executor.instances ## What changes were proposed in this pull request? Removes check that `spark.executor.instances` is set to 0 when

svn commit: r28395 - in /dev/spark/2.3.3-SNAPSHOT-2018_07_27_10_02-d5f340f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Fri Jul 27 17:16:12 2018 New Revision: 28395 Log: Apache Spark 2.3.3-SNAPSHOT-2018_07_27_10_02-d5f340f docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24881][SQL] New Avro option - compression

2018-07-27 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master c9bec1d37 -> 0a0f68bae [SPARK-24881][SQL] New Avro option - compression ## What changes were proposed in this pull request? In the PR, I added new option for Avro datasource - `compression`. The option allows to specify compression codec

spark git commit: [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided"

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/master ef6c8395c -> c9bec1d37 [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided" ## What changes were proposed in this pull request? Please see [SPARK-24927][1] for more details. [1]: https://issues.apache.org/jira/br

spark git commit: [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided"

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 f339e2fd7 -> 73764737d [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided" ## What changes were proposed in this pull request? Please see [SPARK-24927][1] for more details. [1]: https://issues.apache.org/jir

spark git commit: [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided"

2018-07-27 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.3 fa552c3c1 -> d5f340f27 [SPARK-24927][BUILD][BRANCH-2.3] The scope of snappy-java cannot be "provided" ## What changes were proposed in this pull request? Please see [SPARK-24927][1] for more details. [1]: https://issues.apache.org/jir

svn commit: r28393 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_27_08_01-ef6c839-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Fri Jul 27 15:16:12 2018 New Revision: 28393 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_27_08_01-ef6c839 docs [This commit notification would consist of 1470 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-27 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 21fcac164 -> ef6c8395c [SPARK-23928][SQL] Add shuffle collection function. ## What changes were proposed in this pull request? This PR adds a new collection function: shuffle. It generates a random permutation of the given array. This imp

svn commit: r28389 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_27_00_01-21fcac1-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-27 Thread pwendell
Author: pwendell Date: Fri Jul 27 07:16:39 2018 New Revision: 28389 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_27_00_01-21fcac1 docs [This commit notification would consist of 1470 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---