[4/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/dataso

[2/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/sql/core/src/test/java/test/org/apache/spark/sql/sources/v2/JavaSimpleDataSourceV2.java -- diff --git a/sql/core/src/test/java/test/org/apache/spark/sql/sources/v2/JavaSim

[7/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

[SPARK-24882][SQL] improve data source v2 API ## What changes were proposed in this pull request? Improve the data source v2 API according to the [design doc](https://docs.google.com/document/d/1DDXCTCrup4bKWByTalkXWgavcPdvur8a4eEu8x1BzPM/edit?usp=sharing) summary of the changes 1. rename `Read

[1/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

Repository: spark Updated Branches: refs/heads/master 55f36641f -> e75488718 http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --

[6/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala -- diff --git a/external/kafka-0-10-sql/src/test/scala/org/apache/spa

[5/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousInputPartitionReader.java -- diff --git a/sql/core/src/main/java/org/apache/spark/sql/sou

[3/7] spark git commit: [SPARK-24882][SQL] improve data source v2 API

http://git-wip-us.apache.org/repos/asf/spark/blob/e7548871/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriteSupportProvider.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/exe

svn commit: r28891 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_22_00_02-55f3664-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Aug 22 07:16:44 2018 New Revision: 28891 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_22_00_02-55f3664 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23698][PYTHON] Resolve undefined names in Python 3

Repository: spark Updated Branches: refs/heads/master e75488718 -> 71f38ac24 [SPARK-23698][PYTHON] Resolve undefined names in Python 3 ## What changes were proposed in this pull request? Fix issues arising from the fact that builtins __file__, __long__, __raw_input()__, __unicode__, __xrange

spark git commit: [SPARK-25105][PYSPARK][SQL] Include PandasUDFType in the import all of pyspark.sql.functions

Repository: spark Updated Branches: refs/heads/master 71f38ac24 -> 2381953ab [SPARK-25105][PYSPARK][SQL] Include PandasUDFType in the import all of pyspark.sql.functions ## What changes were proposed in this pull request? Include PandasUDFType in the import all of pyspark.sql.functions ## H

spark git commit: [SPARK-25181][CORE] Limit Thread Pool size in BlockManager Master and Slave endpoints

Repository: spark Updated Branches: refs/heads/master 2381953ab -> 68ec4d641 [SPARK-25181][CORE] Limit Thread Pool size in BlockManager Master and Slave endpoints ## What changes were proposed in this pull request? Limit Thread Pool size in BlockManager Master and Slave endpoints. Currently

svn commit: r28897 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_22_12_02-68ec4d6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Aug 22 19:16:30 2018 New Revision: 28897 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_22_12_02-68ec4d6 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-25184][SS] Fixed race condition in StreamExecution that caused flaky test in FlatMapGroupsWithState

Repository: spark Updated Branches: refs/heads/master 68ec4d641 -> 310632498 [SPARK-25184][SS] Fixed race condition in StreamExecution that caused flaky test in FlatMapGroupsWithState ## What changes were proposed in this pull request? The race condition that caused test failure is between 2

spark git commit: [SPARK-25163][SQL] Fix flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuiteCheck

Repository: spark Updated Branches: refs/heads/master 310632498 -> 49a1993b1 [SPARK-25163][SQL] Fix flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuiteCheck ## What changes were proposed in this pull request? `ExternalAppendOnlyMapSuiteCheck` test is flaky. We use a `SparkListener`

spark git commit: [SPARK-24785][SHELL] Making sure REPL prints Spark UI info and then Welcome message

Repository: spark Updated Branches: refs/heads/master 49a1993b1 -> 2bc7b7553 [SPARK-24785][SHELL] Making sure REPL prints Spark UI info and then Welcome message ## What changes were proposed in this pull request? After https://github.com/apache/spark/pull/21495 the welcome message is printed

svn commit: r28899 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_22_16_02-49a1993-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Aug 22 23:16:07 2018 New Revision: 28899 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_22_16_02-49a1993 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-25127] DataSourceV2: Remove SupportsPushDownCatalystFilters

Repository: spark Updated Branches: refs/heads/master 2bc7b7553 -> 0295ad40d [SPARK-25127] DataSourceV2: Remove SupportsPushDownCatalystFilters ## What changes were proposed in this pull request? They depend on internal Expression APIs. Let's see how far we can get without it. ## How was thi

spark git commit: [SPARK-25167][SPARKR][TEST][MINOR] Minor fixes for R sql tests

Repository: spark Updated Branches: refs/heads/master 0295ad40d -> 1747469a1 [SPARK-25167][SPARKR][TEST][MINOR] Minor fixes for R sql tests ## What changes were proposed in this pull request? A few SQL tests for R were failing in my development environment. In this PR, i am attempting to addr

svn commit: r28901 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_22_20_01-1747469-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Aug 23 03:15:47 2018 New Revision: 28901 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_22_20_01-1747469 docs [This commit notification would consist of 1477 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-25133][SQL][DOC] Avro data source guide

Repository: spark Updated Branches: refs/heads/master 1747469a1 -> 05974f943 [SPARK-25133][SQL][DOC] Avro data source guide ## What changes were proposed in this pull request? Create documentation for AVRO data source. The new page will be linked in https://spark.apache.org/docs/latest/sql-p

spark git commit: [SPARK-23932][SQL][FOLLOW-UP] Fix an example of zip_with function.

Repository: spark Updated Branches: refs/heads/master 05974f943 -> 49720906c [SPARK-23932][SQL][FOLLOW-UP] Fix an example of zip_with function. ## What changes were proposed in this pull request? This is a follow-up pr of #22031 which added `zip_with` function to fix an example. ## How was

spark git commit: [SPARK-23034][SQL] Show RDD/relation names in RDD/Hive table scan nodes

Repository: spark Updated Branches: refs/heads/master 49720906c -> 2a0a8f753 [SPARK-23034][SQL] Show RDD/relation names in RDD/Hive table scan nodes ## What changes were proposed in this pull request? This pr proposed to show RDD/relation names in RDD/Hive table scan nodes. This change made th

spark git commit: [SPARK-25164][SQL] Avoid rebuilding column and path list for each column in parquet reader

Repository: spark Updated Branches: refs/heads/master 2a0a8f753 -> 8cc591c91 [SPARK-25164][SQL] Avoid rebuilding column and path list for each column in parquet reader ## What changes were proposed in this pull request? VectorizedParquetRecordReader::initializeInternal rebuilds the column li