spark git commit: [SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set

2015-08-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 47e473550 -> 2ad75d99f [SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set The issue only happens if `spark.executor.cores` is not set and executor memory is set to a high value. For example, if we ha

spark git commit: [SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set

2015-08-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master c564b2744 -> ef062c159 [SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set The issue only happens if `spark.executor.cores` is not set and executor memory is set to a high value. For example, if we have a

spark git commit: [SPARK-9753] [SQL] TungstenAggregate should also accept InternalRow instead of just UnsafeRow

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master 998f4ff94 -> c564b2744 [SPARK-9753] [SQL] TungstenAggregate should also accept InternalRow instead of just UnsafeRow https://issues.apache.org/jira/browse/SPARK-9753 This PR makes TungstenAggregate to accept `InternalRow` instead of just

spark git commit: [SPARK-9753] [SQL] TungstenAggregate should also accept InternalRow instead of just UnsafeRow

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 5598b6238 -> 47e473550 [SPARK-9753] [SQL] TungstenAggregate should also accept InternalRow instead of just UnsafeRow https://issues.apache.org/jira/browse/SPARK-9753 This PR makes TungstenAggregate to accept `InternalRow` instead of j

spark git commit: [SPARK-9754][SQL] Remove TypeCheck in debug package.

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 c5d43d6c8 -> 5598b6238 [SPARK-9754][SQL] Remove TypeCheck in debug package. TypeCheck no longer applies in the new "Tungsten" world. Author: Reynold Xin Closes #8043 from rxin/SPARK-9754 and squashes the following commits: 4ec471e [

spark git commit: [SPARK-9754][SQL] Remove TypeCheck in debug package.

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master 85be65b39 -> 998f4ff94 [SPARK-9754][SQL] Remove TypeCheck in debug package. TypeCheck no longer applies in the new "Tungsten" world. Author: Reynold Xin Closes #8043 from rxin/SPARK-9754 and squashes the following commits: 4ec471e [Reyn

spark git commit: [SPARK-9719] [ML] Clean up Naive Bayes doc

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.5 2a179a94e -> c5d43d6c8 [SPARK-9719] [ML] Clean up Naive Bayes doc Small documentation cleanups, including: * Adds documentation for `pi` and `theta` * setParam to `setModelType` Author: Feynman Liang Closes #8047 from feynmanliang/

spark git commit: [SPARK-9719] [ML] Clean up Naive Bayes doc

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master cd540c1e5 -> 85be65b39 [SPARK-9719] [ML] Clean up Naive Bayes doc Small documentation cleanups, including: * Adds documentation for `pi` and `theta` * setParam to `setModelType` Author: Feynman Liang Closes #8047 from feynmanliang/SPAR

spark git commit: [SPARK-9756] [ML] Make constructors in ML decision trees private

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.5 ea4dfb90a -> 2a179a94e [SPARK-9756] [ML] Make constructors in ML decision trees private These should be made private until there is a public constructor for providing `rootNode: Node` to use these constructors. jkbradley Author: Feyn

spark git commit: [SPARK-9756] [ML] Make constructors in ML decision trees private

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 49702bd73 -> cd540c1e5 [SPARK-9756] [ML] Make constructors in ML decision trees private These should be made private until there is a public constructor for providing `rootNode: Node` to use these constructors. jkbradley Author: Feynman

spark git commit: [SPARK-8890] [SQL] Fallback on sorting when writing many dynamic partitions

2015-08-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 902334fd5 -> 49702bd73 [SPARK-8890] [SQL] Fallback on sorting when writing many dynamic partitions Previously, we would open a new file for each new dynamic written out using `HadoopFsRelation`. For formats like parquet this is very costl

spark git commit: [SPARK-8890] [SQL] Fallback on sorting when writing many dynamic partitions

2015-08-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 295266049 -> ea4dfb90a [SPARK-8890] [SQL] Fallback on sorting when writing many dynamic partitions Previously, we would open a new file for each new dynamic written out using `HadoopFsRelation`. For formats like parquet this is very c

spark git commit: [SPARK-9748] [MLLIB] Centriod typo in KMeansModel

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master e2fbbe731 -> 902334fd5 [SPARK-9748] [MLLIB] Centriod typo in KMeansModel A minor typo (centriod -> centroid). Readable variable names help every users. Author: Bertrand Dechoux Closes #8037 from BertrandDechoux/kmeans-typo and squashes t

spark git commit: [SPARK-8481] [MLLIB] GaussianMixtureModel predict accepting single vector

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.5 547120287 -> 295266049 [SPARK-8481] [MLLIB] GaussianMixtureModel predict accepting single vector Resubmit of [https://github.com/apache/spark/pull/6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with ma

spark git commit: [SPARK-8481] [MLLIB] GaussianMixtureModel predict accepting single vector

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 881548ab2 -> e2fbbe731 [SPARK-8481] [MLLIB] GaussianMixtureModel predict accepting single vector Resubmit of [https://github.com/apache/spark/pull/6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with master

spark git commit: [SPARK-9674] Re-enable ignored test in SQLQuerySuite

2015-08-07 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-1.5 d13b5c895 -> 547120287 [SPARK-9674] Re-enable ignored test in SQLQuerySuite The original code that this test tests is removed in https://github.com/apache/spark/commit/9270bd06fd0b16892e3f37213b5bc7813ea11fdd. It was ignored shortly b

spark git commit: [SPARK-9674] Re-enable ignored test in SQLQuerySuite

2015-08-07 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 05d04e10a -> 881548ab2 [SPARK-9674] Re-enable ignored test in SQLQuerySuite The original code that this test tests is removed in https://github.com/apache/spark/commit/9270bd06fd0b16892e3f37213b5bc7813ea11fdd. It was ignored shortly befor

spark git commit: Revert "[SPARK-8481] [MLLIB] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector"

2015-08-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.4 e5a994f21 -> 4b5bbc589 Revert "[SPARK-8481] [MLLIB] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector" This reverts commit 07f778978d80f0af57d3dafda4c566a813ad2d09. Project: http://git-wip-us

spark git commit: [SPARK-9733][SQL] Improve physical plan explain for data sources

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 1b0f784a1 -> d13b5c895 [SPARK-9733][SQL] Improve physical plan explain for data sources All data sources show up as "PhysicalRDD" in physical plan explain. It'd be better if we can show the name of the data source. Without this patch:

spark git commit: [SPARK-9733][SQL] Improve physical plan explain for data sources

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master aeddeafc0 -> 05d04e10a [SPARK-9733][SQL] Improve physical plan explain for data sources All data sources show up as "PhysicalRDD" in physical plan explain. It'd be better if we can show the name of the data source. Without this patch: ```

spark git commit: [SPARK-9667][SQL] followup: Use GenerateUnsafeProjection.canSupport to test Exchange supported data types.

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 70bf170b9 -> 1b0f784a1 [SPARK-9667][SQL] followup: Use GenerateUnsafeProjection.canSupport to test Exchange supported data types. This way we recursively test the data types. cc chenghao-intel Author: Reynold Xin Closes #8036 from

spark git commit: [SPARK-9667][SQL] followup: Use GenerateUnsafeProjection.canSupport to test Exchange supported data types.

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master 9897cc5e3 -> aeddeafc0 [SPARK-9667][SQL] followup: Use GenerateUnsafeProjection.canSupport to test Exchange supported data types. This way we recursively test the data types. cc chenghao-intel Author: Reynold Xin Closes #8036 from rxin

spark git commit: [SPARK-8382] [SQL] Improve Analysis Unit test framework

2015-08-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 76eaa7018 -> 2432c2e23 [SPARK-8382] [SQL] Improve Analysis Unit test framework Author: Wenchen Fan Closes #8025 from cloud-fan/analysis and squashes the following commits: 51461b1 [Wenchen Fan] move test file to test folder ec88ace [Wenc

spark git commit: [SPARK-9736] [SQL] JoinedRow.anyNull should delegate to the underlying rows.

2015-08-07 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 2432c2e23 -> 9897cc5e3 [SPARK-9736] [SQL] JoinedRow.anyNull should delegate to the underlying rows. JoinedRow.anyNull currently loops through every field to check for null, which is inefficient if the underlying rows are UnsafeRows. It sho

spark git commit: [SPARK-8382] [SQL] Improve Analysis Unit test framework

2015-08-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 6c2f30c10 -> ff0abca2b [SPARK-8382] [SQL] Improve Analysis Unit test framework Author: Wenchen Fan Closes #8025 from cloud-fan/analysis and squashes the following commits: 51461b1 [Wenchen Fan] move test file to test folder ec88ace [

spark git commit: [SPARK-9736] [SQL] JoinedRow.anyNull should delegate to the underlying rows.

2015-08-07 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-1.5 ff0abca2b -> 70bf170b9 [SPARK-9736] [SQL] JoinedRow.anyNull should delegate to the underlying rows. JoinedRow.anyNull currently loops through every field to check for null, which is inefficient if the underlying rows are UnsafeRows. It

spark git commit: [SPARK-9674][SPARK-9667] Remove SparkSqlSerializer2

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master ebfd91c54 -> 76eaa7018 [SPARK-9674][SPARK-9667] Remove SparkSqlSerializer2 It is now subsumed by various Tungsten operators. Author: Reynold Xin Closes #7981 from rxin/SPARK-9674 and squashes the following commits: 144f96e [Reynold Xin]

spark git commit: [SPARK-9674][SPARK-9667] Remove SparkSqlSerializer2

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 7a6f95050 -> 6c2f30c10 [SPARK-9674][SPARK-9667] Remove SparkSqlSerializer2 It is now subsumed by various Tungsten operators. Author: Reynold Xin Closes #7981 from rxin/SPARK-9674 and squashes the following commits: 144f96e [Reynold

spark git commit: [SPARK-9467][SQL]Add SQLMetric to specialize accumulators to avoid boxing

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 064ba906a -> 7a6f95050 [SPARK-9467][SQL]Add SQLMetric to specialize accumulators to avoid boxing This PR adds SQLMetric/SQLMetricParam/SQLMetricValue to specialize accumulators to avoid boxing. All SQL metrics should use these classes

spark git commit: [SPARK-9467][SQL]Add SQLMetric to specialize accumulators to avoid boxing

2015-08-07 Thread rxin
Repository: spark Updated Branches: refs/heads/master e57d6b561 -> ebfd91c54 [SPARK-9467][SQL]Add SQLMetric to specialize accumulators to avoid boxing This PR adds SQLMetric/SQLMetricParam/SQLMetricValue to specialize accumulators to avoid boxing. All SQL metrics should use these classes rath

spark git commit: [SPARK-9683] [SQL] copy UTF8String when convert unsafe array/map to safe

2015-08-07 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.5 8ece4ccda -> 064ba906a [SPARK-9683] [SQL] copy UTF8String when convert unsafe array/map to safe When we convert unsafe row to safe row, we will do copy if the column is struct or string type. However, the string inside unsafe array/map

spark git commit: [SPARK-9683] [SQL] copy UTF8String when convert unsafe array/map to safe

2015-08-07 Thread davies
Repository: spark Updated Branches: refs/heads/master 15bd6f338 -> e57d6b561 [SPARK-9683] [SQL] copy UTF8String when convert unsafe array/map to safe When we convert unsafe row to safe row, we will do copy if the column is struct or string type. However, the string inside unsafe array/map are