Repository: spark Updated Branches: refs/heads/master 500f84e49 -> e1b85f310
SPARK-2955 [BUILD] Test code fails to compile with "mvn compile" without "install" (This is the corrected follow-up to https://issues.apache.org/jira/browse/SPARK-2903) Right now, `mvn compile test-compile` fails to compile Spark. (Don't worry; `mvn package` works, so this is not major.) The issue stems from test code in some modules depending on test code in other modules. That is perfectly fine and supported by Maven. It takes extra work to get this to work with scalatest, and this has been attempted: https://github.com/apache/spark/blob/master/sql/catalyst/pom.xml#L86 This formulation is not quite enough, since the SQL Core module's tests fail to compile for lack of finding test classes in SQL Catalyst, and likewise for most Streaming integration modules depending on core Streaming test code. Example: ``` [error] /Users/srowen/Documents/spark/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala:23: not found: type PlanTest [error] class QueryTest extends PlanTest { [error] ^ [error] /Users/srowen/Documents/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:28: package org.apache.spark.sql.test is not a value [error] test("SPARK-1669: cacheTable should be idempotent") { [error] ^ ... ``` The issue I believe is that generation of a `test-jar` is bound here to the `compile` phase, but the test classes are not being compiled in this phase. It should bind to the `test-compile` phase. It works when executing `mvn package` or `mvn install` since test-jar artifacts are actually generated available through normal Maven mechanisms as each module is built. They are then found normally, regardless of scalatest configuration. It would be nice for a simple `mvn compile test-compile` to work since the test code is perfectly compilable given the Maven declarations. On the plus side, this change is low-risk as it only affects tests. yhuai made the original scalatest change and has glanced at this and thinks it makes sense. Author: Sean Owen <[email protected]> Closes #1879 from srowen/SPARK-2955 and squashes the following commits: ad8242f [Sean Owen] Generate test-jar on test-compile for modules whose tests are needed by others' tests Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e1b85f31 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e1b85f31 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e1b85f31 Branch: refs/heads/master Commit: e1b85f3102e5e25d0168b80aa953e1e76054a945 Parents: 500f84e Author: Sean Owen <[email protected]> Authored: Thu Aug 14 22:08:44 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Thu Aug 14 22:08:44 2014 -0700 ---------------------------------------------------------------------- sql/catalyst/pom.xml | 28 ++++++++++++++-------------- streaming/pom.xml | 8 ++++---- 2 files changed, 18 insertions(+), 18 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/e1b85f31/sql/catalyst/pom.xml ---------------------------------------------------------------------- diff --git a/sql/catalyst/pom.xml b/sql/catalyst/pom.xml index 58d44e7..830711a 100644 --- a/sql/catalyst/pom.xml +++ b/sql/catalyst/pom.xml @@ -77,28 +77,28 @@ <!-- This plugin forces the generation of jar containing catalyst test classes, so that the tests classes of external modules can use them. The two execution profiles - are necessary - first one for 'mvn package', second one for 'mvn compile'. Ideally, + are necessary - first one for 'mvn package', second one for 'mvn test-compile'. Ideally, 'mvn compile' should not compile test classes and therefore should not need this. However, an open Maven bug (http://jira.codehaus.org/browse/MNG-3559) causes the compilation to fail if catalyst test-jar is not generated. Hence, the - second execution profile for 'mvn compile'. + second execution profile for 'mvn test-compile'. --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <executions> - <execution> - <goals> - <goal>test-jar</goal> - </goals> - </execution> - <execution> - <id>test-jar-on-compile</id> - <phase>compile</phase> - <goals> - <goal>test-jar</goal> - </goals> - </execution> + <execution> + <goals> + <goal>test-jar</goal> + </goals> + </execution> + <execution> + <id>test-jar-on-test-compile</id> + <phase>test-compile</phase> + <goals> + <goal>test-jar</goal> + </goals> + </execution> </executions> </plugin> </plugins> http://git-wip-us.apache.org/repos/asf/spark/blob/e1b85f31/streaming/pom.xml ---------------------------------------------------------------------- diff --git a/streaming/pom.xml b/streaming/pom.xml index 1072f74..ce35520 100644 --- a/streaming/pom.xml +++ b/streaming/pom.xml @@ -81,11 +81,11 @@ <!-- This plugin forces the generation of jar containing streaming test classes, so that the tests classes of external modules can use them. The two execution profiles - are necessary - first one for 'mvn package', second one for 'mvn compile'. Ideally, + are necessary - first one for 'mvn package', second one for 'mvn test-compile'. Ideally, 'mvn compile' should not compile test classes and therefore should not need this. However, an open Maven bug (http://jira.codehaus.org/browse/MNG-3559) causes the compilation to fail if streaming test-jar is not generated. Hence, the - second execution profile for 'mvn compile'. + second execution profile for 'mvn test-compile'. --> <plugin> <groupId>org.apache.maven.plugins</groupId> @@ -97,8 +97,8 @@ </goals> </execution> <execution> - <id>test-jar-on-compile</id> - <phase>compile</phase> + <id>test-jar-on-test-compile</id> + <phase>test-compile</phase> <goals> <goal>test-jar</goal> </goals> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
