Repository: incubator-edgent Updated Branches: refs/heads/master 99b7c8e21 -> be0d4b199
[Edgent-325] improve jdbc, kafka, and java7 test doc Project: http://git-wip-us.apache.org/repos/asf/incubator-edgent/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-edgent/commit/0d588ae6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-edgent/tree/0d588ae6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-edgent/diff/0d588ae6 Branch: refs/heads/master Commit: 0d588ae6d509540f414dac96a4a2785800f1c3ba Parents: 99b7c8e Author: Dale LaBossiere <dlab...@us.ibm.com> Authored: Wed Dec 21 17:06:33 2016 -0500 Committer: Dale LaBossiere <dlab...@us.ibm.com> Committed: Wed Dec 21 17:06:33 2016 -0500 ---------------------------------------------------------------------- DEVELOPMENT.md | 65 ++++++++++++++++++-- .../test/connectors/jdbc/JdbcStreamsTest.java | 10 ++- .../kafka/KafkaStreamsTestManual.java | 43 +++++++++++++ scripts/connectors/kafka/README | 4 +- scripts/connectors/kafka/README-kafka | 25 ++++++++ 5 files changed, 137 insertions(+), 10 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/DEVELOPMENT.md ---------------------------------------------------------------------- diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 00be84b..f64a0ed 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -104,8 +104,7 @@ The build process has been tested on Linux and macOS. To build on Windows probably needs some changes, please get involved and contribute them! - -**Continuous Integration** +### Continuous Integration When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build. @@ -160,6 +159,57 @@ Running the `reports` target produces two reports: * `builds/distributions/reports/tests/index.html` - JUnit test report * `builds/distributions/reports/coverage/index.html` - Code coverage report. +### Testing the Kafka Connector + +The kafka connector tests aren't run by default as the connector must +connect to a running Kafka/Zookeeper config. + +There are apparently ways to embedd Kafka and Zookeeper for testing purposes but +we're not there yet. Contributions welcome! + +Setting up the servers is easy. +Follow the steps in the [KafkaStreamsTestManual](connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java) javadoc. + +Once kafka/zookeeper are running you can run the tests and samples: +```sh +#### run the kafka tests +./gradlew connectors:kafka:test --tests '*.*Manual' + +#### run the sample +cd java8/scripts/connectors/kafka +cat README +./runkafkasample.sh sub +./runkafkasample.sh pub +``` + +### Testing the JDBC Connector + +The JDBC connector tests are written to run against Apache Derby +as the backing dbms and the derby jar needs to be on the classpath. +The tests are skipped if derby can't be loaded. +The test harness adds $DERBY_HOME/db/lib/derby.jar to the classpath. +See [JdbcStreamsTest](connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java) +for more info but the following should suffice: + +```sh +export DERBY_HOME=$JAVA_HOME/db + +#### if JAVA_HOME isn't set - e.g., on OSX... +export DERBY_HOME=`/usr/libexec/java_home`/db +``` + +Once DERBY_HOME is set the tests and samples can be run as follows +```sh +#### run the jdbc tests +./gradlew connectors:jdbc:test + +#### run the sample +cd java8/scripts/connectors/jdbc +cat README +./runjdbcsample.sh writer +./runjdbcsample.sh reader +``` + ### Testing Edgent with Java7 All of the standard build system _tasks_ above must be run with @@ -173,14 +223,17 @@ See [JAVA_SUPPORT](JAVA_SUPPORT.md) for information about what Edgent features are supported in the different environments. ``` sh - # run with JAVA_HOME set for Java8 + # run with JAVA_HOME/PATH set for Java8 $ ./gradlew test7Compile # compile the Edgent tests to operate in a Java7 environment - # run with JAVA_HOME set for Java7 -$ ./gradlew test7Run # run the tests with a Java7 VM +$ sh # muck with EVs for Java7 in a subshell + $ export JAVA_HOME=`/usr/libexec/java_home -v 1.7` # on OSX + $ export PATH=$JAVA_HOME/bin:$PATH + $ ./gradlew test7Run # run the tests with a Java7 VM + $ exit # run with JAVA_HOME set for Java8 -$ ./gradlew test7Reports # generate the JUnit and coverage tests +$ ./gradlew test7Reports # generate the JUnit and coverage reports ``` ### Publish to Maven Repository http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java ---------------------------------------------------------------------- diff --git a/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java b/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java index 4b1e8ca..a15b9d9 100644 --- a/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java +++ b/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java @@ -52,9 +52,15 @@ import org.junit.Test; * Manually install Derby for other JDKs if required. * Arrange for the classpath to be configured by one of: * <ul> - * <li>set the DERBY_HOME environment variable. build.xml adds - * $DERBY_HOME/lib/derby.jar to the classpath when running the tests.</li> * <li>manually add derby.jar to the classpath</li> + * <li>set the DERBY_HOME environment variable. connectors/jdbc/build.gradle adds + * $DERBY_HOME/lib/derby.jar to the classpath when running the tests. + * e.g., try + * <ul> + * <li> export DERBY_HOME=$JAVA_HOME/db</li> + * <li> OSX: export DERBY_HOME=`/usr/libexec/java_home`/db + * </ul> + * </li> * </ul> * The tests are "skipped" if the dbms's jdbc driver can't be found. */ http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java ---------------------------------------------------------------------- diff --git a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java index 4328af8..0a9c3ab 100644 --- a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java +++ b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java @@ -37,6 +37,49 @@ import org.apache.edgent.topology.Topology; import org.apache.edgent.topology.plumbing.PlumbingStreams; import org.junit.Test; +/** + * Test the Kafka connector. + * <p> + * The tests expect a Kafka/Zookeeper running on the local host at their + * default ports: 9092 and 2181 respectively. + * <p> + * The following system properties may be used to override that: + * <ul> + * <li>org.apache.edgent.test.connectors.kafka.bootstrap.servers=localhost:9092</li> + * <li>org.apache.edgent.test.connectors.kafka.zookeeper.connect=localhost:2181</li> + * </ul> + * <p> + * Setting up a Kafka/Zookeeper config on the default localhost ports is simple + * and well documented at https://kafka.apache.org/quickstart. This should do it: + * <p> + * After downloading kafka: + * <pre>{@code + * tar zxf ~/Downloads/kafka_2.11-0.10.1.0.tgz + * cd kafka_2.11-0.10.1.0/ + * + * # start the servers (best in separate windows) + * bin/zookeeper-server-start.sh config/zookeeper.properties + * bin/kafka-server-start.sh config/server.properties + * }</pre> + * + * <p> + * The tests and sample require certain test topics. Create them: + * <pre>{@code + * # create our kafka test and sample topics + * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic1 + * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic2 + * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaSampleTopic + * bin/kafka-topics.sh --list --zookeeper localhost:2181 + * + * # quick verify + * bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic1 + * hi + * there + * ^D + * bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic1 --from-beginning + * ... you should see the "hi" and "there" messages. + * }</pre> + */ public class KafkaStreamsTestManual extends ConnectorTestBase { private static final int PUB_DELAY_MSEC = 4*1000; private static final int SEC_TIMEOUT = 10; http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/scripts/connectors/kafka/README ---------------------------------------------------------------------- diff --git a/scripts/connectors/kafka/README b/scripts/connectors/kafka/README index f038193..b1c6bec 100644 --- a/scripts/connectors/kafka/README +++ b/scripts/connectors/kafka/README @@ -6,8 +6,8 @@ By default the samples require the following kafka broker configuration: - kafka topic "kafkaSampleTopic" exists - no authentication -See http://kafka.apache.org for the code and setup information for -a Kafka broker. +See README-kafka for information about setting up a kafka server +and creating the topic. The source code for the samples is in the <edgent-release>/samples directory. http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/scripts/connectors/kafka/README-kafka ---------------------------------------------------------------------- diff --git a/scripts/connectors/kafka/README-kafka b/scripts/connectors/kafka/README-kafka new file mode 100644 index 0000000..e253963 --- /dev/null +++ b/scripts/connectors/kafka/README-kafka @@ -0,0 +1,25 @@ +Setting up a Kafka/Zookeeper config on the default localhost ports is simple +and well documented at https://kafka.apache.org/quickstart. This should do it: + +After downloading kafka: + +tar zxf ~/Downloads/kafka_2.11-0.10.1.0.tgz +cd kafka_2.11-0.10.1.0/ + +# start the servers (best in separate windows) +bin/zookeeper-server-start.sh config/zookeeper.properties +bin/kafka-server-start.sh config/server.properties + +The sample requires a topic. Create it: + +# create our kafka sample topic +bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaSampleTopic +bin/kafka-topics.sh --list --zookeeper localhost:2181 + +# quick verify +bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafkaSampleTopic +hi +there +^D +bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafkaSampleTopic --from-beginning +... you should see the "hi" and "there" messages.
