This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 3ae531e [SPARK-28205][SQL] useV1SourceList configuration should be
for all data sources
add 0a4f985c [SPARK-23098][SQL] Migrate Kafka Batch source to v2.
No new revisions were added by this update.
Summary of changes:
.../{KafkaRelation.scala => KafkaBatch.scala} | 79 ++-----
.../sql/kafka010/KafkaBatchPartitionReader.scala | 108 ++++++++++
.../spark/sql/kafka010/KafkaBatchWrite.scala | 67 ++++++
...aStreamingWrite.scala => KafkaDataWriter.scala} | 54 +----
.../spark/sql/kafka010/KafkaMicroBatchStream.scala | 88 +-------
.../spark/sql/kafka010/KafkaOffsetReader.scala | 28 +++
.../apache/spark/sql/kafka010/KafkaRelation.scala | 36 +---
.../apache/spark/sql/kafka010/KafkaSource.scala | 2 +-
.../spark/sql/kafka010/KafkaSourceProvider.scala | 61 ++++--
.../spark/sql/kafka010/KafkaStreamingWrite.scala | 57 +----
.../sql/kafka010/KafkaContinuousSinkSuite.scala | 2 +-
.../kafka010/KafkaDontFailOnDataLossSuite.scala | 21 +-
.../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +-
.../spark/sql/kafka010/KafkaRelationSuite.scala | 53 ++++-
.../apache/spark/sql/kafka010/KafkaSinkSuite.scala | 233 +++++++++++----------
.../org/apache/spark/sql/DataFrameWriter.scala | 2 +-
.../spark/sql/execution/streaming/CommitLog.scala | 2 +-
.../streaming/CompactibleFileStreamLog.scala | 2 +-
.../sql/execution/streaming/HDFSMetadataLog.scala | 2 +-
.../sql/execution/streaming/OffsetSeqLog.scala | 2 +-
.../sources/RateStreamMicroBatchStream.scala | 2 +-
.../execution/streaming/HDFSMetadataLogSuite.scala | 18 +-
22 files changed, 484 insertions(+), 437 deletions(-)
copy
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/{KafkaRelation.scala
=> KafkaBatch.scala} (54%)
create mode 100644
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala
create mode 100644
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchWrite.scala
copy
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/{KafkaStreamingWrite.scala
=> KafkaDataWriter.scala} (55%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
