This is an automated email from the ASF dual-hosted git repository.
ashrigondekar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 610bb52242c7 [SPARK-54104][GEO][SQL] Disallow casting geospatial types
to/from other data types
add 6d5f111cc86a [SPARK-51972][SS] Introduce State Store file integrity
verification using checksum
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/spark/internal/LogKeys.java | 1 +
.../src/main/resources/error/error-conditions.json | 7 +
.../spark/sql/errors/QueryExecutionErrors.scala | 17 +
.../org/apache/spark/sql/internal/SQLConf.scala | 11 +
.../checkpointing/CheckpointFileManager.scala | 2 +
.../ChecksumCheckpointFileManager.scala | 523 +++++++++++++++++++++
.../state/HDFSBackedStateStoreProvider.scala | 54 ++-
.../sql/execution/streaming/state/RocksDB.scala | 13 +-
.../streaming/state/RocksDBFileManager.scala | 54 ++-
.../sql/execution/streaming/state/StateStore.scala | 59 +++
.../streaming/state/StateStoreChangelog.scala | 11 +-
.../execution/streaming/state/StateStoreConf.scala | 3 +
.../StateDataSourceTransformWithStateSuite.scala | 3 +-
.../streaming/CheckpointFileManagerSuite.scala | 5 +-
.../ChecksumCheckpointFileManagerSuite.scala | 213 +++++++++
.../FailureInjectionCheckpointFileManager.scala | 4 +-
.../RocksDBCheckpointFailureInjectionSuite.scala | 11 +-
.../streaming/state/RocksDBStateStoreSuite.scala | 11 +
.../execution/streaming/state/RocksDBSuite.scala | 4 +
.../streaming/state/StateStoreSuite.scala | 292 +++++++++++-
20 files changed, 1272 insertions(+), 26 deletions(-)
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/ChecksumCheckpointFileManager.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ChecksumCheckpointFileManagerSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
