[
https://issues.apache.org/jira/browse/SPARK-49050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Marnadi updated SPARK-49050:
---------------------------------
Description: Fully integrating the TransformWithState operator with Virtual
Column Families by adding support for the deleteIfExists operator. In
RocksDB.scala, return the columnFamilyId if it exists in
{{removeColFamilyIfExists}} so that we can remove all the keys for the
corresponding columnFamily once deleteIfExists is called. (was: Instead of
creating a new column family for each new state variable, we want to use
virtual column families. We need to write the columnFamilyId to the State
Schema file, and read this in and pass this into the StateStoreProvider.)
Summary: nabling deleteIfExists operator in TWS with Virtual Column
Families (was: Integrate TransformWithState operator with Virtual Column
Families)
> nabling deleteIfExists operator in TWS with Virtual Column Families
> -------------------------------------------------------------------
>
> Key: SPARK-49050
> URL: https://issues.apache.org/jira/browse/SPARK-49050
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Eric Marnadi
> Priority: Major
> Labels: pull-request-available
>
> Fully integrating the TransformWithState operator with Virtual Column
> Families by adding support for the deleteIfExists operator. In RocksDB.scala,
> return the columnFamilyId if it exists in {{removeColFamilyIfExists}} so that
> we can remove all the keys for the corresponding columnFamily once
> deleteIfExists is called.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
