nlu90 commented on a change in pull request #2891: [WIP] Refactor
StatefulStorage
URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r186224681
##########
File path:
heron/spi/src/java/org/apache/heron/spi/statefulstorage/IStatefulStorage.java
##########
@@ -16,37 +16,65 @@
import java.util.Map;
-import org.apache.heron.proto.system.PhysicalPlans;
-
+/**
+ * The interface of all storage classes for checkpoints.
+ * For each checkpoint, two types of data are stored:
+ * - Component Meta Data (one per component).
+ * - Instance Checkpoint Data (one per instance or patition)
+ * Each Stateful Storage implementation needs to handle them accordingly.
+ */
public interface IStatefulStorage {
/**
* Initialize the Stateful Storage
- *
+ * @param topologyName The name of the topology.
* @param conf An unmodifiableMap containing basic configuration
- * Attempts to modify the returned map,
- * whether direct or via its collection views, result in an
UnsupportedOperationException.
*/
- void init(Map<String, Object> conf) throws StatefulStorageException;
+ void init(String topologyName, final Map<String, Object> conf)
+ throws StatefulStorageException;
/**
* Closes the Stateful Storage
*/
void close();
- // Store the checkpoint
- void store(Checkpoint checkpoint) throws StatefulStorageException;
-
- // Retrieve the checkpoint
- Checkpoint restore(String topologyName, String checkpointId,
- PhysicalPlans.Instance instanceInfo) throws
StatefulStorageException;
-
- // TODO(mfu): We should refactor all interfaces in IStatefulStorage,
- // TODO(mfu): instead providing Class Checkpoint, we should provide an
Context class,
- // TODO(mfu): It should:
- // TODO(mfu): 1. Provide meta data access, like topologyName
- // TODO(mfu): 2. Provide utils method to parse the protobuf object, like
getTaskId()
- // TODO(mfu): 3. Common methods, like getCheckpointDir()
- // Dispose the checkpoint
- void dispose(String topologyName, String oldestCheckpointId, boolean
deleteAll)
- throws StatefulStorageException;
+ /**
+ * Store instance checkpoint.
+ * @param info The information (reference key) for the checkpoint partition.
+ * @param checkpoint The checkpoint data.
+ */
+ void storeCheckpoint(final CheckpointPartitionInfo info, final Checkpoint
checkpoint)
+ throws StatefulStorageException;
+
+ /**
+ * Retrieve instance checkpoint.
+ * @param info The information (reference key) for the checkpoint partition.
+ * @return The checkpoint data from the specified blob id.
+ */
+ Checkpoint restoreCheckpoint(final CheckpointPartitionInfo info) throws
StatefulStorageException;
+
+ /**
+ * Store medata data for component. Ideally this function should only be
called once
+ * for each component.
+ * @param info The information (reference key) for the checkpoint partition.
+ * @param metadata The checkpoint metadata from a component.
+ */
+ void storeComponentMetaData(final CheckpointPartitionInfo info, final
CheckpointMetadata metadata)
Review comment:
If only instance 0 writes the metadata, then in the case of states are
stored on local disks within each container, all other instances will not have
access to the metadata. Just like the problem we met previously for deleting
states from ckptmgr-0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
