[
https://issues.apache.org/jira/browse/GEARPUMP-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251495#comment-15251495
]
Manu Zhang commented on GEARPUMP-63:
------------------------------------
Comments history,
[~kkasravi]
{quote}
Perhaps the different types of storage could share a common API or aspects of
collections so AppStorage doesn't need to change every time a new type of
storage is added. Also at least with JarStore see
[#1204|https://github.com/gearpump/gearpump/issues/1204] where lack of
streaming causes Master to exit.
Should AppStorage contain write, read methods?
{quote}
[~whjiang]
{quote}
Hi @kkasravi, I agree that sharing common API is one goal. This is why the
checkpoint store and offset store in current codebase are unified.
The read/write methods are for KVStore instead of AppStorage. AppStorage is a
quite general concept. It includes everything an application needs for storage
(except connector).
{quote}
[~clockfly]
{quote}
+1 for the ideas. The difficulty will be how to make the storage implementation
plugable.
So that user can choose the storage system they want, like casandra, hdfs, or
hbase...
{quote}
>From [netcomm|https://github.com/netcomm],
{quote}
maybe i lose something, why not just base on akka persistence, it now support
hbase/casandra/... plug-ins.
{quote}
The rest conversations are about using akka persistence as a general storage
solution and I'll create a new jira for it.
> Gearpump Storage framework
> --------------------------
>
> Key: GEARPUMP-63
> URL: https://issues.apache.org/jira/browse/GEARPUMP-63
> Project: Apache Gearpump
> Issue Type: New Feature
> Reporter: Manu Zhang
> Assignee: Weihua Jiang
>
> imported from https://github.com/gearpump/gearpump/issues/1197 on behalf of
> [~whjiang]. His original proposal,
> In general, a Gearpump application requires following storage support:
> # Jar-file storage to store the application jar file(s).
> # application log. Currently we store logs in each node which makes
> application log analysis difficult.
> # application metrics.
> # application configuration.
> # data source offset store (for at-least once semantics of streaming
> application)
> # application state checkpoint store (for transaction semantics)
> The general idea is:
> # Provide a storage system satisfied the above requirements.
> # Assume this storage is highly available. That means, it is user's duty to
> provide such kind of a storage. For test purpose, user can use some non-HA
> storage system. But, in product use, it shall be HAed.
> # Isolate usage from implementation. That is, Gearpump doesn't rely on
> Hadoop-common or HDFS or one specific implementation to provide such storage.
> User is free to implement its own storage.
> # This is a daemon provided functionality and can be used by every Gearpump
> application.
> # This storage shall provide data retentation functionality and access
> control.
> # This storage provides a set of API to meet the above requirements instead
> of one low-level API.
> # User can override the system setting to provide dedicated implementation
> for certain sub-storage system, e.g. chekcpoint store.
> # Akka replication shall store minimal info for an application and leave the
> majority to this storage system. I.e. akka replication is more like a seed to
> this storage system.
> # In release, each storage implementation (e.g. storage-hdfs) is a standalone
> module/artifact.
> The draft of this storage looks like (quite initial, tentative to change):
> {code}
> trait Storage {
> def createAppStorage(AppName, AppId) : AppStorage
> def getAppStorage(AppId) : Option[AppStorage]
> }
> trait AppStorage {
> def open
> def close
> def getJarStore: JarStore
> def getMetricsStore: AppMetricsStore
> def getKVStore: KVStore
> def getLogAppender: LogAppender
> def getConfiguration(ProcessorId): UserConfig
> def setConfiguration(ProcessorId, UserConfig)
> }
> trait JarStore {
> def copyFromLocal(localPath, remotePath)
> def copyToLocal(remotePath, localPath)
> }
> ///assume K is sortable
> trait KVStore[K,V] {
> def append(key, value)
> def read(key): Try[Option[V]]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
