In general, Mnemonic can be integrated into many projects. First, projects can use Mnemonic to take data off Java heap so GC can be much reduced, and use GET/SET to access data fields so serdes can be eliminated. Later we can expand Mnemonic to excise persistent/non-volatile programming on large scaled distributed systems with TB sized fast persistent memory devices.
Regarding solving Hadoop Namenode pressure of large scale of cluster scenarios. This issue is due to HDFS. Last year we found the use of FileInputStream in HDFS causes unpredicted long Garbage Collection pauses due to the overhead of finalizers and significantly impacted HDFS performance and its scalability. We recorded the issue in https://issues.apache.org/jira/browse/HDFS-8562 Uma explained what we can do for using Mnemonic to improve HDFS performance and scalability. One big advantage is Mnemonic does not need to hold File System cache for random access, which will benefit large scale of clusters. Thanks yanping -----Original Message----- From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] Sent: Tuesday, February 23, 2016 8:06 PM To: general@incubator.apache.org Subject: Re: [DISCUSS] Mnemonic incubator proposal Hi Liang, Thank you for your interest. Sure we would consider you adding in interested contributors list. >Mnemonic is trying to solve performance issues associated with serialization/deserialization of java object when dealing with JVM & disk directly as well as GC pressure caused by caching ? Yes. >whether Mnemonic could solve Hadoop Namenode pressure of large scale of cluster scenaros, or not? Yeah, we are thinking on some aspects considering memory and GC overheads in Namenode too. Example couple of JIRAs already there in HDFS to move some of data structure to off heap. So, we had plans to get the standard data structures from this library and can make use of them push. Also we could make advantage if persistence here. @Yanping/Gary, may be you could add more points if you have? [Gary] Thanks Uma, in addition, you can plug-in your special allocators that could be optimized for namenode usage patterns, by this way, the performance could be better and more predictable. Thanks. Regards, Uma On 2/23/16, 6:48 PM, "Liang Chen" <chenliang...@huawei.com> wrote: >Interesting, would love to become the contributor > >My understanding: Mnemonic is trying to solve performance issues >associated with serialization/deserialization of java object when >dealing with JVM & disk directly as well as GC pressure caused by >caching ? > >one question: whether Mnemonic could solve Hadoop Namenode pressure of >large scale of cluster scenaros, or not? > > > >-- >View this message in context: >http://apache-incubator-general.996316.n3.nabble.com/DISCUSS-Mnemonic-i >ncu >bator-proposal-tp48502p48533.html >Sent from the Apache Incubator - General mailing list archive at >Nabble.com. > >--------------------------------------------------------------------- >To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >For additional commands, e-mail: general-h...@incubator.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
