Thanks, Shawn and Shalin How does the frequency of commit affect zookeeper?
Thanks On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > SolrCloud will update Zookeeper on state changes (node goes to > recovery, comes back up etc) or for leader election and during > collection API commands. It doesn't correlate directly with indexing > but is correlated with how frequently you call commit. > > On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <s...@elyograg.org> wrote: > > On 3/18/2014 5:46 PM, Chris W wrote: > >> > >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into issues > >> with Zookeeper transaction logs > >> > >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on > disk > >> java.io.IOException: Unreasonable length = 1048587 > >> at > >> > org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100) > >> at > >> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233) > >> at > >> > >> > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602) > >> at > >> > >> > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157) > >> at > >> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) > >> > >> To unblock temporarily, i deleted the most recent txn log. How do i tell > >> zookeeper to not grow the transaction log beyond x MegaBytes? > >> > >> How often does the transaction log get updated? Does zk transactions log > >> grow everytime we index data into a new collection? > > > > > > Zookeeper is a separate project at Apache. ZK file management is > discussed > > in the ZK documentation. > > > > > http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance > > > > There is a bug filed on Zookeeper for the issue you are seeing, with a > > fairly simple workaround. It is fixed in the 3.4.6 version, which was > > released last week. I will see whether we can get ZK upgraded to 3.4.6 > in > > the Solr 4.8.0 release. I don't think we want to risk doing that > upgrade in > > 4.7.1, but I could be wrong. > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1513 > > http://zookeeper.apache.org/releases.html > > > > I am actually not sure how often SolrCloud updates Zookeeper. It happens > > whenever the collections API is called for sure, and it may happen > anytime > > you index data as well. > > > > Thanks, > > Shawn > > > > > > -- > Regards, > Shalin Shekhar Mangar. > -- Best -- C
