PermGen per Solr core?
Hi, I am running multiple Solr cores (solr-tomcat 1.4.0+ds1-1ubuntu1) under Tomcat (6.0.24-2ubuntu1.4) on Ubuntu 10.04.1. I have a master server where all Solr writes go, and a slave server that replicates all cores from the master, and accepts all read-only queries. After maxing out PermGen space a few times on the slave, adjusting it, and doing some analysis, it looks to me that each Solr core on the slave uses about 4.6 meg of PermGen space. I have looked at the number of Solr cores running each time I have maxed out PermGen and this does appear to be linear. To get the 4.6 meg/core number I am simply dividing MaxPermSize by the number of cores running when PermGen has filled. I am also using PrintGCDetails which confirms PermGen is maxed when the server fails. I have a basic understanding of PermGen and CMS garbage collection. Although 4.6 meg generally sounds like a small amount of data, with respect to tenured generation per Solr core it sounds high to me. Maybe I just don't know what I'm talking about, but I was hoping someone had an opinion on whether or not this sounded too high, and if so, what I can do to shrink it. I have read that redeploying servlets inherently makes this problem worse, but I very rarely do that. This seems to be a case of PermGen filling as more Solr cores are added to my server (a few are added each day as part of normal business). If there is any additional information I can provide about my deployment that would help please let me know. Thanks. Andrew Davidoff
Index version on slave incrementing to higher than master
Hi,
I am running solr 1.4.0+ds1-1ubuntu1. I have a master server that has a
number of solr instances running on it (150 or so), and nightly most of
them have documents written to them. The script that does these writes
(adds) does a commit and an optimize on the indexes when it's entirely
finished updating them, then initiates replication on the slave per
instance. In this configuration, the index versions between master and
slave remain in synch.
The optimize portion, which, again, happens nightly, is taking a lot of
time and I think it's unnecessary. I was hoping to stop doing this explicit
optimize, and to let my merge policy handle that. However, if I don't do an
optimize, and only do a commit before initiating slave replication, some
hours later the slave is, for reasons that are unclear to me, incrementing
its index version to 1 higher than the master.
I am not really sure I understand the logs, but it looks like the
incremented index version is the result of an optimize on the slave, but I
am never issuing any commands against the slave aside from initiating
replication, and I don't think there's anything in my solr configuration
that would be initiating this. I do have autoCommit on with maxDocs of
1000, but since I am initiating slave replication after doing a commit on
the master, I don't think there would ever be any uncommitted documents on
the slave. I do have a merge policy configured, but it's not clear to me
that it has anything to do with this. And if it did, I'd expect to see
similar behavior on the master (right?).
I have included a snipped from my slave logs that shows this issue. In this
snipped index version 1286065171264 is what the master has,
and 1286065171265 is what the slave increments itself to, which is then out
of synch with the master in terms of version numbers. Nothing that I know
of is issuing any commands to the slave at this time. If I understand these
logs (I might not), it looks like something issued an optimize that took
1023720ms? Any ideas?
Thanks in advance.
Andy
Jul 12, 2012 12:21:14 PM org.apache.solr.update.SolrIndexWriter close
FINE: Closing Writer DirectUpdateHandler2
Jul 12, 2012 12:21:14 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h8,version=1286065171264,generation=620,filenames=[_h6.fnm,
_h5.nrm, segments_h8, _h4.nrm, _h5.tii, _h4
.tii, _h5.tis, _h4.tis, _h4.fdx, _h5.fnm, _h6.tii, _h4.fdt, _h5.fdt,
_h5.fdx, _h5.frq, _h4.fnm, _h6.frq, _h6.tis, _h4.prx, _h4.frq, _h6.nrm,
_h5.prx, _h6.prx, _h6.fdt, _h6
.fdx]
commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h9,version=1286065171265,generation=621,filenames=[_h7.tis,
_h7.fdx, _h7.fnm, _h7.fdt, _h7.prx, segment
s_h9, _h7.nrm, _h7.tii, _h7.frq]
Jul 12, 2012 12:21:14 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1286065171265
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher
INFO: Opening Searcher@4ac62082 main
Jul 12, 2012 12:21:14 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative
_inserts=0,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=18,cumulative_hits=14,cumulative_hitratio=0.77,cumulative_inserts=4,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=18,cumulative_hits=14,cumulative_hitratio=0.77,cumulative_inserts=4,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=150,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=69,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0
Re: Index version on slave incrementing to higher than master
Erick,
Thank you. I think originally my thought was that if I had my slave
configuration really close to my master config, it would be very easy to
promote a slave to a master (and vice versa) if necessary. But I think you
are correct that ripping out from the slave config anything that would
modify an index in any way makes sense. I will give this a try very soon.
Thanks again.
Andy
On Sat, Jul 14, 2012 at 5:22 PM, Erick Erickson wrote:
> Gotta admit it's a bit puzzling, and surely you want to move to the 3x
> versions ..
>
> But at a guess, things might be getting confused on the slaves given
> you have a merge policy on them. There's no reason to have any
> policies on the slaves; slaves should just be about copying the files
> from the master, all the policies,commits,optimizes should be done on
> the master. About all the slave does is copy the current state of the index
> from the master.
>
> So I'd try removing everything but the replication from the slaves,
> including
> any autocommit stuff and just let replication do it's thing.
>
> And I'd replicate after the optimize if you keep the optimize going. You
> should
> end up with one segment in the index after that, on both the master and
> slave.
> You can't get any more merged than that.
>
> Of course you'll also copy the _entire_ index every time after you've
> optimized...
>
> Best
> Erick
>
> On Fri, Jul 13, 2012 at 12:31 AM, Andrew Davidoff
> wrote:
> > Hi,
> >
> > I am running solr 1.4.0+ds1-1ubuntu1. I have a master server that has a
> > number of solr instances running on it (150 or so), and nightly most of
> > them have documents written to them. The script that does these writes
> > (adds) does a commit and an optimize on the indexes when it's entirely
> > finished updating them, then initiates replication on the slave per
> > instance. In this configuration, the index versions between master and
> > slave remain in synch.
> >
> > The optimize portion, which, again, happens nightly, is taking a lot of
> > time and I think it's unnecessary. I was hoping to stop doing this
> explicit
> > optimize, and to let my merge policy handle that. However, if I don't do
> an
> > optimize, and only do a commit before initiating slave replication, some
> > hours later the slave is, for reasons that are unclear to me,
> incrementing
> > its index version to 1 higher than the master.
> >
> > I am not really sure I understand the logs, but it looks like the
> > incremented index version is the result of an optimize on the slave, but
> I
> > am never issuing any commands against the slave aside from initiating
> > replication, and I don't think there's anything in my solr configuration
> > that would be initiating this. I do have autoCommit on with maxDocs of
> > 1000, but since I am initiating slave replication after doing a commit on
> > the master, I don't think there would ever be any uncommitted documents
> on
> > the slave. I do have a merge policy configured, but it's not clear to me
> > that it has anything to do with this. And if it did, I'd expect to see
> > similar behavior on the master (right?).
> >
> > I have included a snipped from my slave logs that shows this issue. In
> this
> > snipped index version 1286065171264 is what the master has,
> > and 1286065171265 is what the slave increments itself to, which is then
> out
> > of synch with the master in terms of version numbers. Nothing that I know
> > of is issuing any commands to the slave at this time. If I understand
> these
> > logs (I might not), it looks like something issued an optimize that took
> > 1023720ms? Any ideas?
> >
> > Thanks in advance.
> >
> > Andy
> >
> >
> >
> > Jul 12, 2012 12:21:14 PM org.apache.solr.update.SolrIndexWriter close
> > FINE: Closing Writer DirectUpdateHandler2
> > Jul 12, 2012 12:21:14 PM org.apache.solr.core.SolrDeletionPolicy onCommit
> > INFO: SolrDeletionPolicy.onCommit: commits:num=2
> >
> >
> commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h8,version=1286065171264,generation=620,filenames=[_h6.fnm,
> > _h5.nrm, segments_h8, _h4.nrm, _h5.tii, _h4
> > .tii, _h5.tis, _h4.tis, _h4.fdx, _h5.fnm, _h6.tii, _h4.fdt, _h5.fdt,
> > _h5.fdx, _h5.frq, _h4.fnm, _h6.frq, _h6.tis, _h4.prx, _h4.frq, _h6.nrm,
> > _h5.prx, _h6.prx, _h6.fdt, _h6
> > .fdx]
> >
> >
> commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h9,version=1286065171265,generation=621,filenames=[_h7.tis,
> > _h7.fdx, _h7.fnm, _h7.fdt, _h7.prx, segme
Re: Index version on slave incrementing to higher than master
Thanks Erick, I will look harder at our current configuration and how we're handling config replication, but I just realized that a backup script was doing a commit and an optimize on the slave prior to taking the backup. This happens daily, after updates and replication from the master. This is something I put in place many ages ago and didn't think to look at until now :/ Based on the times in the logs and the conditions under which my problem was occurring (when I wasn't optimizing on the master before initiating replication) it seems clear that this backup script is my problem. Sorry for taking your time with something that was clearly my own dang fault. I appreciate your suggestions and responses regardless! Andy On Mon, Jul 16, 2012 at 7:35 AM, Erick Erickson wrote: > Andrew: > > I'm not entirely sure that's your problem, but it's the first thing I'd > try. > > As for your config files, see the section "Replicating solrconfig.xml" > here: http://wiki.apache.org/solr/SolrReplication. That at least > allows you to centralize separate solrconfigs for master and > slave, making promoting a slave to a master a bit easier.... > > Best > Erick > > On Sun, Jul 15, 2012 at 2:00 PM, Andrew Davidoff > wrote: > > Erick, > > > > Thank you. I think originally my thought was that if I had my slave > > configuration really close to my master config, it would be very easy to > > promote a slave to a master (and vice versa) if necessary. But I think > you > > are correct that ripping out from the slave config anything that would > > modify an index in any way makes sense. I will give this a try very soon. > > > > Thanks again. > > Andy > > > > > > On Sat, Jul 14, 2012 at 5:22 PM, Erick Erickson >wrote: > > > >> Gotta admit it's a bit puzzling, and surely you want to move to the 3x > >> versions .. > >> > >> But at a guess, things might be getting confused on the slaves given > >> you have a merge policy on them. There's no reason to have any > >> policies on the slaves; slaves should just be about copying the files > >> from the master, all the policies,commits,optimizes should be done on > >> the master. About all the slave does is copy the current state of the > index > >> from the master. > >> > >> So I'd try removing everything but the replication from the slaves, > >> including > >> any autocommit stuff and just let replication do it's thing. > >> > >> And I'd replicate after the optimize if you keep the optimize going. You > >> should > >> end up with one segment in the index after that, on both the master and > >> slave. > >> You can't get any more merged than that. > >> > >> Of course you'll also copy the _entire_ index every time after you've > >> optimized... > >> > >> Best > >> Erick > >> > >> On Fri, Jul 13, 2012 at 12:31 AM, Andrew Davidoff > >> wrote: > >> > Hi, > >> > > >> > I am running solr 1.4.0+ds1-1ubuntu1. I have a master server that has > a > >> > number of solr instances running on it (150 or so), and nightly most > of > >> > them have documents written to them. The script that does these writes > >> > (adds) does a commit and an optimize on the indexes when it's entirely > >> > finished updating them, then initiates replication on the slave per > >> > instance. In this configuration, the index versions between master and > >> > slave remain in synch. > >> > > >> > The optimize portion, which, again, happens nightly, is taking a lot > of > >> > time and I think it's unnecessary. I was hoping to stop doing this > >> explicit > >> > optimize, and to let my merge policy handle that. However, if I don't > do > >> an > >> > optimize, and only do a commit before initiating slave replication, > some > >> > hours later the slave is, for reasons that are unclear to me, > >> incrementing > >> > its index version to 1 higher than the master. > >> > > >> > I am not really sure I understand the logs, but it looks like the > >> > incremented index version is the result of an optimize on the slave, > but > >> I > >> > am never issuing any commands against the slave aside from initiating > >> > replication, and I don't think there's anything in my solr > configuration > >> > that would be
