index backup works only if there are committed index

[solr jay]

Hi,

I noticed that the backup request

http://master_host:port/solr/replication?command=backup

works only if there are committed index data, i.e.
core.getDeletionPolicy().getLatestCommit() is not null. Otherwise, no backup
is created. It sounds logical because if nothing has been committed since
your last backup, it doesn't help much to do a new backup. However, consider
this scenario:

1. a backup process is scheduled at 1:00AM every Monday
2. just before 1:00AM, the system is shutdown (for whatever reason), and
then restarts
3. No index is committed before 1:00AM
4. at 1:00AM, backup process starts and no committed index is found, and
therefore no backup (until next week)

The probability of this scenario is probably small, but it still could
happen, and it seems to me that if I want to backup index, a backup should
be created whether there are new committed index or not.

Your thoughts?

Thanks,

-- 
J

enablereplication does not work

[solr jay]

Hi,

http://localhost:8549/solr/replication?command=enablereplication

does not seem working. After making the request, I run

http://localhost:8549/solr/replication?command=indexversion

and here is the response:




0
0

0
0



Notice the indexversion is 0, which is the value after you disable
replication. On the other hand

http://localhost:8549/solr/replication?command=details

returns:





0
7



692 bytes


  /tmp/solr/solrdata/index


true
false
1249517184279
2


commit




This response format is experimental.  It is likely to change in the future.




Notice that the indexversion is 1249517184279.

thanks,

-- 
J

Re: enablereplication does not work

[solr jay]

You are right. Replication was disabled after the server was restarted, and
then I saw the behavior. After I added some data, command "indexversion"
returns the right values. So it seems Solr behaved correctly.

Thanks,

2009/8/5 Noble Paul നോബിള്‍ नोब्ळ् 

> how is the replicationhandler configured? if there was no
> commit/optimize thhen it would show the version as '0'
>
> On Thu, Aug 6, 2009 at 5:50 AM, solr jay wrote:
> > Hi,
> >
> > http://localhost:8549/solr/replication?command=enablereplication
> >
> > does not seem working. After making the request, I run
> >
> > http://localhost:8549/solr/replication?command=indexversion
> >
> > and here is the response:
> >
> >
> > 
> > 
> > 0
> > 0
> > 
> > 0
> > 0
> > 
> >
> >
> > Notice the indexversion is 0, which is the value after you disable
> > replication. On the other hand
> >
> > http://localhost:8549/solr/replication?command=details
> >
> > returns:
> >
> >
> > 
> >
> > 
> > 0
> > 7
> > 
> >
> > 
> > 692 bytes
> >
> > 
> >  /tmp/solr/solrdata/index
> > 
> > 
> > true
> > false
> > 1249517184279
> > 2
> >
> > 
> > commit
> > 
> > 
> >
> > 
> > This response format is experimental.  It is likely to change in the
> future.
> > 
> > 
> >
> >
> > Notice that the indexversion is 1249517184279.
> >
> > thanks,
> >
> > --
> > J
> >
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>



-- 
J

Re: enablereplication does not work

[solr jay]

By the way, I was using command=indexversion to verify replication is on or
off. Since it seems not reliable, is there a better to do it?

Thanks,

On Thu, Aug 6, 2009 at 8:43 AM, solr jay  wrote:

> You are right. Replication was disabled after the server was restarted, and
> then I saw the behavior. After I added some data, command "indexversion"
> returns the right values. So it seems Solr behaved correctly.
>
> Thanks,
>
> 2009/8/5 Noble Paul നോബിള്‍ नोब्ळ् 
>
> how is the replicationhandler configured? if there was no
>> commit/optimize thhen it would show the version as '0'
>>
>> On Thu, Aug 6, 2009 at 5:50 AM, solr jay wrote:
>> > Hi,
>> >
>> > http://localhost:8549/solr/replication?command=enablereplication
>> >
>> > does not seem working. After making the request, I run
>> >
>> > http://localhost:8549/solr/replication?command=indexversion
>> >
>> > and here is the response:
>> >
>> >
>> > 
>> > 
>> > 0
>> > 0
>> > 
>> > 0
>> > 0
>> > 
>> >
>> >
>> > Notice the indexversion is 0, which is the value after you disable
>> > replication. On the other hand
>> >
>> > http://localhost:8549/solr/replication?command=details
>> >
>> > returns:
>> >
>> >
>> > 
>> >
>> > 
>> > 0
>> > 7
>> > 
>> >
>> > 
>> > 692 bytes
>> >
>> > 
>> >  /tmp/solr/solrdata/index
>> > 
>> > 
>> > true
>> > false
>> > 1249517184279
>> > 2
>> >
>> > 
>> > commit
>> > 
>> > 
>> >
>> > 
>> > This response format is experimental.  It is likely to change in the
>> future.
>> > 
>> > 
>> >
>> >
>> > Notice that the indexversion is 1249517184279.
>> >
>> > thanks,
>> >
>> > --
>> > J
>> >
>>
>>
>>
>> --
>> -
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>
>
>
> --
> J
>



-- 
J

cleanup old index directories on slaves

[solr jay]

Is there a reliable way to safely clean up index directories? This is needed
mainly on slave side as in several situations, an old index directory is
replaced with a new one, and I'd like to remove those that are no longer in
use.

Thanks,

-- 
J

solr health check

[solr jay]

Hi,

I am looking at this piece of configuration in solrconfig.xml


solr

 solrconfig.xml
 schema.xml

q=solr&version=2.0&start=0&rows=0


server-enabled
  


It wasn't clear to me what 'server-enabled' means here. Is it a file name?
If it is file name, where the file should be?

I added server-enabledand admin/ping
stopped working, which is good, but I couldn't make it work again, and admin
UI generate an exception. Anyone used this feature before?

Thanks,

J


HTTP ERROR: 500

PWC6033: Unable to compile class for JSP

PWC6197: An error occurred at line: 28 in the jsp file: /admin/action.jsp
PWC6199: Generated servlet error:
Type mismatch: cannot convert from Logger to Logger

PWC6197: An error occurred at line: 28 in the jsp file: /admin/action.jsp
PWC6199: Generated servlet error:
The method log(Level, String) is undefined for the type Logger



org.apache.jasper.JasperException: PWC6033: Unable to compile class for JSP

PWC6197: An error occurred at line: 28 in the jsp file: /admin/action.jsp
PWC6199: Generated servlet error:
Type mismatch: cannot convert from Logger to Logger

PWC6197: An error occurred at line: 28 in the jsp file: /admin/action.jsp
PWC6199: Generated servlet error:
The method log(Level, String) is undefined for the type Logger


at
org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:94)
at
org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:267)
at org.apache.jasper.compiler.Compiler.generateClass(Compiler.java:332)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:389)
at
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:579)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:344)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:367)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:268)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:273)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:295)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
RequestURI=/solr/admin/action.jsp

reindexed data on master not replicated to slave

[solr jay]

Hi,

When index data were corrupted on master instance, I wanted to wipe out all
the index data and re-index everything. I was hoping the newly created index
data would be replicated to slaves, but it wasn't.

Here are the steps I performed:

1. stop master
2. delete the directory 'index'
3. start master
4. disable replication on master
5. index all data from scratch
6. enable replication on master

It seemed from log file that the slave instances discovered that new index
are available and claimed that new index installed, and then trying to
update index properties, but looking into the index directory on slaves, you
will find that no index data files were updated or added, plus slaves keep
trying to get new index. Here are some from slave's log file:

Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Starting replication process
Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Number of files in latest snapshot in master: 69
Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Total time taken for download : 0 secs
Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Conf files are not downloaded or are in sync
Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller modifyIndexProps
INFO: New index installed. Updating index properties...
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Master's version: 1246488421310, generation: 9
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Slave's version: 1246385166228, generation: 56
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Starting replication process
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Number of files in latest snapshot in master: 69
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Total time taken for download : 0 secs
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Conf files are not downloaded or are in sync
Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller modifyIndexProps
INFO: New index installed. Updating index properties...


Is this process incorrect, or it is a bug? If the process is incorrect, what
is the right process?

Thanks,

J

Re: reindexed data on master not replicated to slave

[solr jay]

it's nightly build of May 10. I'll try the latest.

Thanks,

J


On Thu, Jul 2, 2009 at 8:09 PM, Otis Gospodnetic  wrote:

>
> Jay,
>
> You didn't mention which version of Solr you are using.  It looks like some
> trunk or nightly version.  Maybe you can try the latest nightly?
>
>  Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message 
> > From: solr jay 
> > To: solr-user@lucene.apache.org
> > Sent: Thursday, July 2, 2009 9:14:48 PM
> > Subject: reindexed data on master not replicated to slave
> >
> > Hi,
> >
> > When index data were corrupted on master instance, I wanted to wipe out
> all
> > the index data and re-index everything. I was hoping the newly created
> index
> > data would be replicated to slaves, but it wasn't.
> >
> > Here are the steps I performed:
> >
> > 1. stop master
> > 2. delete the directory 'index'
> > 3. start master
> > 4. disable replication on master
> > 5. index all data from scratch
> > 6. enable replication on master
> >
> > It seemed from log file that the slave instances discovered that new
> index
> > are available and claimed that new index installed, and then trying to
> > update index properties, but looking into the index directory on slaves,
> you
> > will find that no index data files were updated or added, plus slaves
> keep
> > trying to get new index. Here are some from slave's log file:
> >
> > Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Starting replication process
> > Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Number of files in latest snapshot in master: 69
> > Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Total time taken for download : 0 secs
> > Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Conf files are not downloaded or are in sync
> > Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> modifyIndexProps
> > INFO: New index installed. Updating index properties...
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Master's version: 1246488421310, generation: 9
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Slave's version: 1246385166228, generation: 56
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Starting replication process
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Number of files in latest snapshot in master: 69
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Total time taken for download : 0 secs
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> > INFO: Conf files are not downloaded or are in sync
> > Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> modifyIndexProps
> > INFO: New index installed. Updating index properties...
> >
> >
> > Is this process incorrect, or it is a bug? If the process is incorrect,
> what
> > is the right process?
> >
> > Thanks,
> >
> > J
>
>

Re: reindexed data on master not replicated to slave

[solr jay]

I tried it with the latest nightly build and got the same result.

Actually that was the symptom and it made me looking at the index directory.
The same log messages repeated again and again, never end.



2009/7/2 Noble Paul നോബിള്‍ नोब्ळ् 

> jay , I see updating index properties... twice
>
>
>
> this should happen rarely. in your case it should have happened only
> once. because you cleaned up the master only once
>
>
> On Fri, Jul 3, 2009 at 6:09 AM, Otis
> Gospodnetic wrote:
> >
> > Jay,
> >
> > You didn't mention which version of Solr you are using.  It looks like
> some trunk or nightly version.  Maybe you can try the latest nightly?
> >
> >  Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > - Original Message 
> >> From: solr jay 
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, July 2, 2009 9:14:48 PM
> >> Subject: reindexed data on master not replicated to slave
> >>
> >> Hi,
> >>
> >> When index data were corrupted on master instance, I wanted to wipe out
> all
> >> the index data and re-index everything. I was hoping the newly created
> index
> >> data would be replicated to slaves, but it wasn't.
> >>
> >> Here are the steps I performed:
> >>
> >> 1. stop master
> >> 2. delete the directory 'index'
> >> 3. start master
> >> 4. disable replication on master
> >> 5. index all data from scratch
> >> 6. enable replication on master
> >>
> >> It seemed from log file that the slave instances discovered that new
> index
> >> are available and claimed that new index installed, and then trying to
> >> update index properties, but looking into the index directory on slaves,
> you
> >> will find that no index data files were updated or added, plus slaves
> keep
> >> trying to get new index. Here are some from slave's log file:
> >>
> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Starting replication process
> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Number of files in latest snapshot in master: 69
> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Total time taken for download : 0 secs
> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Conf files are not downloaded or are in sync
> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
> modifyIndexProps
> >> INFO: New index installed. Updating index properties...
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Master's version: 1246488421310, generation: 9
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Slave's version: 1246385166228, generation: 56
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Starting replication process
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Number of files in latest snapshot in master: 69
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Total time taken for download : 0 secs
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> fetchLatestIndex
> >> INFO: Conf files are not downloaded or are in sync
> >> Jul 1, 2009 4:00:33 PM org.apache.solr.handler.SnapPuller
> modifyIndexProps
> >> INFO: New index installed. Updating index properties...
> >>
> >>
> >> Is this process incorrect, or it is a bug? If the process is incorrect,
> what
> >> is the right process?
> >>
> >> Thanks,
> >>
> >> J
> >
> >
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Re: reindexed data on master not replicated to slave

[solr jay]

It looks that the problem is here or before that in
SnapPuller.fetchLatestIndex():


  terminateAndWaitFsyncService();
  LOG.info("Conf files are not downloaded or are in sync");
  if (isSnapNeeded) {
modifyIndexProps(tmpIndexDir.getName());
  } else {
successfulInstall = copyIndexFiles(tmpIndexDir, indexDir);
  }
  if (successfulInstall) {
logReplicationTimeAndConfFiles(modifiedConfFiles);
doCommit();
  }


Debugged into the place, and noticed that isSnapNeeded is true and therefore

modifyIndexProps(tmpIndexDir.getName());

executed, but from the function name it looks that installing index actually
happens in

successfulInstall = copyIndexFiles(tmpIndexDir, indexDir);


The function returns false, but the caller (doSnapPull) never checked the
return value.


Thanks,

J


On Mon, Jul 6, 2009 at 8:02 AM, solr jay  wrote:

> There is only one index directory: index/
>
> Here is the content of index.properties
>
> #index properties
> #Fri Jul 03 14:17:12 PDT 2009
> index=index.20090703021705
>
>
> Thanks,
>
> J
>
> 2009/7/5 Noble Paul നോബിള്‍ नोब्ळ् 
>
> BTW , how many index dirs are there in the data dir ? what is there in
>> the /index.properties ?
>>
>> On Sat, Jul 4, 2009 at 12:15 AM, solr jay wrote:
>> >
>> >
>> > I tried it with the latest nightly build and got the same result.
>> >
>> > Actually that was the symptom and it made me looking at the index
>> directory.
>> > The same log messages repeated again and again, never end.
>> >
>> >
>> >
>> > 2009/7/2 Noble Paul നോബിള്‍ नोब्ळ् 
>> >>
>> >> jay , I see updating index properties... twice
>> >>
>> >>
>> >>
>> >> this should happen rarely. in your case it should have happened only
>> >> once. because you cleaned up the master only once
>> >>
>> >>
>> >> On Fri, Jul 3, 2009 at 6:09 AM, Otis
>> >> Gospodnetic wrote:
>> >> >
>> >> > Jay,
>> >> >
>> >> > You didn't mention which version of Solr you are using.  It looks
>> like
>> >> > some trunk or nightly version.  Maybe you can try the latest nightly?
>> >> >
>> >> >  Otis
>> >> > --
>> >> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >> >
>> >> >
>> >> >
>> >> > - Original Message 
>> >> >> From: solr jay 
>> >> >> To: solr-user@lucene.apache.org
>> >> >> Sent: Thursday, July 2, 2009 9:14:48 PM
>> >> >> Subject: reindexed data on master not replicated to slave
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> When index data were corrupted on master instance, I wanted to wipe
>> out
>> >> >> all
>> >> >> the index data and re-index everything. I was hoping the newly
>> created
>> >> >> index
>> >> >> data would be replicated to slaves, but it wasn't.
>> >> >>
>> >> >> Here are the steps I performed:
>> >> >>
>> >> >> 1. stop master
>> >> >> 2. delete the directory 'index'
>> >> >> 3. start master
>> >> >> 4. disable replication on master
>> >> >> 5. index all data from scratch
>> >> >> 6. enable replication on master
>> >> >>
>> >> >> It seemed from log file that the slave instances discovered that new
>> >> >> index
>> >> >> are available and claimed that new index installed, and then trying
>> to
>> >> >> update index properties, but looking into the index directory on
>> >> >> slaves, you
>> >> >> will find that no index data files were updated or added, plus
>> slaves
>> >> >> keep
>> >> >> trying to get new index. Here are some from slave's log file:
>> >> >>
>> >> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
>> >> >> fetchLatestIndex
>> >> >> INFO: Starting replication process
>> >> >> Jul 1, 2009 3:59:33 PM org.apache.solr.handler.SnapPuller
>> >> >> fetchLatestIndex
>> >> >> INFO: Number of files in latest snapshot in master: 69
>> >> &g

Re: reindexed data on master not replicated to slave

[solr jay]

It seemed that the patch fixed the symptom, but not the problem itself.

Now the log messages looks good. After one download and installed the index,
it printed out

*Jul 7, 2009 10:35:10 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Slave in sync with master.*

but the files inside index directory did not change. Both index.properties
and replication.properties were updated though.


Just a couple of files:

from master instance:

-rw-r--r--  1 worun  wheel 181 Jul  7 09:28 _6.fdt
-rw-r--r--  1 worun  wheel  12 Jul  7 09:28 _6.fdx
-rw-r--r--  1 worun  wheel 131 Jul  7 09:28 _6.fnm
-rw-r--r--  1 worun  wheel  27 Jul  7 09:28 _6.frq
-rw-r--r--  1 worun  wheel  11 Jul  7 09:28 _6.nrm


from slave instance:

-rw-r--r--  1 jianhanguo  admin  70 Jul  6 18:55 _14_5.del
-rw-r--r--  1 jianhanguo  admin4016 Jul  6 18:55 _15.fdt
-rw-r--r--  1 jianhanguo  admin 268 Jul  6 18:55 _15.fdx
-rw-r--r--  1 jianhanguo  admin 131 Jul  6 18:55 _15.fnm
-rw-r--r--  1 jianhanguo  admin 726 Jul  6 18:55 _15.frq


Thanks,

J

2009/7/7 Noble Paul നോബിള്‍ नोब्ळ् 

> Jay ,
> I am opening an issue SOLR-1264
> https://issues.apache.org/jira/browse/SOLR-1264
>
> I have attached a patch as well . I guess that is the fix. could you
> please confirm that.
>
>
> On Tue, Jul 7, 2009 at 12:59 AM, solr jay wrote:
> > It looks that the problem is here or before that in
> > SnapPuller.fetchLatestIndex():
> >
> >
> >   terminateAndWaitFsyncService();
> >   LOG.info("Conf files are not downloaded or are in sync");
> >   if (isSnapNeeded) {
> > modifyIndexProps(tmpIndexDir.getName());
> >   } else {
> > successfulInstall = copyIndexFiles(tmpIndexDir, indexDir);
> >   }
> >   if (successfulInstall) {
> > logReplicationTimeAndConfFiles(modifiedConfFiles);
> > doCommit();
> >   }
> >
> >
> > Debugged into the place, and noticed that isSnapNeeded is true and
> therefore
> >
> > modifyIndexProps(tmpIndexDir.getName());
> >
> > executed, but from the function name it looks that installing index
> actually
> > happens in
> >
> > successfulInstall = copyIndexFiles(tmpIndexDir, indexDir);
> >
> >
> > The function returns false, but the caller (doSnapPull) never checked the
> > return value.
> >
> >
> > Thanks,
> >
> > J
> >
> >
> > On Mon, Jul 6, 2009 at 8:02 AM, solr jay  wrote:
> >>
> >> There is only one index directory: index/
> >>
> >> Here is the content of index.properties
> >>
> >> #index properties
> >> #Fri Jul 03 14:17:12 PDT 2009
> >> index=index.20090703021705
> >>
> >>
> >> Thanks,
> >>
> >> J
> >>
> >> 2009/7/5 Noble Paul നോബിള്‍ नोब्ळ् 
> >>>
> >>> BTW , how many index dirs are there in the data dir ? what is there in
> >>> the /index.properties ?
> >>>
> >>> On Sat, Jul 4, 2009 at 12:15 AM, solr jay wrote:
> >>> >
> >>> >
> >>> > I tried it with the latest nightly build and got the same result.
> >>> >
> >>> > Actually that was the symptom and it made me looking at the index
> >>> > directory.
> >>> > The same log messages repeated again and again, never end.
> >>> >
> >>> >
> >>> >
> >>> > 2009/7/2 Noble Paul നോബിള്‍ नोब्ळ् 
> >>> >>
> >>> >> jay , I see updating index properties... twice
> >>> >>
> >>> >>
> >>> >>
> >>> >> this should happen rarely. in your case it should have happened only
> >>> >> once. because you cleaned up the master only once
> >>> >>
> >>> >>
> >>> >> On Fri, Jul 3, 2009 at 6:09 AM, Otis
> >>> >> Gospodnetic wrote:
> >>> >> >
> >>> >> > Jay,
> >>> >> >
> >>> >> > You didn't mention which version of Solr you are using.  It looks
> >>> >> > like
> >>> >> > some trunk or nightly version.  Maybe you can try the latest
> >>> >> > nightly?
> >>> >> >
> >>> >> >  Otis
> >>> >> > --
> >>> >> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >>> >> >
> >>> >> >
> >&

Re: reindexed data on master not replicated to slave

[solr jay]

I see. So I tried it again. Now index.properties has

#index properties
#Tue Jul 07 12:13:49 PDT 2009
index=index.20090707121349

but there is no such directory index.20090707121349 under the data
directory.

Thanks,

J

On Tue, Jul 7, 2009 at 11:50 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Tue, Jul 7, 2009 at 11:50 PM, solr jay  wrote:
>
> > It seemed that the patch fixed the symptom, but not the problem itself.
> >
> > Now the log messages looks good. After one download and installed the
> > index,
> > it printed out
> >
> > *Jul 7, 2009 10:35:10 AM org.apache.solr.handler.SnapPuller
> > fetchLatestIndex
> > INFO: Slave in sync with master.*
> >
> > but the files inside index directory did not change. Both
> index.properties
> > and replication.properties were updated though.
> >
>
> Note that in this case, Solr would have created a new index directory. Are
> you comparing the files on the slave in the new index directory? You can
> get
> the new index directory's name from index.properties.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: reindexed data on master not replicated to slave

[solr jay]

Ok, Here is the problem. In the function, the two directories tmpIndexDir
and indexDir are the same (in this case only?), and then at the end of the
function, the directory tmpIndexDir is deleted, which deletes the new index
directory.


  } finally {
delTree(tmpIndexDir);
  }


On Tue, Jul 7, 2009 at 12:17 PM, solr jay  wrote:

> I see. So I tried it again. Now index.properties has
>
> #index properties
> #Tue Jul 07 12:13:49 PDT 2009
> index=index.20090707121349
>
> but there is no such directory index.20090707121349 under the data
> directory.
>
> Thanks,
>
> J
>
>
> On Tue, Jul 7, 2009 at 11:50 AM, Shalin Shekhar Mangar <
> shalinman...@gmail.com> wrote:
>
>> On Tue, Jul 7, 2009 at 11:50 PM, solr jay  wrote:
>>
>> > It seemed that the patch fixed the symptom, but not the problem itself.
>> >
>> > Now the log messages looks good. After one download and installed the
>> > index,
>> > it printed out
>> >
>> > *Jul 7, 2009 10:35:10 AM org.apache.solr.handler.SnapPuller
>> > fetchLatestIndex
>> > INFO: Slave in sync with master.*
>> >
>> > but the files inside index directory did not change. Both
>> index.properties
>> > and replication.properties were updated though.
>> >
>>
>> Note that in this case, Solr would have created a new index directory. Are
>> you comparing the files on the slave in the new index directory? You can
>> get
>> the new index directory's name from index.properties.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>

Re: reindexed data on master not replicated to slave

[solr jay]

In fact, I saw the directory was created and then deleted.

On Tue, Jul 7, 2009 at 12:29 PM, solr jay  wrote:

> Ok, Here is the problem. In the function, the two directories tmpIndexDir
> and indexDir are the same (in this case only?), and then at the end of the
> function, the directory tmpIndexDir is deleted, which deletes the new index
> directory.
>
>
>   } finally {
> delTree(tmpIndexDir);
>
>   }
>
>
> On Tue, Jul 7, 2009 at 12:17 PM, solr jay  wrote:
>
>> I see. So I tried it again. Now index.properties has
>>
>> #index properties
>> #Tue Jul 07 12:13:49 PDT 2009
>> index=index.20090707121349
>>
>> but there is no such directory index.20090707121349 under the data
>> directory.
>>
>> Thanks,
>>
>> J
>>
>>
>> On Tue, Jul 7, 2009 at 11:50 AM, Shalin Shekhar Mangar <
>> shalinman...@gmail.com> wrote:
>>
>>> On Tue, Jul 7, 2009 at 11:50 PM, solr jay  wrote:
>>>
>>> > It seemed that the patch fixed the symptom, but not the problem itself.
>>> >
>>> > Now the log messages looks good. After one download and installed the
>>> > index,
>>> > it printed out
>>> >
>>> > *Jul 7, 2009 10:35:10 AM org.apache.solr.handler.SnapPuller
>>> > fetchLatestIndex
>>> > INFO: Slave in sync with master.*
>>> >
>>> > but the files inside index directory did not change. Both
>>> index.properties
>>> > and replication.properties were updated though.
>>> >
>>>
>>> Note that in this case, Solr would have created a new index directory.
>>> Are
>>> you comparing the files on the slave in the new index directory? You can
>>> get
>>> the new index directory's name from index.properties.
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
>

Re: reindexed data on master not replicated to slave

[solr jay]

I guess in this case it doesn't matter whether the two directories
tmpIndexDir and indexDir are the same or not. It looks that the index
directory is switched to tmpIndexDir and then it is deleted inside
"finally".

On Tue, Jul 7, 2009 at 12:31 PM, solr jay  wrote:

> In fact, I saw the directory was created and then deleted.
>
>
> On Tue, Jul 7, 2009 at 12:29 PM, solr jay  wrote:
>
>> Ok, Here is the problem. In the function, the two directories tmpIndexDir
>> and indexDir are the same (in this case only?), and then at the end of the
>> function, the directory tmpIndexDir is deleted, which deletes the new index
>> directory.
>>
>>
>>   } finally {
>> delTree(tmpIndexDir);
>>
>>   }
>>
>>
>> On Tue, Jul 7, 2009 at 12:17 PM, solr jay  wrote:
>>
>>> I see. So I tried it again. Now index.properties has
>>>
>>> #index properties
>>> #Tue Jul 07 12:13:49 PDT 2009
>>> index=index.20090707121349
>>>
>>> but there is no such directory index.20090707121349 under the data
>>> directory.
>>>
>>> Thanks,
>>>
>>> J
>>>
>>>
>>> On Tue, Jul 7, 2009 at 11:50 AM, Shalin Shekhar Mangar <
>>> shalinman...@gmail.com> wrote:
>>>
>>>> On Tue, Jul 7, 2009 at 11:50 PM, solr jay  wrote:
>>>>
>>>> > It seemed that the patch fixed the symptom, but not the problem
>>>> itself.
>>>> >
>>>> > Now the log messages looks good. After one download and installed the
>>>> > index,
>>>> > it printed out
>>>> >
>>>> > *Jul 7, 2009 10:35:10 AM org.apache.solr.handler.SnapPuller
>>>> > fetchLatestIndex
>>>> > INFO: Slave in sync with master.*
>>>> >
>>>> > but the files inside index directory did not change. Both
>>>> index.properties
>>>> > and replication.properties were updated though.
>>>> >
>>>>
>>>> Note that in this case, Solr would have created a new index directory.
>>>> Are
>>>> you comparing the files on the slave in the new index directory? You can
>>>> get
>>>> the new index directory's name from index.properties.
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>
>


-- 
J

index version on slave

[solr jay]

If you ask for the index version of a slave instance, you always get version
number being 0. Is it expected behavior?

I am using this url

http://slave_host:8983/solr/replication?command=indexversion

This request returns correct version on master.

If you use the 'details' command, you get the right version number (and
generation number, and it gives more than what you want).

Thanks,

-- 
J

Re: index version on slave

[solr jay]

oh, in case of index data corrupted on slave, I want to download the entire
index from master. During downloading, I want the slave be out of service
and put it back after it finished. I was trying figure out how to determine
downloading is done. Right now, I am calling

http://slave_host:8983/solr/replication?command=details<http://slave_host:8983/solr/replication?command=indexversion>

and compare the index version on slave and on master, and put the instance
back in service when this two are the same. It works fine except that the
response claims that the structure of the response may change.

Is this the right way to do it?

Thanks,


2009/7/21 Noble Paul നോബിള്‍ नोब्ळ् 

> on the slave this command would not work well. The indexversion is not
> the actual index version. It is the current replicateable index
> version.
>
> why do you call that API directly?
>
>
> On Tue, Jul 21, 2009 at 12:53 AM, solr jay wrote:
> > If you ask for the index version of a slave instance, you always get
> version
> > number being 0. Is it expected behavior?
> >
> > I am using this url
> >
> > http://slave_host:8983/solr/replication?command=indexversion
> >
> > This request returns correct version on master.
> >
> > If you use the 'details' command, you get the right version number (and
> > generation number, and it gives more than what you want).
> >
> > Thanks,
> >
> > --
> > J
> >
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>



-- 
J