Re: Choosing what document to return.
Hi Mei Thank you very much - looks good :-) Noam. -- View this message in context: http://old.nabble.com/Choosing-what-document-to-return.-tp27259310p27293083.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Replication Handler Severe Error: Unable to move index file
On Fri, Jan 22, 2010 at 4:24 AM, Trey wrote:
> Unfortunately, when I went back to look at the logs this morning, the log
> file had been blown away... that puts a major damper on my debugging
> capabilities - so sorry about that. As a double whammy, we optimize
> nightly, so the old index files have completely changed at this point.
>
> I do not remember seeing an exception / stack trace in the logs associated
> with the "SEVERE *Unable to move file*" entry, but we were grepping the
> logs, so if it was outputted onto another line it could have possibly been
> there. I wouldn't really expect to see anything based upon the code in
> SnapPuller.java:
>
> /**
> * Copy a file by the File#renameTo() method. If it fails, it is
> considered a failure
> *
> * Todo may be we should try a simple copy if it fails
> */
> private boolean copyAFile(File tmpIdxDir, File indexDir, String fname,
> List copiedfiles) {
> File indexFileInTmpDir = new File(tmpIdxDir, fname);
> File indexFileInIndex = new File(indexDir, fname);
> boolean success = indexFileInTmpDir.renameTo(indexFileInIndex);
> if (!success) {
> LOG.error("Unable to move index file from: " + indexFileInTmpDir
> + " to: " + indexFileInIndex);
> for (String f : copiedfiles) {
> File indexFile = new File(indexDir, f);
> if (indexFile.exists())
> indexFile.delete();
> }
> delTree(tmpIdxDir);
> return false;
> }
> return true;
> }
>
> In terms of whether this is an off case: this is the first occurrence of
> this I have seen in the logs. We tried to replicate the conditions under
> which the exception occurred, but were unable. I'll send along some more
> useful info if this happens again.
>
> In terms of the behavior we saw: It appears that a replication occurred and
> the "Unable to move file" error occurred. As a result, it looks like the
> ENTIRE index was subsequently replicated again into a temporary directory
> (several times, over and over).
>
> The end result was that we had multiple full copies of the index in
> temporary index folders on the slave, and the original still couldn't be
> updated (the move to ./index wouldn't work). Does Solr ever hold files open
> in a manner that would prevent a file in the index directory from being
> overridden?
There is a TODO which says manual it try to copy if move (renameTo)
fails. We never did it because we never observed renameTo failing.
>
>
> 2010/1/21 Noble Paul നോബിള് नोब्ळ्
>
>> is it a one off case? do you observerve this frequently?
>>
>> On Thu, Jan 21, 2010 at 11:26 AM, Otis Gospodnetic
>> wrote:
>> > It's hard to tell without poking around, but one of the first things I'd
>> do would be to look for /home/solr/cores/core8/index.20100119103919/_6qv.fnm
>> - does this file/dir really exist? Or, rather, did it exist when the error
>> happened.
>> >
>> > I'm not looking at the source code now, but is that really the only error
>> you got? No exception stack trace?
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>> >
>> >
>> >
>> > - Original Message
>> >> From: Trey
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Wed, January 20, 2010 11:54:43 PM
>> >> Subject: Replication Handler Severe Error: Unable to move index file
>> >>
>> >> Does anyone know what would cause the following error?:
>> >>
>> >> 10:45:10 AM org.apache.solr.handler.SnapPuller copyAFile
>> >>
>> >> SEVERE: *Unable to move index file* from:
>> >> /home/solr/cores/core8/index.20100119103919/_6qv.fnm to:
>> >> /home/solr/cores/core8/index/_6qv.fnm
>> >> This occurred a few days back and we noticed that several full copies of
>> the
>> >> index were subsequently pulled from the master to the slave, effectively
>> >> evicting our live index from RAM (the linux os cache), and killing our
>> query
>> >> performance due to disk io contention.
>> >>
>> >> Has anyone experienced this behavior recently? I found an old thread
>> about
>> >> this error from early 2009, but it looks like it was patched almost a
>> year
>> >> ago:
>> >>
>> http://old.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-td21157722.html
>> >>
>> >>
>> >> Additional Relevant information:
>> >> -We are using the Solr 1.4 official release + a field collapsing patch
>> from
>> >> mid December (which I believe should only affect query side, not
>> indexing /
>> >> replication).
>> >> -Our Replication PollInterval for slaves checking the master is very
>> small
>> >> (15 seconds)
>> >> -We have a multi-box distributed search with each box possessing
>> multiple
>> >> cores
>> >> -We issue a manual (rolling) optimize across the cores on the master
>> once a
>> >> day (occurred ~ 1-2 hours before the above timeline)
>> >> -maxWarmingSearchers is set to 1.
>> >
>> >
>>
>>
>>
>> --
>> -
>> Noble Paul | Systems Architect| AOL | http://aol.com
>>
>
--
--
after migration to solr 1.4, queries always return 0 items
We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 with no luck. The new 1.4 Solr server is up and running but always returns 0 results when you query it. It seems to have the correct schema but the number of documents is 0 even though I used a backup of the production data. I created the 1.4 solr server using a few machine and following the instructions on the solr wiki. For the data, I used the 'backup' command to make a copy of the live index and moved to the data directory on the new 1.4 server. When I start the new server, the tomcat catinla logs don't have any errors (only info msgs). The logs show the correct solr home and I can see its reading the schema.xml (acts_as_solr). Finally, I see that the data directory where the index is has new subfolders for spellchecker & spellchecker2 so I assume it is looking the correct directory. I'd appreciate any tips on how to debug this. Below are some more details on the setup.. thanks in advance. Athir
Re: after migration to solr 1.4, queries always return 0 items
You omitted the details. Is there an "index" directory under your data directory? That's the heart of the Lucene index there. Look at Solr's admin, it'll tell you the data directory location it is using, be sure it is set where you think it should be. Erik On Jan 24, 2010, at 8:59 AM, athir nuaimi wrote: We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 with no luck. The new 1.4 Solr server is up and running but always returns 0 results when you query it. It seems to have the correct schema but the number of documents is 0 even though I used a backup of the production data. I created the 1.4 solr server using a few machine and following the instructions on the solr wiki. For the data, I used the 'backup' command to make a copy of the live index and moved to the data directory on the new 1.4 server. When I start the new server, the tomcat catinla logs don't have any errors (only info msgs). The logs show the correct solr home and I can see its reading the schema.xml (acts_as_solr). Finally, I see that the data directory where the index is has new subfolders for spellchecker & spellchecker2 so I assume it is looking the correct directory. I'd appreciate any tips on how to debug this. Below are some more details on the setup.. thanks in advance. Athir
Re: after migration to solr 1.4, queries always return 0 items
Hi Eric. Thanks for the quick reply. I though the backup went in /data. I have restored the backup to /data/index and am now getting results. On 2010-01-24, at 9:46 AM, Erik Hatcher wrote: > You omitted the details. > > Is there an "index" directory under your data directory? That's the heart of > the Lucene index there. Look at Solr's admin, it'll tell you the data > directory location it is using, be sure it is set where you think it should > be. > > Erik > > On Jan 24, 2010, at 8:59 AM, athir nuaimi wrote: > >> We have a working system based on Solr 1.3 that I tried to upgrade to 1.4 >> with no luck. The new 1.4 Solr server is up and running but always returns >> 0 results when you query it. It seems to have the correct schema but the >> number of documents is 0 even though I used a backup of the production data. >> >> I created the 1.4 solr server using a few machine and following the >> instructions on the solr wiki. For the data, I used the 'backup' command >> to make a copy of the live index and moved to the data directory on the new >> 1.4 server. When I start the new server, the tomcat catinla logs don't >> have any errors (only info msgs). The logs show the correct solr home and I >> can see its reading the schema.xml (acts_as_solr). Finally, I see that the >> data directory where the index is has new subfolders for spellchecker & >> spellchecker2 so I assume it is looking the correct directory. >> >> I'd appreciate any tips on how to debug this. Below are some more details >> on the setup.. >> >> thanks in advance. >> Athir >> >
strategy for snapshottig Solr data directory on EC2
We are running two Solr servers (master/slave) on EC2 and have the solr home directories on EBS drives that we snapshot every 12 hours. While that will mean that we will lose at most 12 hours of data, I wondered if there was a way I could reduce the window of data loss. With our mysql servers, we snapshot every 12 hours but also copy the binary logs to S3 every 5 minutes. We are doing commits every 10 minutes on the master and will be using the built-in java replication (today we are using snapshotting to replicate but are in the process of migrating from 1.3 to 1.4). On a related note, are we doing the right thing in having our slave solr home directory on an EBS volume? If the slave were to die and we had to create a fresh one, will it just resync the entire index from the master? is the reason to have the slave on an EBS volume so that the slave has less data to resync on startup? thanks in advance Athir
Re: strategy for snapshottig Solr data directory on EC2
Our setup on ec2 is as follows: a) mysql master on ebs volume. b) solr master on its own ebs volume c) solr slaves do not use ebs -- but rather use the ephemeral instance stores. There is a small period of time where the solr slave has to re-sync the data from the solr master. Cheers, Bill -- From: "athir nuaimi" Sent: Sunday, January 24, 2010 7:35 AM To: Subject: strategy for snapshottig Solr data directory on EC2 We are running two Solr servers (master/slave) on EC2 and have the solr home directories on EBS drives that we snapshot every 12 hours. While that will mean that we will lose at most 12 hours of data, I wondered if there was a way I could reduce the window of data loss. With our mysql servers, we snapshot every 12 hours but also copy the binary logs to S3 every 5 minutes. We are doing commits every 10 minutes on the master and will be using the built-in java replication (today we are using snapshotting to replicate but are in the process of migrating from 1.3 to 1.4). On a related note, are we doing the right thing in having our slave solr home directory on an EBS volume? If the slave were to die and we had to create a fresh one, will it just resync the entire index from the master? is the reason to have the slave on an EBS volume so that the slave has less data to resync on startup? thanks in advance Athir
Default value attribute in RSS DIH
Hey All, Can anyone tell me what the attribute name is for defining a default value in the field tag of the RSS data import handler?? Basically I want to do something like Any Ideas? Regards, Dave
Retrieve Field Names:
How to retrieve field names from the dynamic schema in the client side program ? Any links or reference would be helpful. (Similar to the schema browser on the SOLR admin screen.) Kumar_/|\_ www.saisk.com ku...@saisk.com "making a profound difference with knowledge and creativity..."
Re: Retrieve Field Names:
The schema browser is actually an Ajax client to Solr itself, using the luke request handler to retrieve the field info. Take a look at your Solr logs as you peruse the schema browse and you'll see the requests being made. Just do that :) Erik On Jan 24, 2010, at 9:44 PM, Kumaravel Kandasami wrote: How to retrieve field names from the dynamic schema in the client side program ? Any links or reference would be helpful. (Similar to the schema browser on the SOLR admin screen.) Kumar_/|\_ www.saisk.com ku...@saisk.com "making a profound difference with knowledge and creativity..."
using bq with standard request handler
Hi all, I am using a query like: http://localhost:8080/solr/select?q=product_category:Grocery&fq=in_stock:true&debugQuery=true&; sort=display_priority+desc,prop_l_price+asc To first sort on display_priority and then on price.. Is it possible to use display_priority/price fields in bq itself to acheive the same result.. I tried forming some queries that but was unable to get desired results... Any kind of help would be highly appreciated... Regards, Gunjan -- View this message in context: http://old.nabble.com/using-bq-with-standard-request-handler-tp27302951p27302951.html Sent from the Solr - User mailing list archive at Nabble.com.
