Fwd: Searching within list of regions with 1:1 document-region mapping
Hi, I have a Solr index of around 100 million documents with each document being given a region id growing at a rate of about 10 million documents per month - the average document size being aronud 10KB of pure text. The total number of region ids are themselves in the range of 2.5 million. I want to search for a query with a given list of region ids. The number of region ids in this list is usually around 250-300 (most of the time), but can be upto 500, with a maximum cap of around 2000 ids in one request. What is the best way to model such queries besides using an IN param in the query, or using a Filter FQ in the query or some other means? If it may help, the index is on a VM with 4 virtual-cores and has currently 4GB of Java memory allocated out of 16GB in the machine. The number of queries do not exceed more than 1 per minute for now. If needed, we can throw more hardware to the index - but the index will still be only on a single machine for atleast 6 months. Best Regards, Sandeep Gupta
Re: Class name of parsing the fq clause
Thanks Jack for detailing out the parser logic.
Would it be possible for you to say something more about filter cache code
flow... sometimes we do not use fq parameter in query string and pass the
raw query
Regards
Sandeep
On Mon, Oct 21, 2013 at 7:11 PM, Jack Krupansky wrote:
> Start with org.apache.solr.handler.**component.QueryComponent#**prepare
> which fetches the fq parameters and indirectly invokes the query parser(s):
>
> String[] fqs = req.getParams().getParams(**CommonParams.FQ);
> if (fqs!=null && fqs.length!=0) {
> List filters = rb.getFilters();
> // if filters already exists, make a copy instead of modifying the
> original
> filters = filters == null ? new ArrayList(fqs.length) : new
> ArrayList(filters);
> for (String fq : fqs) {
> if (fq != null && fq.trim().length()!=0) {
> QParser fqp = QParser.getParser(fq, null, req);
> filters.add(fqp.getQuery());
> }
> }
> // only set the filters if they are not empty otherwise
> // fq=&someotherParam= will trigger all docs filter for every request
> // if filter cache is disabled
> if (!filters.isEmpty()) {
> rb.setFilters( filters );
>
> Note that this line actually invokes the parser:
>
> filters.add(fqp.getQuery());
>
> Then in org.apache.lucene.search.**Query.QParser#getParser:
>
> QParserPlugin qplug = req.getCore().getQueryPlugin(**parserName);
> QParser parser = qplug.createParser(qstr, localParams, req.getParams(),
> req);
>
> And for the common case of the Lucene query parser, org.apache.solr.search.
> **LuceneQParserPlugin#**createParser:
>
> public QParser createParser(String qstr, SolrParams localParams,
> SolrParams params, SolrQueryRequest req) {
> return new LuceneQParser(qstr, localParams, params, req);
> }
>
> And then in org.apache.lucene.search.**Query.QParser#getQuery:
>
> public Query getQuery() throws SyntaxError {
> if (query==null) {
>query=parse();
>
> And then in org.apache.lucene.search.**Query.LuceneQParser#parse:
>
> lparser = new SolrQueryParser(this, defaultField);
>
> lparser.setDefaultOperator
> (QueryParsing.**getQueryParserDefaultOperator(**getReq().getSchema(),
> getParam(QueryParsing.OP)));
>
> return lparser.parse(qstr);
>
> And then in org.apache.solr.parser.**SolrQueryParserBase#parse:
>
> Query res = TopLevelQuery(null); // pass null so we can tell later if an
> explicit field was provided or not
>
> And then in org.apache.solr.parser.**QueryParser#TopLevelQuery, the
> parsing begins.
>
> And org.apache.solr.parser.**QueryParser.jj is the grammar for a basic
> Solr/Lucene query, and org.apache.solr.parser.**QueryParser.java is
> generated by JFlex, and a lot of the logic is in the base class of the
> generated class, org.apache.solr.parser.**SolrQueryParserBase.java.
>
> Good luck! Happy hunting!
>
> -- Jack Krupansky
>
> -Original Message- From: YouPeng Yang
> Sent: Monday, October 21, 2013 2:57 AM
> To: solr-user@lucene.apache.org
> Subject: Class name of parsing the fq clause
>
>
> Hi
> I search the solr with fq clause,which is like:
> fq=BEGINTIME:[2013-08-25T16:**00:00Z TO *] AND BUSID:(M3 OR M9)
>
>
> I am curious about the parsing process . I want to study it.
> What is the Java file name describes the parsing process of the fq
> clause.
>
>
> Thanks
>
> Regards.
>
Re: Class name of parsing the fq clause
Yes.. it is not related to this particular mail thread. I will post
separate mail.
Thanks
Sandeep
On Wed, Oct 23, 2013 at 4:36 PM, Jack Krupansky wrote:
> Not in just a few words. Do you have specific questions? I mean none of
> that relates to parsing of fq, the topic of this particular email thread,
> right?
>
> -- Jack Krupansky
>
> -Original Message- From: Sandeep Gupta
> Sent: Wednesday, October 23, 2013 3:58 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Class name of parsing the fq clause
>
>
> Thanks Jack for detailing out the parser logic.
> Would it be possible for you to say something more about filter cache code
> flow... sometimes we do not use fq parameter in query string and pass the
> raw query
>
> Regards
> Sandeep
>
>
> On Mon, Oct 21, 2013 at 7:11 PM, Jack Krupansky *
> *wrote:
>
> Start with org.apache.solr.handler.component.QueryComponent#
>> prepare
>>
>> which fetches the fq parameters and indirectly invokes the query
>> parser(s):
>>
>> String[] fqs = req.getParams().getParams(CommonParams.FQ);
>>
>> if (fqs!=null && fqs.length!=0) {
>> List filters = rb.getFilters();
>> // if filters already exists, make a copy instead of modifying the
>> original
>> filters = filters == null ? new ArrayList(fqs.length) : new
>> ArrayList(filters);
>> for (String fq : fqs) {
>> if (fq != null && fq.trim().length()!=0) {
>> QParser fqp = QParser.getParser(fq, null, req);
>> filters.add(fqp.getQuery());
>> }
>> }
>> // only set the filters if they are not empty otherwise
>> // fq=&someotherParam= will trigger all docs filter for every request
>> // if filter cache is disabled
>> if (!filters.isEmpty()) {
>> rb.setFilters( filters );
>>
>> Note that this line actually invokes the parser:
>>
>> filters.add(fqp.getQuery());
>>
>> Then in org.apache.lucene.search.Query.QParser#getParser:
>>
>> QParserPlugin qplug = req.getCore().getQueryPlugin(parserName);
>> QParser parser = qplug.createParser(qstr, localParams, req.getParams(),
>> req);
>>
>> And for the common case of the Lucene query parser,
>> org.apache.solr.search.
>> **LuceneQParserPlugin#createParser:
>>
>>
>> public QParser createParser(String qstr, SolrParams localParams,
>> SolrParams params, SolrQueryRequest req) {
>> return new LuceneQParser(qstr, localParams, params, req);
>> }
>>
>> And then in org.apache.lucene.search.Query.QParser#getQuery:
>>
>>
>> public Query getQuery() throws SyntaxError {
>> if (query==null) {
>>query=parse();
>>
>> And then in org.apache.lucene.search.Query.LuceneQParser#parse:
>>
>>
>> lparser = new SolrQueryParser(this, defaultField);
>>
>> lparser.setDefaultOperator
>> (QueryParsing.getQueryParserDefaultOperator(**
>> **getReq().getSchema(),
>> getParam(QueryParsing.OP)));
>>
>> return lparser.parse(qstr);
>>
>> And then in org.apache.solr.parser.SolrQueryParserBase#parse:
>>
>>
>> Query res = TopLevelQuery(null); // pass null so we can tell later if an
>> explicit field was provided or not
>>
>> And then in org.apache.solr.parser.QueryParser#TopLevelQuery, the
>> parsing begins.
>>
>> And org.apache.solr.parser.QueryParser.jj is the grammar for a basic
>> Solr/Lucene query, and org.apache.solr.parser.QueryParser.java is
>>
>> generated by JFlex, and a lot of the logic is in the base class of the
>> generated class, org.apache.solr.parser.SolrQueryParserBase.java.
>>
>>
>> Good luck! Happy hunting!
>>
>> -- Jack Krupansky
>>
>> -Original Message- From: YouPeng Yang
>> Sent: Monday, October 21, 2013 2:57 AM
>> To: solr-user@lucene.apache.org
>> Subject: Class name of parsing the fq clause
>>
>>
>> Hi
>> I search the solr with fq clause,which is like:
>> fq=BEGINTIME:[2013-08-25T16:00:00Z TO *] AND BUSID:(M3 OR
>> M9)
>>
>>
>>
>> I am curious about the parsing process . I want to study it.
>> What is the Java file name describes the parsing process of the fq
>> clause.
>>
>>
>> Thanks
>>
>> Regards.
>>
>>
>
Solr subset searching in 100-million document index
Hi, We have a Solr index of around 100 million documents with each document being given a region id growing at a rate of about 10 million documents per month - the average document size being aronud 10KB of pure text. The total number of region ids are themselves in the range of 2.5 million. We want to search for a query with a given list of region ids. The number of region ids in this list is usually around 250-300 (most of the time), but can be upto 500, with a maximum cap of around 2000 ids in one request. What is the best way to model such queries besides using an IN param in the query, or using a Filter FQ in the query? Are there any other faster methods available? If it may help, the index is on a VM with 4 virtual-cores and has currently 4GB of Java memory allocated out of 16GB in the machine. The number of queries do not exceed more than 1 per minute for now. If needed, we can throw more hardware to the index - but the index will still be only on a single machine for atleast 6 months. Regards, Sandeep Gupta
Re: Solr subset searching in 100-million document index
Hi Joel, Thanks a lot for the information - I haven't worked with PostFilter's before but found an example at http://java.dzone.com/articles/custom-security-filtering-solr. Will try it over the next few days and come back if still have questions. Thanks again! Keep Walking, ~ Sandeep On Thu, Oct 24, 2013 at 8:25 PM, Joel Bernstein wrote: > Sandeep, > > This type of operation can often be expressed as a PostFilter very > efficiently. This is particularly true if the region id's are integer keys. > > Joel > > On Thu, Oct 24, 2013 at 7:46 AM, Sandeep Gupta > wrote: > > > Hi, > > > > We have a Solr index of around 100 million documents with each document > > being given a region id growing at a rate of about 10 million documents > per > > month - the average document size being aronud 10KB of pure text. The > total > > number of region ids are themselves in the range of 2.5 million. > > > > We want to search for a query with a given list of region ids. The number > > of region ids in this list is usually around 250-300 (most of the time), > > but can be upto 500, with a maximum cap of around 2000 ids in one > request. > > > > > > What is the best way to model such queries besides using an IN param in > the > > query, or using a Filter FQ in the query? Are there any other faster > > methods available? > > > > > > If it may help, the index is on a VM with 4 virtual-cores and has > currently > > 4GB of Java memory allocated out of 16GB in the machine. The number of > > queries do not exceed more than 1 per minute for now. If needed, we can > > throw more hardware to the index - but the index will still be only on a > > single machine for atleast 6 months. > > > > Regards, > > Sandeep Gupta > > > > > > -- >
Re: Need Help in migrating Solr version 1.4 to 4.3
Thanks for all the answers. Sure I am going to create new index again with Solr 4.3. Also in application development side, as I said that I am going to use HTTPSolrServer API and I found that we shouldn't create this object multiple times (as per the wiki document http://wiki.apache.org/solr/Solrj#HttpSolrServer) So I am planning to have my Server class as singleton. Please advice little bit in this front also. Regards Sandeep On Tue, Jun 25, 2013 at 11:16 PM, André Widhani wrote: > fwiw, I can confirm that Solr 4.x can definitely not read indexes created > with 1.4. > > You'll get an exception like the following: > > Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format > version is not supported (resource: segment _16ofy in resource > ChecksumIndexInput(MMapIndexInput(path="/var/opt/dcx/solr2/core-tex60l254lpachcjhtz4se-index2/data/index/segments_1dlof"))): > 2.x. This version of Lucene only supports indexes created with release 3.0 > and later. > > But as Erick mentioned, you could get away with optimizing the index with > 3.x instead of re-indexing from scratch before moving on to 4.x - I think I > did that once and it worked. > > Regards, > André > > > Von: Erick Erickson [erickerick...@gmail.com] > Gesendet: Dienstag, 25. Juni 2013 19:37 > An: solr-user@lucene.apache.org > Betreff: Re: Need Help in migrating Solr version 1.4 to 4.3 > > bq: I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes > > Solr/Lucene explicitly try to read _one_ major revision backwards. > Solr 3.x should be able to read 1.4 indexes. Solr 4.x should be > able to read Solr 3.x. No attempt is made to allow Solr 4.x to read > Solr 1.4 indexes, so I wouldn't even try. > > Shalin's comment is best. If at all possible I'd just forget about > reading the old index and re-index from scratch. But if you _do_ > try upgrading 1.4 -> 3.x -> 4.x, you probably want to optimize > at each step. That'll (I think) rewrite all the segments in the > current format. > > Good luck! > Erick > > On Tue, Jun 25, 2013 at 12:59 AM, Shalin Shekhar Mangar > wrote: > > You must carefully go through the upgrade instructions starting from > > 1.4 upto 4.3. In particular the instructions for 1.4 to 3.1 and from > > 3.1 to 4.0 should be given special attention. > > > > On Tue, Jun 25, 2013 at 11:43 AM, Sandeep Gupta > wrote: > >> Hello All, > >> > >> We are planning to migrate solr 1.4 to Solr 4.3 version. > >> And I am seeking some help in this side. > >> > >> Considering Schema file change: > >> By default there are lots of changes if I compare original Solr 1.4 > schema > >> file to Sol 4.3 schema file. > >> And that is the reason we are not copying paste of schema file. > >> In our Solr 1.4 schema implementation, we have some custom fields with > type > >> "textgen" and "text" > >> So in migration of these custom fields to Solr 4.3, should I use type > of > >> "text_general" as replacement of "textgen" and > >> "text_en" as replacement of "text"? > >> Please confirm the same. > > > > Please check the text_general definition in 4.3 against the textgen > > fieldtype in Solr 1.4 to see if they're equivalent. Same for text_en > > and text. > > > >> > >> Considering Solrconfig change: > >> As we didn't have lots of changes in 1.4 solrconfig file except the > >> dataimport request handler. > >> And therefore in migration side, we are simply modifying the Solr 4.3 > >> solrconfig file with his request handler. > > > > And you need to add the dataimporthandler jar into Solr's lib > > directory. DIH is not added automatically anymore. > > > >> > >> Considering the application development: > >> > >> We used all the queries as BOOLEAN type style (was not good) I mean put > >> all the parameter in query fields i.e > >> *:* AND EntityName: <<>> AND : AND . > >> > >> I think we should simplify our queries using other fields like df, qf > > >> > > > > Probably. AND queries are best done by filter queries (fq). > > > >> We also used to create Solr server object via CommonsHttpSolrServer() > so I > >> am planning to use now HttpSolrServer API> > > > > Yes. Also, there was a compatibility break between Solr 1.4 and 3.1 in > > the javabin format so old clients using javabin won't be able to > > communicate with Solr until you upgrade both solr client and solr > > servers. > > > >> > >> Please let me know the suggestion for above points also what are the > other > >> factors I need to take care while considering the migration. > > > > There is no substitute for reading the upgrade sections in the > changes.txt. > > > > I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes. You > > will most likely need to re-index your documents. > > > > You should also think about switching to SolrCloud to take advantage > > of its features. > > > > -- > > Regards, > > Shalin Shekhar Mangar. >
Re: Need Help in migrating Solr version 1.4 to 4.3
Thanks for all the answers. Sure I am going to create new index again with Solr 4.3. Also in application development side, as I said that I am going to use HTTPSolrServer API and I found that we shouldn't create this object multiple times (as per the wiki document http://wiki.apache.org/solr/Solrj#HttpSolrServer) So I am planning to have my Server class as singleton. Please advice little bit in this front also. Regards Sandeep On Tue, Jun 25, 2013 at 11:16 PM, André Widhani wrote: > fwiw, I can confirm that Solr 4.x can definitely not read indexes created > with 1.4. > > You'll get an exception like the following: > > Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format > version is not supported (resource: segment _16ofy in resource > ChecksumIndexInput(MMapIndexInput(path="/var/opt/dcx/solr2/core-tex60l254lpachcjhtz4se-index2/data/index/segments_1dlof"))): > 2.x. This version of Lucene only supports indexes created with release 3.0 > and later. > > But as Erick mentioned, you could get away with optimizing the index with > 3.x instead of re-indexing from scratch before moving on to 4.x - I think I > did that once and it worked. > > Regards, > André > > > Von: Erick Erickson [erickerick...@gmail.com] > Gesendet: Dienstag, 25. Juni 2013 19:37 > An: solr-user@lucene.apache.org > Betreff: Re: Need Help in migrating Solr version 1.4 to 4.3 > > bq: I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes > > Solr/Lucene explicitly try to read _one_ major revision backwards. > Solr 3.x should be able to read 1.4 indexes. Solr 4.x should be > able to read Solr 3.x. No attempt is made to allow Solr 4.x to read > Solr 1.4 indexes, so I wouldn't even try. > > Shalin's comment is best. If at all possible I'd just forget about > reading the old index and re-index from scratch. But if you _do_ > try upgrading 1.4 -> 3.x -> 4.x, you probably want to optimize > at each step. That'll (I think) rewrite all the segments in the > current format. > > Good luck! > Erick > > On Tue, Jun 25, 2013 at 12:59 AM, Shalin Shekhar Mangar > wrote: > > You must carefully go through the upgrade instructions starting from > > 1.4 upto 4.3. In particular the instructions for 1.4 to 3.1 and from > > 3.1 to 4.0 should be given special attention. > > > > On Tue, Jun 25, 2013 at 11:43 AM, Sandeep Gupta > wrote: > >> Hello All, > >> > >> We are planning to migrate solr 1.4 to Solr 4.3 version. > >> And I am seeking some help in this side. > >> > >> Considering Schema file change: > >> By default there are lots of changes if I compare original Solr 1.4 > schema > >> file to Sol 4.3 schema file. > >> And that is the reason we are not copying paste of schema file. > >> In our Solr 1.4 schema implementation, we have some custom fields with > type > >> "textgen" and "text" > >> So in migration of these custom fields to Solr 4.3, should I use type > of > >> "text_general" as replacement of "textgen" and > >> "text_en" as replacement of "text"? > >> Please confirm the same. > > > > Please check the text_general definition in 4.3 against the textgen > > fieldtype in Solr 1.4 to see if they're equivalent. Same for text_en > > and text. > > > >> > >> Considering Solrconfig change: > >> As we didn't have lots of changes in 1.4 solrconfig file except the > >> dataimport request handler. > >> And therefore in migration side, we are simply modifying the Solr 4.3 > >> solrconfig file with his request handler. > > > > And you need to add the dataimporthandler jar into Solr's lib > > directory. DIH is not added automatically anymore. > > > >> > >> Considering the application development: > >> > >> We used all the queries as BOOLEAN type style (was not good) I mean put > >> all the parameter in query fields i.e > >> *:* AND EntityName: <<>> AND : AND . > >> > >> I think we should simplify our queries using other fields like df, qf > > >> > > > > Probably. AND queries are best done by filter queries (fq). > > > >> We also used to create Solr server object via CommonsHttpSolrServer() > so I > >> am planning to use now HttpSolrServer API> > > > > Yes. Also, there was a compatibility break between Solr 1.4 and 3.1 in > > the javabin format so old clients using javabin won't be able to > > communicate with Solr until you upgrade both solr client and solr > > servers. > > > >> > >> Please let me know the suggestion for above points also what are the > other > >> factors I need to take care while considering the migration. > > > > There is no substitute for reading the upgrade sections in the > changes.txt. > > > > I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes. You > > will most likely need to re-index your documents. > > > > You should also think about switching to SolrCloud to take advantage > > of its features. > > > > -- > > Regards, > > Shalin Shekhar Mangar. >
Re: Need Help in migrating Solr version 1.4 to 4.3
Thanks Shawn. To have singleton design pattern for SolrServer object creation, I found that there are so many ways described in http://en.wikipedia.org/wiki/Singleton_pattern So which is the best one, out of 5 examples mentioned in above url, for web application in general practice. I am sure lots of people (in this mailing list) will have practical experience as which type of singleton pattern need to be implement for creation of SolrServer object. Waiting for some comments in this front ? Regards Sandeep On Wed, Jun 26, 2013 at 9:20 PM, Shawn Heisey wrote: > On 6/25/2013 11:52 PM, Sandeep Gupta wrote: > > Also in application development side, > > as I said that I am going to use HTTPSolrServer API and I found that we > > shouldn't create this object multiple times > > (as per the wiki document > http://wiki.apache.org/solr/Solrj#HttpSolrServer) > > So I am planning to have my Server class as singleton. > > Please advice little bit in this front also. > > This is always the way that SolrServer objects are intended to be used, > including CommonsHttpSolrServer in version 1.4. The only major > difference between the two objects is that the new one uses > HttpComponents 4.x and the old one uses HttpClient 3.x. There are other > differences, but they are just the result of incremental improvements > from version to version. > > Thanks, > Shawn > >
Re: Need Help in migrating Solr version 1.4 to 4.3
Thanks again Shawn for your comments. I am little worried about the multi threading of web application which uses servlets. I also found one of your explanation (please confirm the same whether its your comment only) in http://lucene.472066.n3.nabble.com/Memory-problems-with-HttpSolrServer-td4060985.html for the question : http://stackoverflow.com/questions/11931179/httpsolrserver-instance-management As you said correctly that creation of SolrServer object depends on number of shards/solrcores and thereafter need to think for implementation which may use singleton pattern. In my web application side, I have only one solrcore which is default one "collection1" so I will create one SolrServer object for my application. Sure If we decide to go for Solr Cloud then also I will create one object. Thanks Upayavira, yes I will do the re-index. Anything you want to suggest as you did the same migration. Thanks Sandeep On Thu, Jun 27, 2013 at 1:33 PM, Upayavira wrote: > I have done this - upgraded a 1.4 index to 3.x then on to 4.x. It > worked, but... > > New field types have been introduced over time that facilitate new > functionality. To continue to use an upgraded index, you need to > continue using the old field types, and thus loose some of the coolness > of newer versions. > > So, a re-index will set you in far better stead, if it is at all > possible. > > Upayavira > > On Tue, Jun 25, 2013, at 06:37 PM, Erick Erickson wrote: > > bq: I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes > > > > Solr/Lucene explicitly try to read _one_ major revision backwards. > > Solr 3.x should be able to read 1.4 indexes. Solr 4.x should be > > able to read Solr 3.x. No attempt is made to allow Solr 4.x to read > > Solr 1.4 indexes, so I wouldn't even try. > > > > Shalin's comment is best. If at all possible I'd just forget about > > reading the old index and re-index from scratch. But if you _do_ > > try upgrading 1.4 -> 3.x -> 4.x, you probably want to optimize > > at each step. That'll (I think) rewrite all the segments in the > > current format. > > > > Good luck! > > Erick > > > > On Tue, Jun 25, 2013 at 12:59 AM, Shalin Shekhar Mangar > > wrote: > > > You must carefully go through the upgrade instructions starting from > > > 1.4 upto 4.3. In particular the instructions for 1.4 to 3.1 and from > > > 3.1 to 4.0 should be given special attention. > > > > > > On Tue, Jun 25, 2013 at 11:43 AM, Sandeep Gupta > wrote: > > >> Hello All, > > >> > > >> We are planning to migrate solr 1.4 to Solr 4.3 version. > > >> And I am seeking some help in this side. > > >> > > >> Considering Schema file change: > > >> By default there are lots of changes if I compare original Solr 1.4 > schema > > >> file to Sol 4.3 schema file. > > >> And that is the reason we are not copying paste of schema file. > > >> In our Solr 1.4 schema implementation, we have some custom fields > with type > > >> "textgen" and "text" > > >> So in migration of these custom fields to Solr 4.3, should I use > type of > > >> "text_general" as replacement of "textgen" and > > >> "text_en" as replacement of "text"? > > >> Please confirm the same. > > > > > > Please check the text_general definition in 4.3 against the textgen > > > fieldtype in Solr 1.4 to see if they're equivalent. Same for text_en > > > and text. > > > > > >> > > >> Considering Solrconfig change: > > >> As we didn't have lots of changes in 1.4 solrconfig file except the > > >> dataimport request handler. > > >> And therefore in migration side, we are simply modifying the Solr 4.3 > > >> solrconfig file with his request handler. > > > > > > And you need to add the dataimporthandler jar into Solr's lib > > > directory. DIH is not added automatically anymore. > > > > > >> > > >> Considering the application development: > > >> > > >> We used all the queries as BOOLEAN type style (was not good) I mean > put > > >> all the parameter in query fields i.e > > >> *:* AND EntityName: <<>> AND : AND . > > >> > > >> I think we should simplify our queries using other fields like df, qf > > > >> > > > > > > Probably. AND queries are best done by filter queries (fq). > > > > >
Re: HTTP Status 503 - Server is shutting down
Hello, I am able to configure solr 4.3.1 version with tomcat6. I followed these steps: 1. Extract solr431 package. In my case I did in "E:\solr-4.3.1\example\solr" 2. Now copied solr dir from extracted package (E:\solr-4.3.1\example\solr) into TOMCAT_HOME dir. In my case TOMCAT_HOME dir is pointed to E:\Apache\Tomcat 6.0. 3. I can refer now SOLR_HOME as " E:\Apache\Tomcat 6.0\solr" (please remember this) 4. Copy the solr.war file from extracted package to SOLR HOME dir i.e E:\Apache\Tomcat 6.0\solr. This is required to create the context. As I donot want to pass this as JAVA OPTS 5. Create solr1.xml file into TOMCAT_HOME\conf\Catalina\localhost (I gave file name as solr1.xml ) 6. Also copy solr.war file into TOMCAT_HOME\webapps for deployment purpose 7. If you start tomcat you will get errors as mentioned by Shawn. S0 you need to copy all the 5 jar files from solr extracted package ( E:\solr-4.3.1\example\lib\ext ) to TOMCAT_HOME\lib dir.(jul-to-slf4j-1.6.6, jcl-over-slf4j-1.6.6, slf4j-log4j12-1.6.6, slf4j-api-1.6.6,log4j-1.2.16) 8. Also copy the log4js.properties file from E:\solr-4.3.1\example\resources dir to TOMCAT_HOME\lib dir. 9. Now if you start the tomcat you wont having any problem. 10. As in my side I am using additional jar for data import requesthandler . So for this please modify the solrconfig.xml file to point the location of data import jar. 11. What I did : In solrconfig.xml file : In section I add one line after this section (If I use above line then I need to create lib dir inside Collection1 dir) 12. In SOLR_HOME (E:\Apache\Tomcat 6.0\solr) I created a lib folder because in my solrconfig.xml file I am referring this lib dir. And copied all the dataimport related jar files.(solr-dataimporthandler-4.3.1***) I did it in this way because I do not want to use TOMCAT_HOME\lib. 13. Now restart the tomcat I am sure there should not be any problem. If there is some problem, refer solr.log file which is in TOMCAT_HOME\logs dir. As I said in point 12 that I do not want to put jar files related to solr ino TOMCAT_HOME\lib dir, but for logging mechanism I have to do. I tried to put all the 5 jars into this folder and removed from TOMCAT lib.. but then I got the error. In Ideal scenario, we should not put all the jar files related to solr into TOMCAT lib dir Regards Sandeep On Mon, Jul 15, 2013 at 12:27 AM, PeterKerk wrote: > Ok, still getting the same error "HTTP Status 503 - Server is shutting > down", > so here's what I did now: > > - reinstalled tomcat > - deployed solr-4.3.1.war in C:\Program Files\Apache Software > Foundation\Tomcat 6.0\webapps > - copied log4j-1.2.16.jar,slf4j-api-1.6.6.jar,slf4j-log4j12-1.6.6.jar to > C:\Program Files\Apache Software Foundation\Tomcat > 6.0\webapps\solr-4.3.1\WEB-INF\lib > - copied log4j.properties from > C:\Dropbox\Databases\solr-4.3.1\example\resources to > C:\Dropbox\Databases\solr-4.3.1\example\lib > - restarted tomcat > > > Now this shows in my Tomcat console: > > 14-jul-2013 20:54:38 org.apache.catalina.core.AprLifecycleListener init > INFO: The APR based Apache Tomcat Native library which allows optimal > performanc > e in production environments was not found on the java.library.path: > C:\Program > Files\Apache Software Foundation\Tomcat > 6.0\bin;C:\Windows\Sun\Java\bin;C:\Windo > ws\system32;C:\Windows;C:\Program Files\Common Files\Microsoft > Shared\Windows Li > ve;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows > Live;C:\Windows\ > > system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShe > ll\v1.0\;C:\Program Files\TortoiseSVN\bin;c:\msxsl;C:\Program Files > (x86)\Window > s Live\Shared;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program > File > s (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files > (x86)\Windows > Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL > Server\110 > \Tools\Binn\;C:\Program Files (x86)\Microsoft SQL > Server\110\Tools\Binn\;C:\Prog > ram Files\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files > (x86)\Microsoft SQ > L Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft > SQL S > erver\110\DTS\Binn\;C:\Program Files (x86)\Java\jre6\bin;C:\Program > Files\Java\j > re631\bin;. > 14-jul-2013 20:54:39 org.apache.coyote.http11.Http11Protocol init > INFO: Initializing Coyote HTTP/1.1 on http-8080 > 14-jul-2013 20:54:39 org.apache.catalina.startup.Catalina load > INFO: Initialization processed in 287 ms > 14-jul-2013 20:54:39 org.apache.catalina.core.StandardService start > INFO: Starting service Catalina > 14-jul-2013 20:54:39 org.apache.catalina.core.StandardEngine start > INFO: Starting Servlet Engine: Apache Tomcat/6.0.37 > 14-jul-2013 20:54:39 org.apache.catalina.startup.HostConfig > deployDescriptor > INFO: Deploying configuration descriptor manager.xml > 14-jul-2013 20:54:39 org.apache.catalina.startup.HostConfig deployWAR > INFO: Deploying web application archive solr-4.3.1.war > log4
Re: Book contest idea - feedback requested
Hi Alex, great please go ahead.. -Sandeep On Tue, Jul 16, 2013 at 9:40 AM, Ali, Saqib wrote: > Hello Alex, > > This sounds like an excellent idea! :) > > Saqib > > > On Mon, Jul 15, 2013 at 8:11 PM, Alexandre Rafalovitch > wrote: > > > Hello, > > > > Packt Publishing has kindly agreed to let me run a contest with e-copies > of > > my book as prizes: > > http://www.packtpub.com/apache-solr-for-indexing-data/book > > > > Since my book is about learning Solr and targeted at beginners and early > > intermediates, here is what I would like to do. I am asking for feedback > on > > whether people on the mailing list like the idea or have specific > > objections to it. > > > > 1) The basic idea is to get Solr users and write and vote on what they > find > > hard with Solr, especially in understanding the features (as contrasted > > with just missing ones). > > 2) I'll probably set it up as a User Voice forum, which has all the > > mechanisms for suggesting and voting on ideas. With an easier interface > > than Jira > > 3) The top N voted ideas will get the books as prizes and I will try to > > fix/document/create JIRAs for those issues. > > 4) I am hoping to specifically reach out to the communities where Solr > is a > > component and where they don't necessarily hang out on our mailing list. > I > > am thinking SolrNet, Drupal, project Blacklight, Cloudera, CrafterCMS, > > SiteCore, Typo3, SunSpot, Nutch. Obviously, anybody and everybody from > this > > list would be absolutely welcome to participate as well. > > > > Yes? No? Suggestions? > > > > Also, if you are maintainer of one of the products/services/libraries > that > > has Solr in it and want to reach out to your community yourself, I think > it > > would be a lot better than If I did it. Contact me directly and I will > let > > you know what template/FAQ I want you to include in the announcement > > message when it is ready. > > > > Thank you all in advance for the comments and suggestions. > > > > Regards, > >Alex. > > > > Personal website: http://www.outerthoughts.com/ > > LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch > > - Time is the quality of nature that keeps events from happening all at > > once. Lately, it doesn't seem to be working. (Anonymous - via GTD book) > > >
Re: solr 4.3.1 Installation
This problem looks to me because of solr logging ... see below detail description (taken one of the mail thread) - Solr 4.3.0 and later does not have ANY slf4j jarfiles in the .war file, so you need to put them in your classpath. Jarfiles are included in the example, in example/lib/ext, and those jarfiles set up logging to use log4j, a much more flexible logging framework than JDK logging. JDK logging is typically set up with a file called logging.properties, which I think you must use a system property to configure. You aren't using JDK logging, you are using log4j, which uses a file called log4j.properties. http://wiki.apache.org/solr/SolrLogging#Using_the_example_logging_setup_in_containers_other_than_Jetty On Tue, Jul 16, 2013 at 6:28 PM, Sujatha Arun wrote: > Hi , > > We have been using solr 3.6.1 .Recently downloaded the solr 4.3.1 version > and installed the same as multicore setup as follows > > Folder Structure > solr.war > solr > conf >core0 > core1 > solr.xml > > Created the context fragment xml file in tomcat/conf/catalina/localhost > which refers to the solr.war file and the solr home folder > > copied the muticore conf folder without the zoo.cfg file > > I get the following error and admin page does not load > 16 Jul, 2013 11:36:09 PM org.apache.catalina.core.StandardContext start > SEVERE: Error filterStart > 16 Jul, 2013 11:36:09 PM org.apache.catalina.core.StandardContext start > SEVERE: Context [/solr_4.3.1] startup failed due to previous errors > 16 Jul, 2013 11:36:39 PM org.apache.catalina.startup.HostConfig > checkResources > INFO: Undeploying context [/solr_4.3.1] > 16 Jul, 2013 11:36:39 PM org.apache.catalina.core.StandardContext start > SEVERE: Error filterStart > 16 Jul, 2013 11:36:39 PM org.apache.catalina.core.StandardContext start > SEVERE: Context [/solr_4.3.1] startup failed due to previous errors > > > Please let me know what I am missing If i need to install this with the > default multicore setup without the cloud .Thanks > > Regards > Sujatha >
Re: HTTP Status 503 - Server is shutting down
Hi, I think I will also wait for other people reply as I do not have much idea now. I suggested the things because I did it recently but I have only one collection (default one) . As you said and I can guess... you have multiple collections like tt, shop and home in one solr instance.. By default all the collections should go inside solr dir (tomcat\solr)... And may be you need to modify the solr.xml file (tomcat\solr\solr.xml) See below. There is another xml file, I have given name as solr.xml also (\tomcat\conf\localhost\solr.xml) which has solr home path... and therefore starting of tomcat read this file after host-manager.xml Thanks -Sandeep On Wed, Jul 17, 2013 at 3:40 PM, PeterKerk wrote: > I can now approach http://localhost:8080/solr-4.3.1/#/, thanks!! > > I also noticed you mentioning something about a data import handler. > > Now, what I will be requiring after I've completed the basic setup of > Tomcat6 and Solr431 I want to migrate my Solr350 (now running on Cygwin) > cores to that environment. > > C:\Dropbox\Databases\apache-solr-3.5.0\example\example-DIH\solr\tt > C:\Dropbox\Databases\apache-solr-3.5.0\example\example-DIH\solr\shop > C:\Dropbox\Databases\apache-solr-3.5.0\example\example-DIH\solr\homes > > Where do I need to copy the above cores for this all to work? > What I don't understand is how Tomcat knows where it can find my Solr 4.3.1 > folder, in my case C:\Dropbox\Databases\solr-4.3.1, is that folder even any > longer required? > > Many thanks again! :) > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/HTTP-Status-503-Server-is-shutting-down-tp4065958p4078567.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Synonyms with wildcard search
Hello All, I want to know whether it is possible to make a query of word which has synonym+wildcard. For example : I have one field which is type of text_en (default fieldType in 4.3.1) And synonym.txt file has this entry colour => color Now when I am using full text search as colour* (with wild card) then search result is not returning the keyword of type colorology... (as in case If I use color* then I am getting this word) So any suggestions as how I can achieve this Or its not possible. Thanks Sandeep
Need Help in migrating Solr version 1.4 to 4.3
Hello All, We are planning to migrate solr 1.4 to Solr 4.3 version. And I am seeking some help in this side. Considering Schema file change: By default there are lots of changes if I compare original Solr 1.4 schema file to Sol 4.3 schema file. And that is the reason we are not copying paste of schema file. In our Solr 1.4 schema implementation, we have some custom fields with type "textgen" and "text" So in migration of these custom fields to Solr 4.3, should I use type of "text_general" as replacement of "textgen" and "text_en" as replacement of "text"? Please confirm the same. Considering Solrconfig change: As we didn't have lots of changes in 1.4 solrconfig file except the dataimport request handler. And therefore in migration side, we are simply modifying the Solr 4.3 solrconfig file with his request handler. Considering the application development: We used all the queries as BOOLEAN type style (was not good) I mean put all the parameter in query fields i.e *:* AND EntityName: <<>> AND : AND . I think we should simplify our queries using other fields like df, qf We also used to create Solr server object via CommonsHttpSolrServer() so I am planning to use now HttpSolrServer API> Please let me know the suggestion for above points also what are the other factors I need to take care while considering the migration. Regards Sandeep
