Re: Text field size
Thanks for the answers. I'll definitely change the schema, adding new fields to improve quality of results. In the meanwhile I cannot throw away everything, because this is a production project. What I'm worried about is the size of this text field (about 4/5MB for each document). How can be big a text field? And what is the performances slowdown if it is so big. My idea is to add new fields with more or less relevance, depending from the content, and then reduce the size of big text field in the next days. Ciao, Vincenzo -- mobile: 3498513251 skype: free.dev > Il giorno 02 gen 2016, alle ore 15:20, Binoy Dalal > ha scritto: > > Indexing the various columns of your database table as separate fields into > solr will allow for more flexibility in terms of filter queries, faceting > and sorting your results. If you're looking to implement such features then > you should definitely be indexing every table column as a separate solr > field. > Additionally, this will also allow for greater flexibility for which fields > you want to query and/or display to the user. > >> On Sat, Jan 2, 2016 at 7:13 PM Upayavira wrote: >> >> Ask yourself what you want out of the index, how you want to query it, >> then the way to structure your index will become more clear. >> >> What sort of queries do you need to execute? >> >> Upayavira >> >>> On Sat, Jan 2, 2016, at 01:30 PM, Vincenzo D'Amore wrote: >>> Hi All, >>> >>> Recently I have started to work on an new project. I found a collection >>> where there are few documents (~8000), but each document has only a big >>> text field with about 4/5 MB. >>> >>> As far as I understood the text field is created by copying all the text >>> fields existing into a mssql table. So each row is transformed in a >>> document and all row fields are copied into a big text solr field. >>> >>> Now it seems obvious to me that rows fields should be copied into >>> different solr fields but I was curious to know what's the opinion of the >>> forum about this situation and what's the best approach to re-design the >>> collection. >>> >>> Bests, >>> Vincenzo >>> >>> -- >>> Vincenzo D'Amore >>> email: v.dam...@gmail.com >>> skype: free.dev >>> mobile: +39 349 8513251 >>> >>> >>> Ciao, >>> Vincenzo >>> >>> -- >>> mobile: 3498513251 >>> skype: free.dev > -- > Regards, > Binoy Dalal
Re: Issue with if() statement
OK. I was able to get it to work by looking at {!switch} code. I had a heck
of a time trying to get parse() to work as a separate class.
q.add(new TermQuery(new Term("state", "CO"), Occur.MUST);
return q;
That never did anything.
Here is some code that worked...
http://localhost:8983/solr/providersearchfull/select?wt=json&state1=state:NY&fl=ss,score&df=state&q={!orquery}$state,$state1
Notice $state was not sent... It works.
Thoughts?
package hg.parsers;
import org.apache.commons.lang.StringUtils;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;
import org.apache.solr.common.params.CommonParams;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.BooleanClause;
import org.apache.solr.parser.QueryParser;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.spans.SpanTermQuery;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.search.QueryParsing;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.search.QParser;
import org.apache.solr.search.QParserPlugin;
import org.apache.solr.search.SyntaxError;
/**
* Created by bbell on 1/3/16.
*/
public class OrQueryParserPlugin extends QParserPlugin {
//public static String NAME = "lucene";
@Override
public void init(NamedList args) {
}
@Override
public QParser createParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
return new QParser(qstr, localParams, params, req) {
QParser subParser;
@Override
public Query parse() throws SyntaxError {
String v = localParams.get(QueryParsing.V);
if (v == null || v.length() == 0) return null;
String subQ = null;
String[] arr = v.split(",");
System.out.println("parse arr:" + arr.length + "," + v);
if (arr != null && arr.length == 2) {
System.out.println("parse arr[0].length:" +
arr[0].length());
System.out.println("parse arr[1].length:" +
arr[1].length());
if (arr[0].trim().length() > 0) {
System.out.println("parse arr[0] params:" +
params.get(arr[0].trim()));
if (arr[0].trim().substring(0, 1).equals("$")) {
String val = params.get(arr[0].trim().substring(1));
if (val != null && val.length() > 0) {
subQ = val;
System.out.println("parse arr[0]:" +
arr[0].trim() + "," + subQ);
}
} else {
subQ = arr[0].trim();
System.out.println("parse arr[0]:" + subQ);
}
}
if (subQ == null && arr[1].trim().length() > 0) {
System.out.println("parse arr[1] params:" +
params.get(arr[1].trim()));
if (arr[1].trim().substring(0, 1).equals("$")) {
String val = params.get(arr[1].trim().substring(1));
if (val != null && val.length() > 0) {
subQ = val;
System.out.println("parse arr[1]:" +
arr[1].trim() + "," + subQ);
}
} else {
subQ = arr[1].trim();
System.out.println("parse arr[1]:" + subQ);
}
}
} else {
subQ = null;
}
subParser = subQuery(subQ, null);
return subParser.getQuery();
}
@Override
public String[] getDefaultHighlightFields() {
return subParser.getDefaultHighlightFields();
}
@Override
public Query getHighlightQuery() throws SyntaxError {
return subParser.getHighlightQuery();
}
@Override
public void addDebugInfo(NamedList debugInfo) {
subParser.addDebugInfo(debugInfo);
}
};
}
}
.
On Sat, Jan 2, 2016 at 5:03 AM, Upayavira wrote:
> Hrmph. I've got an Ant based codebase for building custom components
> against Solr. I've been asked on numerous occasions to publish this
> codebase. It is now at:
>
> https://github.com/upayavira/custom-solr-components
>
> There's no sample code in there yet. I'll see if I can stick one in
> there now, but I'm not sure how much time I have today to pull it all
> together (making it look nice and clear takes much more effort than
> making it work!).
>
> Upayavira
>
> On Sat, Jan 2, 2016, at 02:15 AM, William Bell wrote:
> > Sampl
Re: Text field size
AFAIK your text field can be as big as your resources make possible. The flip side is that, the bigger the field, the longer it'll take to search, and more importantly return. If you plan on storing and returning your entire 4/5 MB field, it is going to take you a lot of time. I mean A LOT! For one of my instances where we had all text fields of around 100kb in size with around a million similarly sized docs, it took a good 10 secs to search and return the fields. So you should plan accordingly. If you absolutely have to have such large fields then you should think about highlighting the results and then just returning the snippets instead of the entire document. I suggest that instead of making changes to your running production system, you set up a separate instance where you can play around and finalize your schema design. Once that is done, simply index all your content and point your app to the new solr instance and then take down the old one. This way you'll be sure to not mess things up in the prod system and will have minimal downtime. On Sun, Jan 3, 2016 at 3:27 PM Vincenzo D'Amore wrote: > Thanks for the answers. I'll definitely change the schema, adding new > fields to improve quality of results. > > In the meanwhile I cannot throw away everything, because this is a > production project. What I'm worried about is the size of this text field > (about 4/5MB for each document). How can be big a text field? And what is > the performances slowdown if it is so big. > > My idea is to add new fields with more or less relevance, depending from > the content, and then reduce the size of big text field in the next days. > > Ciao, > Vincenzo > > -- > mobile: 3498513251 > skype: free.dev > > > Il giorno 02 gen 2016, alle ore 15:20, Binoy Dalal < > binoydala...@gmail.com> ha scritto: > > > > Indexing the various columns of your database table as separate fields > into > > solr will allow for more flexibility in terms of filter queries, faceting > > and sorting your results. If you're looking to implement such features > then > > you should definitely be indexing every table column as a separate solr > > field. > > Additionally, this will also allow for greater flexibility for which > fields > > you want to query and/or display to the user. > > > >> On Sat, Jan 2, 2016 at 7:13 PM Upayavira wrote: > >> > >> Ask yourself what you want out of the index, how you want to query it, > >> then the way to structure your index will become more clear. > >> > >> What sort of queries do you need to execute? > >> > >> Upayavira > >> > >>> On Sat, Jan 2, 2016, at 01:30 PM, Vincenzo D'Amore wrote: > >>> Hi All, > >>> > >>> Recently I have started to work on an new project. I found a collection > >>> where there are few documents (~8000), but each document has only a big > >>> text field with about 4/5 MB. > >>> > >>> As far as I understood the text field is created by copying all the > text > >>> fields existing into a mssql table. So each row is transformed in a > >>> document and all row fields are copied into a big text solr field. > >>> > >>> Now it seems obvious to me that rows fields should be copied into > >>> different solr fields but I was curious to know what's the opinion of > the > >>> forum about this situation and what's the best approach to re-design > the > >>> collection. > >>> > >>> Bests, > >>> Vincenzo > >>> > >>> -- > >>> Vincenzo D'Amore > >>> email: v.dam...@gmail.com > >>> skype: free.dev > >>> mobile: +39 349 8513251 > >>> > >>> > >>> Ciao, > >>> Vincenzo > >>> > >>> -- > >>> mobile: 3498513251 > >>> skype: free.dev > > -- > > Regards, > > Binoy Dalal > -- Regards, Binoy Dalal
Re: Add support in FacetsComponent for facet.method=uif
For those interested I created a separate jira issue for this but forgot to attach earlier. https://issues.apache.org/jira/browse/SOLR-8466 On Jan 2, 2016 8:45 PM, "William Bell" wrote: > Yes we would like backward compatibility. We cannot switch all the facet > fields to DocValues and our faceting is slow. > > Please... > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson wrote: > > > Is there any interest in this? While i think it's important and inline > > with faceting available in the new json facet api, I've seen no > discussion > > on it so I'm wondering if it's best I add support for this using a custom > > facet component even though the majority of the component will be a copy > > which is prefer to not need to maintain separately. > > > > Jamie > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" wrote: > > > > > I had previously piggybacked on another post, but I think it may have > > been > > > lost there. I had a need to do UnInvertedField based faceting in the > > > FacetsComponent and as such started looking at what would be required > to > > > implement something similar to what the JSON Facets based API does in > > this > > > regard. The patch that I have in this regard works and is attached to > > > https://issues.apache.org/jira/browse/SOLR-8096, is that appropriate > or > > > should I create a new ticket to specifically add this support? > > > > > > -Jamie > > > > > > > > > -- > Bill Bell > billnb...@gmail.com > cell 720-256-8076 >
Re: Text field size
Thanks Binoy for sharing your experience that's a good suggestion. Thank you so much. Ciao, Vincenzo -- mobile: 3498513251 skype: free.dev > Il giorno 03 gen 2016, alle ore 14:02, Binoy Dalal > ha scritto: > > AFAIK your text field can be as big as your resources make possible. > The flip side is that, the bigger the field, the longer it'll take to > search, and more importantly return. If you plan on storing and returning > your entire 4/5 MB field, it is going to take you a lot of time. I mean A > LOT! For one of my instances where we had all text fields of around 100kb > in size with around a million similarly sized docs, it took a good 10 secs > to search and return the fields. So you should plan accordingly. If you > absolutely have to have such large fields then you should think about > highlighting the results and then just returning the snippets instead of > the entire document. > > I suggest that instead of making changes to your running production system, > you set up a separate instance where you can play around and finalize your > schema design. Once that is done, simply index all your content and point > your app to the new solr instance and then take down the old one. This way > you'll be sure to not mess things up in the prod system and will have > minimal downtime. > >> On Sun, Jan 3, 2016 at 3:27 PM Vincenzo D'Amore wrote: >> >> Thanks for the answers. I'll definitely change the schema, adding new >> fields to improve quality of results. >> >> In the meanwhile I cannot throw away everything, because this is a >> production project. What I'm worried about is the size of this text field >> (about 4/5MB for each document). How can be big a text field? And what is >> the performances slowdown if it is so big. >> >> My idea is to add new fields with more or less relevance, depending from >> the content, and then reduce the size of big text field in the next days. >> >> Ciao, >> Vincenzo >> >> -- >> mobile: 3498513251 >> skype: free.dev >> Il giorno 02 gen 2016, alle ore 15:20, Binoy Dalal < >>> binoydala...@gmail.com> ha scritto: >>> >>> Indexing the various columns of your database table as separate fields >> into >>> solr will allow for more flexibility in terms of filter queries, faceting >>> and sorting your results. If you're looking to implement such features >> then >>> you should definitely be indexing every table column as a separate solr >>> field. >>> Additionally, this will also allow for greater flexibility for which >> fields >>> you want to query and/or display to the user. >>> On Sat, Jan 2, 2016 at 7:13 PM Upayavira wrote: Ask yourself what you want out of the index, how you want to query it, then the way to structure your index will become more clear. What sort of queries do you need to execute? Upayavira > On Sat, Jan 2, 2016, at 01:30 PM, Vincenzo D'Amore wrote: > Hi All, > > Recently I have started to work on an new project. I found a collection > where there are few documents (~8000), but each document has only a big > text field with about 4/5 MB. > > As far as I understood the text field is created by copying all the >> text > fields existing into a mssql table. So each row is transformed in a > document and all row fields are copied into a big text solr field. > > Now it seems obvious to me that rows fields should be copied into > different solr fields but I was curious to know what's the opinion of >> the > forum about this situation and what's the best approach to re-design >> the > collection. > > Bests, > Vincenzo > > -- > Vincenzo D'Amore > email: v.dam...@gmail.com > skype: free.dev > mobile: +39 349 8513251 > > > Ciao, > Vincenzo > > -- > mobile: 3498513251 > skype: free.dev >>> -- >>> Regards, >>> Binoy Dalal > -- > Regards, > Binoy Dalal
Re: Text field size
Sorry Upayavira, I didn't see your question. This collection is used to search and retrieve the document if there is whatever matching query. I'm not sure if the entire document is returned or if the big text field is used only to search and return some ids, I have to investigate. Ciao, Vincenzo -- mobile: 3498513251 skype: free.dev > Il giorno 02 gen 2016, alle ore 14:43, Upayavira ha > scritto: > > Ask yourself what you want out of the index, how you want to query it, > then the way to structure your index will become more clear. > > What sort of queries do you need to execute? > > Upayavira > >> On Sat, Jan 2, 2016, at 01:30 PM, Vincenzo D'Amore wrote: >> Hi All, >> >> Recently I have started to work on an new project. I found a collection >> where there are few documents (~8000), but each document has only a big >> text field with about 4/5 MB. >> >> As far as I understood the text field is created by copying all the text >> fields existing into a mssql table. So each row is transformed in a >> document and all row fields are copied into a big text solr field. >> >> Now it seems obvious to me that rows fields should be copied into >> different solr fields but I was curious to know what's the opinion of the >> forum about this situation and what's the best approach to re-design the >> collection. >> >> Bests, >> Vincenzo >> >> -- >> Vincenzo D'Amore >> email: v.dam...@gmail.com >> skype: free.dev >> mobile: +39 349 8513251 >> >> >> Ciao, >> Vincenzo >> >> -- >> mobile: 3498513251 >> skype: free.dev
Re: Issue with if() statement
I haven’t fully digested this thread, but wanted to comment on this one.
¶m=$something doesn’t substitute. Only “local” params (inside local param
syntax curly brackets) do that. If you want to substitute a raw parameter use
curly brackets. You could, in this example, use &tt=${state1}, I believe, and
it would substitute. Confusingly, local param curly brackets are _not_ the
same as param substitution curly brackets.
I am skeptical that anything custom is needed for what you’re trying to do, but
again I haven’t fully digested what’s going on here yet. There is some param
substitution and {!switch} voodoo in example/files that ships with Solr, and
that might be helpful. Check out how that works. I’m in the polishing stages
of a a collaborative blog post on example/files that will detail these tricks a
bit - will publish that in the next few days.
—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com
>>
>> This return 236,000
>>
>>
>> http://localhost:8983/solr/providersearch/select?wt=json&state=state:%22CO%22&state1=state:%22NY%22&fl=ss,score&q=*:*&fq={!lucene%20v=$state}
>>
>>
>> This returns 10,000
>>
>> I want to put an IF statement around which v to use.
>>
>>
>>
>>
>>
>>
>> On Fri, Jan 1, 2016 at 1:52 PM, William Bell wrote:
>>
>>> Sure.
>>>
>>> If the state:NY returns results filter by state:NY, if it does not, then
>>> use state:CO. If we have results in NY, use it, otherwise use CO.
>>>
>>> OK?
>>>
>>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira wrote:
>>>
On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
> We are getting weird results with if(exists(a),b,c). We are getting
b+c!!
>
>
http://localhost:8983/solr/providersearch/select?q=*:*&wt=json&state=state:%22CO%22&state1=state:%22NY%22&fq=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>
> I am getting NY and CO!
>
> I only want $state1, which is NY.
>
> Any other ways to craft this?
Does this work at all?
The if() function is a function query that can be used to sort, boost
and as calculated fields. I haven't seen them used in filtering.
Also, the query() function does *not* do a query, it just says "what
would this document score for this query?"
Can you describe in English what you are trying to do?
Upayavira
>>>
>>>
>>>
>>> --
>>> Bill Bell
>>> billnb...@gmail.com
>>> cell 720-256-8076
>>>
>>
>>
>>
>> --
>> Bill Bell
>> billnb...@gmail.com
>> cell 720-256-8076
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
Re: Issue with if() statement
Thanks that indeed works tt=${state1}. But that was just a passing question.
I see an empty "*case*", "default" and a "case.VAL". I think I have
anything I need. After rereading. We might want to add to "case.VAL"
something like "case.'state:CO'" with a single quote. Right now it does not
appear to handle spaces or : I digress.
I got it to work with {!switch}.
http://localhost:8983/solr/providersearchfull/select?wt=xml&state1=state:%22NY%22&fl=score&q=*:*&fq={!switch%20case=$state2%20default=$state1%20v=$state1}
If the user passes state1, it will not be empty and the default will pick
up all 50 states.
If the user does not pass state1, but passes state2, the case empty will
pick up since $state1 is empty, and it works.
http://localhost:8983/solr/providersearchfull/select?wt=xml&state2=state:%22CO%22&fl=score&q=*:*&fq={!switch%20case=$state2%20default=$state1%20v=$state1}
The only that that changes is if I set as a param state1 or state2.
If I send both state1 and state2, state1 takes precedence.
http://localhost:8983/solr/providersearchfull/select?wt=xml&state1=state:%22NY%22&state2=state:%22CO%22&fl=score&q=*:*&fq={!switch%20case=$state2%20default=$state1%20v=$state1}
Kinda interesting.
The query I sent earlier : {!orquery}query1,query2 will return query1 is it
is not empty, otherwise if query1 is empty it will run query2. That was the
code I sent last night.
I am going to send some more code, since I could not get TermQuery to work
like I have it in PayLoad. And need some help debugging that.
Thanks!
On Sun, Jan 3, 2016 at 11:32 AM, Erik Hatcher
wrote:
> I haven’t fully digested this thread, but wanted to comment on this one.
> ¶m=$something doesn’t substitute. Only “local” params (inside local
> param syntax curly brackets) do that. If you want to substitute a raw
> parameter use curly brackets. You could, in this example, use
> &tt=${state1}, I believe, and it would substitute. Confusingly, local
> param curly brackets are _not_ the same as param substitution curly
> brackets.
>
> I am skeptical that anything custom is needed for what you’re trying to
> do, but again I haven’t fully digested what’s going on here yet. There is
> some param substitution and {!switch} voodoo in example/files that ships
> with Solr, and that might be helpful. Check out how that works. I’m in
> the polishing stages of a a collaborative blog post on example/files that
> will detail these tricks a bit - will publish that in the next few days.
>
> —
> Erik Hatcher, Senior Solutions Architect
> http://www.lucidworks.com
Re: Add support in FacetsComponent for facet.method=uif
Interesting that facet.method=dv or facet.method=uif. What is the difference? On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson wrote: > For those interested I created a separate jira issue for this but forgot to > attach earlier. > > https://issues.apache.org/jira/browse/SOLR-8466 > On Jan 2, 2016 8:45 PM, "William Bell" wrote: > > > Yes we would like backward compatibility. We cannot switch all the facet > > fields to DocValues and our faceting is slow. > > > > Please... > > > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson wrote: > > > > > Is there any interest in this? While i think it's important and inline > > > with faceting available in the new json facet api, I've seen no > > discussion > > > on it so I'm wondering if it's best I add support for this using a > custom > > > facet component even though the majority of the component will be a > copy > > > which is prefer to not need to maintain separately. > > > > > > Jamie > > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" wrote: > > > > > > > I had previously piggybacked on another post, but I think it may have > > > been > > > > lost there. I had a need to do UnInvertedField based faceting in the > > > > FacetsComponent and as such started looking at what would be required > > to > > > > implement something similar to what the JSON Facets based API does in > > > this > > > > regard. The patch that I have in this regard works and is attached > to > > > > https://issues.apache.org/jira/browse/SOLR-8096, is that appropriate > > or > > > > should I create a new ticket to specifically add this support? > > > > > > > > -Jamie > > > > > > > > > > > > > > > -- > > Bill Bell > > billnb...@gmail.com > > cell 720-256-8076 > > > -- Bill Bell billnb...@gmail.com cell 720-256-8076
Re: Add support in FacetsComponent for facet.method=uif
The patch adds facet.method=uif and then delegates all of the work to the JSON Faceting API to do the work. I had originally added a facet.method=dv and made the original facet.method=fc work using the UnInvertedField but wanted to avoid making a change that would introduce unexpected behavior. While I think it's strange that facet.method=dv does not exist and fc defaults to dv I think if we wanted to change that it should be done in another ticket. On Sun, Jan 3, 2016 at 4:18 PM, William Bell wrote: > Interesting that facet.method=dv or facet.method=uif. What is the > difference? > > On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson wrote: > > > For those interested I created a separate jira issue for this but forgot > to > > attach earlier. > > > > https://issues.apache.org/jira/browse/SOLR-8466 > > On Jan 2, 2016 8:45 PM, "William Bell" wrote: > > > > > Yes we would like backward compatibility. We cannot switch all the > facet > > > fields to DocValues and our faceting is slow. > > > > > > Please... > > > > > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson > wrote: > > > > > > > Is there any interest in this? While i think it's important and > inline > > > > with faceting available in the new json facet api, I've seen no > > > discussion > > > > on it so I'm wondering if it's best I add support for this using a > > custom > > > > facet component even though the majority of the component will be a > > copy > > > > which is prefer to not need to maintain separately. > > > > > > > > Jamie > > > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" wrote: > > > > > > > > > I had previously piggybacked on another post, but I think it may > have > > > > been > > > > > lost there. I had a need to do UnInvertedField based faceting in > the > > > > > FacetsComponent and as such started looking at what would be > required > > > to > > > > > implement something similar to what the JSON Facets based API does > in > > > > this > > > > > regard. The patch that I have in this regard works and is attached > > to > > > > > https://issues.apache.org/jira/browse/SOLR-8096, is that > appropriate > > > or > > > > > should I create a new ticket to specifically add this support? > > > > > > > > > > -Jamie > > > > > > > > > > > > > > > > > > > > > -- > > > Bill Bell > > > billnb...@gmail.com > > > cell 720-256-8076 > > > > > > > > > -- > Bill Bell > billnb...@gmail.com > cell 720-256-8076 >
Re: Add support in FacetsComponent for facet.method=uif
Ok the path appears to have dv and uif in there.? On Sun, Jan 3, 2016 at 4:40 PM, Jamie Johnson wrote: > The patch adds facet.method=uif and then delegates all of the work to the > JSON Faceting API to do the work. I had originally added a facet.method=dv > and made the original facet.method=fc work using the UnInvertedField but > wanted to avoid making a change that would introduce unexpected behavior. > While I think it's strange that facet.method=dv does not exist and fc > defaults to dv I think if we wanted to change that it should be done in > another ticket. > > On Sun, Jan 3, 2016 at 4:18 PM, William Bell wrote: > > > Interesting that facet.method=dv or facet.method=uif. What is the > > difference? > > > > On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson wrote: > > > > > For those interested I created a separate jira issue for this but > forgot > > to > > > attach earlier. > > > > > > https://issues.apache.org/jira/browse/SOLR-8466 > > > On Jan 2, 2016 8:45 PM, "William Bell" wrote: > > > > > > > Yes we would like backward compatibility. We cannot switch all the > > facet > > > > fields to DocValues and our faceting is slow. > > > > > > > > Please... > > > > > > > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson > > wrote: > > > > > > > > > Is there any interest in this? While i think it's important and > > inline > > > > > with faceting available in the new json facet api, I've seen no > > > > discussion > > > > > on it so I'm wondering if it's best I add support for this using a > > > custom > > > > > facet component even though the majority of the component will be a > > > copy > > > > > which is prefer to not need to maintain separately. > > > > > > > > > > Jamie > > > > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" > wrote: > > > > > > > > > > > I had previously piggybacked on another post, but I think it may > > have > > > > > been > > > > > > lost there. I had a need to do UnInvertedField based faceting in > > the > > > > > > FacetsComponent and as such started looking at what would be > > required > > > > to > > > > > > implement something similar to what the JSON Facets based API > does > > in > > > > > this > > > > > > regard. The patch that I have in this regard works and is > attached > > > to > > > > > > https://issues.apache.org/jira/browse/SOLR-8096, is that > > appropriate > > > > or > > > > > > should I create a new ticket to specifically add this support? > > > > > > > > > > > > -Jamie > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Bill Bell > > > > billnb...@gmail.com > > > > cell 720-256-8076 > > > > > > > > > > > > > > > -- > > Bill Bell > > billnb...@gmail.com > > cell 720-256-8076 > > > -- Bill Bell billnb...@gmail.com cell 720-256-8076
Re: Memory Usage increases by a lot during and after optimization .
Thanks for the reply Shawn and Erick. What *exactly* are you looking at that says Solr is using all your memory? You must be extremely specific when answering this question. This will determine whether we should be looking for a bug or not. A) Before I start the optimization, the server's memory usage is consistent at around 16GB, when Solr startsup and we did some searching. However, when I click on the optimization button, the memory usage increases gradually, until it reaches the maximum of 64GB which the server has. But this only happens to the collection with index of 200GB, and not other collections which has smaller index size (they are at most 1GB at the moment). In another message thread, you indicated that your max heap was set to 14GB. Java will only ever use that much memory for the program that is being run, plus a relatively small amount so that Java itself can operate. Any significantly large resident memory allocation beyond the max heap would be an indication of a bug in Java, not a bug in Solr. A) I am quite curious at this also, because in the Task Manager of the server, the amount of memory usage stated does not tally with the percentage of memory usage. When I start optimizatoin, the memory usage states the JVM is only using 14GB, but the percentage of memory usage is almost 100%, when I have 64GB RAM. I have check the other processes running in the server, and did not found any other processes that takes up a large amount of memory, and the total amount of memory usage for the whole sever is only around 16GB. Regards, Edwin On 3 January 2016 at 01:24, Erick Erickson wrote: > If you happen to be looking at "top" or the like, you > might be seeing virtual memory, see Uwe's > excellent blog here: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Best, > Erick > > On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey wrote: > > On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote: > >> But the problem I'm facing now is that during optimizing, the memory > usage > >> of the server hit the maximum of 64GB, and I believe the optimization > could > >> not be completed fully as there is not enough memory, so when I check > the > >> index again, it says that it is not optimized. Before the optimization, > the > >> memory usage was less than 16GB, so the optimization actually uses up > more > >> than 48GB of memory. > >> > >> Is it normal for an index size of 200GB to use up so much memory during > >> optimization? > > > > What *exactly* are you looking at that says Solr is using all your > > memory? You must be extremely specific when answering this question. > > This will determine whether we should be looking for a bug or not. > > > > It is completely normal for all modern operating systems to use all the > > memory when the amount of data being handled is large. Some of the > > memory will be allocated to programs like Java/Solr, and the operating > > system will use everything else to cache data from I/O operations on the > > disk. This is called the page cache. For Solr to perform well, the > > page cache must be large enough to effectively cache your index data. > > > > https://en.wikipedia.org/wiki/Page_cache > > > > In another message thread, you indicated that your max heap was set to > > 14GB. Java will only ever use that much memory for the program that is > > being run, plus a relatively small amount so that Java itself can > > operate. Any significantly large resident memory allocation beyond the > > max heap would be an indication of a bug in Java, not a bug in Solr. > > > > With the index size at 200GB, I would hope to have at least 128GB of > > memory in the server, but I would *want* 256GB. 64GB may not be enough > > for good performance. > > > > Thanks, > > Shawn > > >
Re: Add support in FacetsComponent for facet.method=uif
Are you looking at 8466 or 8096? The patch on 8466 is the one I'm referencing. I should remove the other as it is more change than I think should be done for this ticket. Jamie On Jan 3, 2016 8:47 PM, "William Bell" wrote: > Ok the path appears to have dv and uif in there.? > > On Sun, Jan 3, 2016 at 4:40 PM, Jamie Johnson wrote: > > > The patch adds facet.method=uif and then delegates all of the work to the > > JSON Faceting API to do the work. I had originally added a > facet.method=dv > > and made the original facet.method=fc work using the UnInvertedField but > > wanted to avoid making a change that would introduce unexpected behavior. > > While I think it's strange that facet.method=dv does not exist and fc > > defaults to dv I think if we wanted to change that it should be done in > > another ticket. > > > > On Sun, Jan 3, 2016 at 4:18 PM, William Bell > wrote: > > > > > Interesting that facet.method=dv or facet.method=uif. What is the > > > difference? > > > > > > On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson > wrote: > > > > > > > For those interested I created a separate jira issue for this but > > forgot > > > to > > > > attach earlier. > > > > > > > > https://issues.apache.org/jira/browse/SOLR-8466 > > > > On Jan 2, 2016 8:45 PM, "William Bell" wrote: > > > > > > > > > Yes we would like backward compatibility. We cannot switch all the > > > facet > > > > > fields to DocValues and our faceting is slow. > > > > > > > > > > Please... > > > > > > > > > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson > > > wrote: > > > > > > > > > > > Is there any interest in this? While i think it's important and > > > inline > > > > > > with faceting available in the new json facet api, I've seen no > > > > > discussion > > > > > > on it so I'm wondering if it's best I add support for this using > a > > > > custom > > > > > > facet component even though the majority of the component will > be a > > > > copy > > > > > > which is prefer to not need to maintain separately. > > > > > > > > > > > > Jamie > > > > > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" > > wrote: > > > > > > > > > > > > > I had previously piggybacked on another post, but I think it > may > > > have > > > > > > been > > > > > > > lost there. I had a need to do UnInvertedField based faceting > in > > > the > > > > > > > FacetsComponent and as such started looking at what would be > > > required > > > > > to > > > > > > > implement something similar to what the JSON Facets based API > > does > > > in > > > > > > this > > > > > > > regard. The patch that I have in this regard works and is > > attached > > > > to > > > > > > > https://issues.apache.org/jira/browse/SOLR-8096, is that > > > appropriate > > > > > or > > > > > > > should I create a new ticket to specifically add this support? > > > > > > > > > > > > > > -Jamie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Bill Bell > > > > > billnb...@gmail.com > > > > > cell 720-256-8076 > > > > > > > > > > > > > > > > > > > > > -- > > > Bill Bell > > > billnb...@gmail.com > > > cell 720-256-8076 > > > > > > > > > -- > Bill Bell > billnb...@gmail.com > cell 720-256-8076 >
