Making managed schema unmutable correctly?

[Alexandre Rafalovitch]

So, I am looking at the Solr 5.5 examples with their all-in by-default
managed schemas. And I am scratching my head on the workflow users are
expected to follow.

One example is straight from documentation:
"With the above configuration, you can use the Schema API to modify
the schema as much as you want, and then later change the value of
mutable to false if you wish to "lock" the schema in place and prevent
future changes."

Which sounds great, except right above the definition in the
solrconfig.xml, it says:
"Do NOT hand edit the managed schema - external modifications will be
ignored and overwritten as a result of schema modification REST API
calls."

And the Config API does not seem to provide any API to switch that
property (schemaFactory.mutable is not an editable property).

So, how is one supposed to lock the schema after modifying it? In the
default, non-cloud, example!

So far, the nearest I get is to unload the core (losing
core.properties), manually modify solrconfig.xml in violation of
instructions and add the core back. What am I missing?

Regards,
   Alex.


Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: indexing pdf files using post tool

[vidya]

Sorry for conveying it in wrong way. I want my data of 1 pdf file to be
indexed with different fields in a document of solr according to data in it
like name;id;title;content etc

Thanks 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/indexing-pdf-files-using-post-tool-tp4263811p4264052.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Text search NGram

[G, Rajesh]

Hi Emir,

The solution we wanted to implement is to show top 100 best match technology 
names from the list of technology names we have. Whatever technology names user 
has typed will first reach SQL Server and exact match will be done if 
possible[name==name] , only those do not exactly match[spelling mistakes, 
jumbled words] will be searched in SOLR.

With the below setup if I query title:(Microsoft Ofice 365) I get the below 
result [note:scores are same?]
{
"title":"Lync - Microsoft Office 365",
"score":7.7472024
},
{
"title":"Microsoft Office 365",
"score":7.7472024
},
When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) {
"title":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 
365 1.0",
"title_ws":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 
365 1.0",
"score":3.9297152
},
  {
"title":"Microsoft Office 365",
"title_ws":"Microsoft Office 365",
"score":3.1437721
}

When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) 
qf=title_ws^1
I don’t get any results

The expected result is
{
"title":"Microsoft Office 365",
"title_ws":"Microsoft Office 365",
},
  {
"title":"Microsoft Office 365 1.0",
"title_ws":"Microsoft Office 365 1.0",
},
  {
"title":"Microsoft Office 365 14.0",
"title_ws":"Microsoft Office 365 14.0",
},
  {
"title":"Microsoft Office 365 14.3",
"title_ws":"Microsoft Office 365 14.3",
},
  {
"title":"Microsoft Office 365 14.4",
"title_ws":"Microsoft Office 365 14.4",
},












  

  
  
  
  
  

  




  



  

  
  
  
  

  
  
  
  

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-mail and attachments by anyone 
other than the intended person(s) is prohibited.

-Original Message-
From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com]
Sent: Monday, March 7, 2016 8:16 PM
To: solr-user@lucene.apache.org
Subject: Re: Text search NGram

Not sure I understood question. What I meant is you to try setting 
omitNorms="false" to your txt_token field type if you want to stick with ngram 
only solution:


 
 
 
 
 
 
  
 
 
 
 
 
   


and to add new field type and field to keep nonngram version of field.
Something like:


 
 
 
 
 
  
 
 
 
 
   


and use copyField to copy to both fields and query title:test OR 
title_simple:test.

Emir


On 07.03.2016 15:31, G, Rajesh wrote:
> Hi Emir,
>
> I have already applied
>
>  and then I have applied 
> . 
> Is this what you wanted me to have in my config?
>
> Thanks
> Rajesh
>
>
>
> Corporate Executive Board India Private Limited. Registration No: 
> U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
> No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.
>
> This e-mail and/or its attachments are intended only for the use of the 
> addressee(s) and may contain confidential and legally privileged information 
> belonging to CEB and/or its subsidiaries, including CEB subsidiaries that 
> offer SHL Talent Measurement products and services. If you have received this 
> e-mail in error, please notify the sender and immediately, destroy all copies 
> of this email and its attachments. The publication, copying, in whole or in 
> part, or use or dissemination in any other way of this e-mail and attachments 
> by anyone other than the intended person(s) is prohibited.
>
> -Original Message-
> From: G, Rajesh

Re: Text search NGram

[Emir Arnautovic]

Hi Rajesh,
Did you reindex afters setting omitNorms to false? Can you send results 
with debug for first query?


What query parser are you using for these queries? You should run your 
queries with debug=true and see how they are rewritten - that should 
explain why some cases do not return expected documents. If you have 
trouble understanding why it is not returned, you can post response to 
this thread.


Thanks,
Emir

On 16.03.2016 09:30, G, Rajesh wrote:

Hi Emir,

The solution we wanted to implement is to show top 100 best match technology 
names from the list of technology names we have. Whatever technology names user 
has typed will first reach SQL Server and exact match will be done if 
possible[name==name] , only those do not exactly match[spelling mistakes, 
jumbled words] will be searched in SOLR.

With the below setup if I query title:(Microsoft Ofice 365) I get the below 
result [note:scores are same?]
{
 "title":"Lync - Microsoft Office 365",
 "score":7.7472024
},
{
 "title":"Microsoft Office 365",
 "score":7.7472024
},
When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) {
 "title":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 365 
1.0",
 "title_ws":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 365 
1.0",
 "score":3.9297152
},
   {
 "title":"Microsoft Office 365",
 "title_ws":"Microsoft Office 365",
 "score":3.1437721
}

When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) 
qf=title_ws^1
I don’t get any results

The expected result is
{
 "title":"Microsoft Office 365",
 "title_ws":"Microsoft Office 365",
},
   {
 "title":"Microsoft Office 365 1.0",
 "title_ws":"Microsoft Office 365 1.0",
},
   {
 "title":"Microsoft Office 365 14.0",
 "title_ws":"Microsoft Office 365 14.0",
},
   {
 "title":"Microsoft Office 365 14.3",
 "title_ws":"Microsoft Office 365 14.3",
},
   {
 "title":"Microsoft Office 365 14.4",
 "title_ws":"Microsoft Office 365 14.4",
},


 
 
 
 
 
 
 
 
 
 
   

   
   
   
   
   

   
 
 
 
 
   
 
 
 
   

   
   
   
   

   
   
   
   

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-mail and attachments by anyone 
other than the intended person(s) is prohibited.

-Original Message-
From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com]
Sent: Monday, March 7, 2016 8:16 PM
To: solr-user@lucene.apache.org
Subject: Re: Text search NGram

Not sure I understood question. What I meant is you to try setting 
omitNorms="false" to your txt_token field type if you want to stick with ngram 
only solution:


  
  
  
  
  
  
   
  
  
  
  
  



and to add new field type and field to keep nonngram version of field.
Something like:


  
  
  
  
  
   
  
  
  
  



and use copyField to copy to both fields and query title:test OR 
title_simple:test.

Emir


On 07.03.2016 15:31, G, Rajesh wrote:

Hi Emir,

I have already applied

 and then I have applied . Is this what you wanted me to 
have in my config?

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged in

Re: Text search NGram

[Emir Arnautovic]

Hi Rajesh,
It seems that title length is not different enough to have different 
fieldNorm - in all titles it is 0.5 so all documents for exact match 
query result in same score.


Query with "Ofice" results in wrong document being first because of its 
fieldNorm=1.0 - seems to me that this document was not reindexed after 
omitNorms=false.


Also noticed that ngram field is bit different in schema than in mail - 
has maxGramSize="800". Does not change explanation, but is easier to 
understand results when max=min.


HTH,
Emir

On 16.03.2016 10:31, G, Rajesh wrote:

Hi Emir,

Yes I have re-indexed after setting omitNorms to false. Attached is the result 
of the query in debug mode.

I am using LuceneQParser

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-mail and attachments by anyone 
other than the intended person(s) is prohibited.

-Original Message-
From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com]
Sent: Wednesday, March 16, 2016 2:39 PM
To: solr-user@lucene.apache.org
Subject: Re: Text search NGram

Hi Rajesh,
Did you reindex afters setting omitNorms to false? Can you send results with 
debug for first query?

What query parser are you using for these queries? You should run your queries 
with debug=true and see how they are rewritten - that should explain why some 
cases do not return expected documents. If you have trouble understanding why 
it is not returned, you can post response to this thread.

Thanks,
Emir

On 16.03.2016 09:30, G, Rajesh wrote:

Hi Emir,

The solution we wanted to implement is to show top 100 best match technology 
names from the list of technology names we have. Whatever technology names user 
has typed will first reach SQL Server and exact match will be done if 
possible[name==name] , only those do not exactly match[spelling mistakes, 
jumbled words] will be searched in SOLR.

With the below setup if I query title:(Microsoft Ofice 365) I get the
below result [note:scores are same?] {
  "title":"Lync - Microsoft Office 365",
  "score":7.7472024
},
{
  "title":"Microsoft Office 365",
  "score":7.7472024
},
When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) {
  "title":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 365 
1.0",
  "title_ws":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 
365 1.0",
  "score":3.9297152
},
{
  "title":"Microsoft Office 365",
  "title_ws":"Microsoft Office 365",
  "score":3.1437721
}

When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice
365) qf=title_ws^1 I don’t get any results

The expected result is
{
  "title":"Microsoft Office 365",
  "title_ws":"Microsoft Office 365", },
{
  "title":"Microsoft Office 365 1.0",
  "title_ws":"Microsoft Office 365 1.0", },
{
  "title":"Microsoft Office 365 14.0",
  "title_ws":"Microsoft Office 365 14.0", },
{
  "title":"Microsoft Office 365 14.3",
  "title_ws":"Microsoft Office 365 14.3", },
{
  "title":"Microsoft Office 365 14.4",
  "title_ws":"Microsoft Office 365 14.4", },


  
  
  
  
  
  
  
  
  
  









  
  
  
  

  
  
  












Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-ma

HMMChineseTokenizer splits up alphanumeric characters

[Zheng Lin Edwin Yeo]

Hi,

I'm using Solr 5.4.0, with the HMMChineseTokenizer in my Solr, and below is
my pipeline.


 









 
 









  
  

I found that HMMChineseTokenizer will split a string that consist of
numbers and characters (alphanumeric). For example, if I have a code that
looks like "1a2b3c4d", it will be split to 1 | a | 2 | b | 3 | c | 4 | d
This has caused the search query speed to slow quite tremendously (like at
least 10 seconds slower), as it has to search through individual tokens.

Would like to check, is there any way that we can solve this issue without
re-indexing? We have quite alot of code in the index which consist of
alphanumeric characters, and we have more than 10 million documents in the
index, so re-indexing with another tokenizer or pipeline is quite a huge
process.


Regards,
Edwin

Re: HMMChineseTokenizer splits up alphanumeric characters

[Zheng Lin Edwin Yeo]

Sorry, the correct pipeline which I'm using should be this:


  









  
  








  
  

Regards,
Edwin


On 16 March 2016 at 18:33, Zheng Lin Edwin Yeo  wrote:

> Hi,
>
> I'm using Solr 5.4.0, with the HMMChineseTokenizer in my Solr, and below
> is my pipeline.
>
>  positionIncrementGap="100" autoGeneratePhraseQueries="false">
>  
> 
> 
> 
>  words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
>  words="stopwords.txt" />
>  generateNumberParts="1" catenateWords="0" catenateNumbers="0"
> catenateAll="0" splitOnCaseChange="1"/>
>  ignoreCase="true" expand="false"/>
> 
>  maxGramSize="15"/>
>  
>  
> 
> 
> 
> 
>  words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
>  words="stopwords.txt" />
>  generateNumberParts="0" catenateWords="0" catenateNumbers="0"
> catenateAll="0" splitOnCaseChange="0"/>
>  ignoreCase="true" expand="false"/>
> 
>   
>   
>
> I found that HMMChineseTokenizer will split a string that consist of
> numbers and characters (alphanumeric). For example, if I have a code that
> looks like "1a2b3c4d", it will be split to 1 | a | 2 | b | 3 | c | 4 | d
> This has caused the search query speed to slow quite tremendously (like at
> least 10 seconds slower), as it has to search through individual tokens.
>
> Would like to check, is there any way that we can solve this issue without
> re-indexing? We have quite alot of code in the index which consist of
> alphanumeric characters, and we have more than 10 million documents in the
> index, so re-indexing with another tokenizer or pipeline is quite a huge
> process.
>
>
> Regards,
> Edwin
>

Re: Issue with Auto Suggester Component

[Manohar Sripada]

Thanks for the response.
If you see the first 5 results- "*ABC* Corporation", "*ABC*D
Corporation", "*Abc
*Tech", "*AbC*orporation", "*ABC*D company". The keyword "*abc*" that I am
trying to search is part of prefix of all the strings. Sorry, it's not
entire keyword to be of higher importance like #1, #3 and #6.
In the 2nd set of results, "The *ABC* Company", "The *ABC*DEF", the keyword
"*abc*" is not part of prefix of 1st string, but it is part of some other
string of each result.

Thanks,
Manohar

On Tue, Mar 15, 2016 at 3:03 PM, Alessandro Benedetti  wrote:

> Hi Manohar,
> I have not clear what should be your ideal ranking of suggestions.
>
> "I want prefix search of
> entire keyword to be of high preference (#1 to #5 in the below example)
> followed by prefix part of any other string (the last 2 in the below
> example). I am not bothered about ordering within 1st and 2nd set.
>
> ABC Corporation
> ABCD Corporation
> Abc Tech
> AbCorporation
> ABCD company
> The ABC Company
> The ABCDEF"
>
> Could you take the example you posted, show an example of query and the
> expected sort order ?
> According to your description of the problem
> Query : abc
> 1 Criteria : entire keyword to be of high preference
> I can't understand why you didn't count #3, #6 but you did #5 .
>
> 2 Criteria : followed by prefix part of any other string
> It is not that clear, probably you mean all the rest.
> Anyway an infix lookup algorithm with a boost for exact search should do
> the trick.
>
> Please give us some more details !
>
> Cheers
>
> On Tue, Mar 15, 2016 at 8:19 AM, Manohar Sripada 
> wrote:
>
> > Consider the below company names indexed. I want the below auto
> suggestions
> > to be listed when searched for "abc". Basically, I want prefix search of
> > entire keyword to be of high preference (#1 to #5 in the below example)
> > followed by prefix part of any other string (the last 2 in the below
> > example). I am not bothered about ordering within 1st and 2nd set.
> >
> > ABC Corporation
> > ABCD Corporation
> > Abc Tech
> > AbCorporation
> > ABCD company
> > The ABC Company
> > The ABCDEF
> >
> > I am using Suggest feature of solr as mentioned in the wiki
> > . I used
> > different Lookup implementations available, but, I couldn't get the
> result
> > as above. Here's is one sample config I used with
> BlendedInfixLookupFactory
> >
> >
> >  **
> > * businessNameBlendedInfixSuggester1*
> > * BlendedInfixLookupFactory*
> > * DocumentDictionaryFactory*
> > * business_name_suggest*
> > * id*
> > *text_suggest*
> > * business_name*
> > * linear*
> > * true*
> > * /app/solrnode/suggest_test_1_blendedinfix1*
> > * 0*
> > * true*
> > * true*
> > * false*
> > * *
> >
> > Can someone please suggest on how I can achieve this?
> >
> > Thanks,
> > Manohar
> >
>
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Re: Text search NGram

[Emir Arnautovic]

Hi Rajesh,
Here is one bit older presentation https://vimeo.com/32701503 but all 
should be still applicable. You can google for more with "understanding 
solr debug".


Regrads,
Emir

On 16.03.2016 11:30, G, Rajesh wrote:

Hi Emir,
Yes I have changed it to 800 to see if it produces different result. Sorry I 
have not inform that before. I have deleted all folder and files in data folder 
and I have re-indexed. Attached is the result with debug on

Can you please let me know whether there are any utility or a blog that will 
help in understanding the result of debug[parsedquery ,explain...]

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-mail and attachments by anyone 
other than the intended person(s) is prohibited.

-Original Message-
From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com]
Sent: Wednesday, March 16, 2016 3:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Text search NGram

Hi Rajesh,
It seems that title length is not different enough to have different fieldNorm 
- in all titles it is 0.5 so all documents for exact match query result in same 
score.

Query with "Ofice" results in wrong document being first because of its
fieldNorm=1.0 - seems to me that this document was not reindexed after 
omitNorms=false.

Also noticed that ngram field is bit different in schema than in mail - has 
maxGramSize="800". Does not change explanation, but is easier to understand 
results when max=min.

HTH,
Emir

On 16.03.2016 10:31, G, Rajesh wrote:

Hi Emir,

Yes I have re-indexed after setting omitNorms to false. Attached is the result 
of the query in debug mode.

I am using LuceneQParser

Thanks
Rajesh



Corporate Executive Board India Private Limited. Registration No: 
U741040HR2004PTC035324. Registered office: 6th Floor, Tower B, DLF Building 
No.10 DLF Cyber City, Gurgaon, Haryana-122002, India.

This e-mail and/or its attachments are intended only for the use of the 
addressee(s) and may contain confidential and legally privileged information 
belonging to CEB and/or its subsidiaries, including CEB subsidiaries that offer 
SHL Talent Measurement products and services. If you have received this e-mail 
in error, please notify the sender and immediately, destroy all copies of this 
email and its attachments. The publication, copying, in whole or in part, or 
use or dissemination in any other way of this e-mail and attachments by anyone 
other than the intended person(s) is prohibited.

-Original Message-
From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com]
Sent: Wednesday, March 16, 2016 2:39 PM
To: solr-user@lucene.apache.org
Subject: Re: Text search NGram

Hi Rajesh,
Did you reindex afters setting omitNorms to false? Can you send results with 
debug for first query?

What query parser are you using for these queries? You should run your queries 
with debug=true and see how they are rewritten - that should explain why some 
cases do not return expected documents. If you have trouble understanding why 
it is not returned, you can post response to this thread.

Thanks,
Emir

On 16.03.2016 09:30, G, Rajesh wrote:

Hi Emir,

The solution we wanted to implement is to show top 100 best match technology 
names from the list of technology names we have. Whatever technology names user 
has typed will first reach SQL Server and exact match will be done if 
possible[name==name] , only those do not exactly match[spelling mistakes, 
jumbled words] will be searched in SOLR.

With the below setup if I query title:(Microsoft Ofice 365) I get the
below result [note:scores are same?] {
   "title":"Lync - Microsoft Office 365",
   "score":7.7472024
},
{
   "title":"Microsoft Office 365",
   "score":7.7472024
},
When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice 365) {
   "title":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 365 
1.0",
   "title_ws":"RIM BlackBerry Enterprise Server (BES) for Microsoft Office 
365 1.0",
   "score":3.9297152
},
 {
   "title":"Microsoft Office 365",
   "title_ws":"Microsoft Office 365",
   "score":3.1437721
}

When I query title:(Microsoft Ofice 365) OR title_ws:(Microsoft Ofice
365) qf=title_ws^1 I don’t get any results

The expected result is
{
   "title"