Thanks for the explanation, I was missing exaclty that! Now things works correctly also using the post script. However I don't think I need norms if I use id of same lenght (UUID), right? I just need strings with omitTermFreqAndPositions="false" I think.
On Thu, Jun 27, 2013 at 7:31 PM, Erick Erickson <erickerick...@gmail.com>wrote: > Right, string fields are a little tricky, they're easy to confuse with > fields that actually _do_ something. > > By default, norms and term frequencies are turned off for types based on ' > class="solr.StrField" '. So any field length normalization (i.e. terms that > appear in shorter fields count more) and term frequencies calculations are > _not_ include in the score calculation. > > Try blowing your index away and adding this to your fields to see the > difference.... > > omitNorms="false" omitTermFreqAndPositions="false" > > You probably want to either turn these on explicitly for your string types > or use a type based on 'class="solr.TextField" ' since these options > default to "false" for text fields. If you use something like > "keywordTokenizerFactory" you also won't get your URL split up into pieces. > And in that case you can also normalize the values with something like > lowerCaseFilter which you can't do with "string" types since they're > completely unanalyzed. > > Best > Erick > > > On Wed, Jun 26, 2013 at 11:34 AM, Flavio Pompermaier > <pomperma...@okkam.it>wrote: > > > Obviously I messed up with email thread...however I found a problem > > indexing my document via post.sh. > > This is basically my schema.xml: > > > > <schema name="dopa-schema" version="1.5"> > > <fields> > > <field name="url" type="string" indexed="true" stored="true" > > required="true" multiValued="false" /> > > <field name="itemid" type="string" indexed="true" stored="true" > > multiValued="true"/> > > <field name="_version_" type="long" indexed="true" stored="true"/> > > </fields> > > <uniqueKey>url</uniqueKey> > > <types> > > <fieldType name="string" class="solr.StrField" sortMissingLast="true" > > /> > > <fieldType name="long" class="solr.TrieLongField" precisionStep="0" > > positionIncrementGap="0"/> > > </types> > > </schema> > > > > and this is the document I tried to upload via post.sh: > > > > <add> > > <doc> > > <field name="url">http://test.example.org/first.html</field> > > <field name="itemid">1000</field> > > <field name="itemid">1000</field> > > <field name="itemid">1000</field> > > <field name="itemid">5000</field> > > </doc> > > <doc> > > <field name="url">http://test.example.org/second.html</field> > > <field name="itemid">1000</field> > > <field name="itemid">5000</field> > > </doc> > > </add> > > > > When playing with administration and debugging tools I discovered that > > searching for q=itemid:5000 gave me the same score for those docs, while > I > > was expecting different term frequencies between the first and the > second. > > In fact, using java to upload documents lead to correct results (3 > > occurrences of item 1000 in the first doc and 1 in the second), e.g.: > > document1.addField("itemid", "1000"); > > document1.addField("itemid", "1000"); > > document1.addField("itemid", "1000"); > > > > Am I right or am I missing something else? > > > > > > On Wed, Jun 26, 2013 at 5:18 PM, Jack Krupansky <j...@basetechnology.com > > >wrote: > > > > > If there is a bug... we should identify it. What's a sample post > command > > > that you issued? > > > > > > > > > -- Jack Krupansky > > > > > > -----Original Message----- From: Flavio Pompermaier > > > Sent: Wednesday, June 26, 2013 10:53 AM > > > > > > To: solr-user@lucene.apache.org > > > Subject: Re: URL search and indexing > > > > > > I was doing exactly that and, thanks to the administration page and > > > explanation/debugging, I checked if results were those expected. > > > Unfortunately, results were not correct submitting updates trough > post.sh > > > script (that use curl in the end). > > > Probably, if it founds the same tag (same value for the same > field-name), > > > it will collapse them. > > > Rewriting the same document in Java and submitting the updates did the > > > things work correctly. > > > > > > In my opinion this is a bug (of the entire process, then I don't know > it > > > this is a problem of curl or of the script itself). > > > > > > Best, > > > Flavio > > > > > > On Wed, Jun 26, 2013 at 4:18 PM, Erick Erickson < > erickerick...@gmail.com > > >* > > > *wrote: > > > > > > Flavio: > > >> > > >> You mention that you're new to Solr, so I thought I'd make sure > > >> you know that the admin/analysis page is your friend! I flat > > >> guarantee that as you try to index/search following the suggestions > > >> you'll scratch your head at your results and.... you'll discover that > > >> the analysis process isn't doing quite what you expect. The > > >> admin/analysis page shows you the transformation of the input > > >> at each stage, i.e. how the input is tokenized, what transformations > > >> are applied to each token etc. It's invaluable! > > >> > > >> Best > > >> Erick > > >> > > >> P.S. Feel free to un-check the "verbose" box, it provides lots > > >> of information but can be overwhelming, especially at first! > > >> > > >> On Wed, Jun 26, 2013 at 12:20 AM, Flavio Pompermaier > > >> <pomperma...@okkam.it> wrote: > > >> > Ok thank you all for the great help! > > >> > Now I'm ready to start playing with my index! > > >> > > > >> > Best, > > >> > Flavio > > >> > > > >> > > > >> > On Tue, Jun 25, 2013 at 11:40 PM, Jack Krupansky < > > >> j...@basetechnology.com>wrote: > > >> > > > >> >> Yeah, URL Classify does only do so much. That's why you need to > > combine > > >> >> multiple methods. > > >> >> > > >> >> As a fourth method, you could code up a short JavaScript "** > > >> >> StatelessScriptUpdateProcessor****" that did something like take a > > >> full > > >> >> domain name (such as output by URL Classify) and turn it into > > multiple > > >> >> values, each with more of the prefix removed, so that " > > >> lucene.apache.org" > > >> >> would index as: > > >> >> > > >> >> lucene.apache.org > > >> >> apache.org > > >> >> apache > > >> >> .org > > >> >> org > > >> >> > > >> >> And then the user could query by any of those partial domain names. > > >> >> > > >> >> But, if you simply tokenize the URL (copy the URL string to a text > > >> field), > > >> >> you automatically get most of that. The user can query by a URL > > >> fragment, > > >> >> such as "apache.org", ".org", "lucene.apache.org", etc. and the > > >> >> tokenization will strip out the punctuation. > > >> >> > > >> >> I'll add this script to my list of examples to add in the next rev > of > > >> >> my > > >> >> book. > > >> >> > > >> >> > > >> >> -- Jack Krupansky > > >> >> > > >> >> -----Original Message----- From: Flavio Pompermaier > > >> >> Sent: Tuesday, June 25, 2013 10:06 AM > > >> >> > > >> >> To: solr-user@lucene.apache.org > > >> >> Subject: Re: URL search and indexing > > >> >> > > >> >> I bought the book and looking at the example I still don't > understand > > >> if it > > >> >> possible query all sub-urls of my URL. > > >> >> For example, if the URLClassifyProcessorFactory takes in input >> > > >> "url_s":" > > >> >> http://lucene.apache.org/solr/****4_0_0/changes/Changes.html< > > http://lucene.apache.org/solr/**4_0_0/changes/Changes.html> > > >> < > > >> http://lucene.apache.org/solr/**4_0_0/changes/Changes.html< > > http://lucene.apache.org/solr/4_0_0/changes/Changes.html> > > >> >" > > >> >> and makes some > > >> >> outputs like > > >> >> - "url_domain_s":"lucene.apache.****org <http://lucene.apache.org > >" > > >> >> - "url_canonical_s":" > > >> >> http://lucene.apache.org/solr/****4_0_0/changes/Changes.html< > > http://lucene.apache.org/solr/**4_0_0/changes/Changes.html> > > >> < > > >> http://lucene.apache.org/solr/**4_0_0/changes/Changes.html< > > http://lucene.apache.org/solr/4_0_0/changes/Changes.html> > > >> > > > >> >> " > > >> >> How should I configure url_domain_s in order to be able to makes > > query > > >> like > > >> >> '*.apache.org'? > > >> >> How should I configure url_canonical_s in order to be able to makes > > >> query > > >> >> like 'http://lucene.apache.org/****solr/*< > > http://lucene.apache.org/**solr/*>< > > >> http://lucene.apache.org/solr/*** <http://lucene.apache.org/solr/*>> > > >> >> '? > > >> >> Is it better to have two different fields for the two queries or > > could > > >> >> I > > >> >> create just one field for the two kind of queries (obviously for > the > > >> former > > >> >> case then I should query something like *://.apache.org/*)? > > >> >> > > >> >> > > >> >> On Tue, Jun 25, 2013 at 3:15 PM, Jack Krupansky < > > >> j...@basetechnology.com>* > > >> >> *wrote: > > >> >> > > >> >> There are examples in my book: > > >> >>> http://www.lulu.com/shop/jack-******krupansky/solr-4x-deep-** > > >> dive-****< > > http://www.lulu.com/shop/jack-****krupansky/solr-4x-deep-dive-****> > > >> < > > >> http://www.lulu.com/shop/jack-****krupansky/solr-4x-deep-dive-****< > > http://www.lulu.com/shop/jack-**krupansky/solr-4x-deep-dive-**> > > >> > > > >> >>> > early-access-release-1/ebook/******product-21079719.html<http:**//** > > >> >>> www.lulu.com/shop/jack-****krupansky/solr-4x-deep-dive-**< > > http://www.lulu.com/shop/jack-**krupansky/solr-4x-deep-dive-**> > > >> >>> early-access-release-1/ebook/****product-21079719.html< > > >> http://www.lulu.com/shop/jack-**krupansky/solr-4x-deep-dive-** > > >> early-access-release-1/ebook/**product-21079719.html< > > > http://www.lulu.com/shop/jack-krupansky/solr-4x-deep-dive-early-access-release-1/ebook/product-21079719.html > > > > > >> > > > >> >>> > > > >> >>> > > >> >>> > > >> >>> But... I still think you should use a tokenized text field as > well - > > >> use > > >> >>> all three: raw string, tokenized text, and URL classification > > fields. > > >> >>> > > >> >>> -- Jack Krupansky > > >> >>> > > >> >>> -----Original Message----- From: Flavio Pompermaier > > >> >>> Sent: Tuesday, June 25, 2013 9:02 AM > > >> >>> To: solr-user@lucene.apache.org > > >> >>> Subject: Re: URL search and indexing > > >> >>> > > >> >>> > > >> >>> That's sound exactly what I'm looking for! However I cannot find > an > > >> >>> example > > >> >>> of how to use it..could you help me please? > > >> >>> Moreover, about id field, isn't true that id field shouldn't be > > >> analyzed > > >> >>> as > > >> >>> suggested in > > >> >>> > > >> http://wiki.apache.org/solr/******UniqueKey#Text_field_in_the_** > > >> ****document< > > http://wiki.apache.org/solr/****UniqueKey#Text_field_in_the_****document > > > > >> < > > >> > > http://wiki.apache.org/solr/****UniqueKey#Text_field_in_the_****document > < > > http://wiki.apache.org/solr/**UniqueKey#Text_field_in_the_**document> > > >> > > > >> >>> <http://wiki.apache.**org/**solr/UniqueKey#Text_field_**** > > >> in_the_document< > > >> http://wiki.apache.org/solr/**UniqueKey#Text_field_in_the_**document< > > http://wiki.apache.org/solr/UniqueKey#Text_field_in_the_document> > > >> > > > >> >>> > > > >> >>> > > >> >>> ? > > >> >>> > > >> >>> > > >> >>> On Tue, Jun 25, 2013 at 2:47 PM, Jan Høydahl < > jan....@cominvent.com > > > > > >> >>> wrote: > > >> >>> > > >> >>> Sure you can query the url directly. Or if you choose you can > split > > >> it up > > >> >>> > > >> >>>> in multiple components, e.g. using > > >> >>>> http://lucene.apache.org/solr/******4_3_0/solr-core/org/** > > >> apache/****< > > http://lucene.apache.org/solr/****4_3_0/solr-core/org/apache/****> > > >> < > > >> http://lucene.apache.org/solr/****4_3_0/solr-core/org/apache/****< > > http://lucene.apache.org/solr/**4_3_0/solr-core/org/apache/**> > > >> > > > >> >>>> solr/update/processor/******URLClassifyProcessor.html<**http** > > >> >>>> :// > lucene.apache.org/solr/4_3_****0/solr-core/org/apache/solr/**** > > <http://lucene.apache.org/solr/4_3_**0/solr-core/org/apache/solr/**> > > >> >>>> update/processor/****URLClassifyProcessor.html< > > >> http://lucene.apache.org/solr/**4_3_0/solr-core/org/apache/** > > >> solr/update/processor/**URLClassifyProcessor.html< > > > http://lucene.apache.org/solr/4_3_0/solr-core/org/apache/solr/update/processor/URLClassifyProcessor.html > > > > > >> > > > >> >>>> > > > >> >>>> > > >> >>>> > > >> >>>> -- > > >> >>>> Jan Høydahl, search solution architect > > >> >>>> Cominvent AS - www.cominvent.com > > >> >>>> > > >> >>>> 25. juni 2013 kl. 14:10 skrev Flavio Pompermaier < > > >> pomperma...@okkam.it>: > > >> >>>> > > >> >>>> > Sorry but maybe I miss something here..could I declare url as > key > > >> field > > >> >>>> and > > >> >>>> > query it too..? > > >> >>>> > At the moment, my schema.xml looks like: > > >> >>>> > > > >> >>>> > <fields> > > >> >>>> > <field name="url" type="string" indexed="true" > stored="true" > > >> >>>> > required="true" multiValued="false" /> > > >> >>>> > > > >> >>>> > <field name="category" type="string" indexed="true" > > >> stored="true"/> > > >> >>>> > <field name="language" type="string" indexed="true" > > >> stored="true"/> > > >> >>>> > ... > > >> >>>> > <field name="_version_" type="long" indexed="true" >>>> > > > >> stored="true"/> > > >> >>>> > > > >> >>>> > </fields> > > >> >>>> > <uniqueKey>url</uniqueKey> > > >> >>>> > > > >> >>>> > Is it ok? or should I add a "baseurl" field of some kind to be > > able > > >> to > > >> >>>> > query all url coming from a certain domain (1st or 2nd level as > > >> well)? > > >> >>>> > > > >> >>>> > Best, > > >> >>>> > Flavio > > >> >>>> > > > >> >>>> > > > >> >>>> > On Tue, Jun 25, 2013 at 12:28 PM, Jan Høydahl < > > >> jan....@cominvent.com> > > >> >>>> wrote: > > >> >>>> > > > >> >>>> >> Probably a good match for the RegExp feature of Solr (given > that > > >> your > > >> >>>> url > > >> >>>> >> is not tokenized) > > >> >>>> >> e.g. q=url:/.*\.it$/ > > >> >>>> >> > > >> >>>> >> -- > > >> >>>> >> Jan Høydahl, search solution architect > > >> >>>> >> Cominvent AS - www.cominvent.com > > >> >>>> >> > > >> >>>> >> 25. juni 2013 kl. 12:17 skrev Flavio Pompermaier < > > >> >>>> pomperma...@okkam.it > > >> >>>> >: > > >> >>>> >> > > >> >>>> >>> Hi to everybody, > > >> >>>> >>> I'm quite new to Solr so maybe my question could be trivial > for > > >> you.. > > >> >>>> >>> In my use case I have to index stuff contained in some URL > so i > > >> use > > >> >>>> >>> url > > >> >>>> >> as > > >> >>>> >>> key of my document and I treat it like a string. > > >> >>>> >>> > > >> >>>> >>> However I'd like to be able to query by domain name, like > *.it > > or > > >> *. > > >> >>>> >>> somesite.com, what's the best strategy? I tought to made a > URL > > >> to > > >> >>>> path > > >> >>>> >>> transfromation and indexed using solr.**** > > >> >>>> PathHierarchyTokenizerFactory > > >> >>>> > > >> >>>> >>> but > > >> >>>> >>> maybe there's a simpler solution..isn't it? > > >> >>>> >>> > > >> >>>> >>> Best, > > >> >>>> >>> Flavio > > >> >>>> >>> > > >> >>>> >>> -- > > >> >>>> >>> > > >> >>>> >>> Flavio Pompermaier > > >> >>>> >>> *Development Department > > >> >>>> >>> *_____________________________******__________________ > > >> >>>> > > >> >>>> >>> *OKKAM**Srl **- www.okkam.it* > > >> >>>> >>> > > >> >>>> >>> *Phone:* +(39) 0461 283 702 > > >> >>>> >>> *Fax:* + (39) 0461 186 6433 > > >> >>>> >>> *Email:* f.pomperma...@okkam.it > > >> >>>> >>> *Headquarters:* Trento (Italy), fraz. Villazzano, Salita dei > > >> Molini 2 > > >> >>>> >>> *Registered office:* Trento (Italy), via Segantini 23 > > >> >>>> >>> > > >> >>>> >>> Confidentially notice. This e-mail transmission may contain > > >> legally > > >> >>>> >>> privileged and/or confidential information. Please do not > read > > it > > >> if > > >> >>>> you > > >> >>>> >>> are not the intended recipient(S). Any use, distribution, >>> > > >> >>>> reproduction > > >> >>>> or > > >> >>>> >>> disclosure by any other person is strictly prohibited. If you > > >> >>>> >>> have > > >> >>>> >> received > > >> >>>> >>> this e-mail in error, please notify the sender and destroy > the > > >> >>>> >>> >>> > > >> >>>> original > > >> >>>> >>> transmission and its attachments without reading or saving it > > in > > >> any > > >> >>>> >> manner. > > >> >>>> >> > > >> >>>> >> > > >> >>>> > > > >> >>>> > > > >> >>>> > -- > > >> >>>> > > > >> >>>> > Flavio Pompermaier > > >> >>>> > *Development Department > > >> >>>> > *_____________________________******__________________ > > >> >>>> > > >> >>>> > *OKKAM**Srl **- www.okkam.it* > > >> >>>> > > > >> >>>> > *Phone:* +(39) 0461 283 702 > > >> >>>> > *Fax:* + (39) 0461 186 6433 > > >> >>>> > *Email:* f.pomperma...@okkam.it > > >> >>>> > *Headquarters:* Trento (Italy), fraz. Villazzano, Salita dei > > Molini > > >> 2 > > >> >>>> > *Registered office:* Trento (Italy), via Segantini 23 > > >> >>>> > > > >> >>>> > Confidentially notice. This e-mail transmission may contain > > legally > > >> >>>> > privileged and/or confidential information. Please do not read > it > > >> if > > > >> >>>> you > > >> >>>> > are not the intended recipient(S). Any use, distribution, > > >> reproduction > > >> >>>> > or > > >> >>>> > disclosure by any other person is strictly prohibited. If you > > have > > >> >>>> received > > >> >>>> > this e-mail in error, please notify the sender and destroy the > > >> original > > >> >>>> > transmission and its attachments without reading or saving it > in > > >> >>>> > any > > >> >>>> manner. > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>> -- > > >> >>> > > >> >>> Flavio Pompermaier > > >> >>> *Development Department > > >> >>> *_____________________________******__________________ > > >> >>> > > >> >>> *OKKAM**Srl **- www.okkam.it* > > >> >>> > > >> >>> *Phone:* +(39) 0461 283 702 > > >> >>> *Fax:* + (39) 0461 186 6433 > > >> >>> *Email:* f.pomperma...@okkam.it > > >> >>> *Headquarters:* Trento (Italy), fraz. Villazzano, Salita dei > Molini > > 2 > > >> >>> *Registered office:* Trento (Italy), via Segantini 23 > > >> >>> > > >> >>> Confidentially notice. This e-mail transmission may contain > legally > > >> >>> privileged and/or confidential information. Please do not read it > if > > >> you > > >> >>> are not the intended recipient(S). Any use, distribution, > > reproduction > > >> or > > >> >>> disclosure by any other person is strictly prohibited. If you have > > >> >>> received > > >> >>> this e-mail in error, please notify the sender and destroy the >>> > > >> original > > >> >>> transmission and its attachments without reading or saving it in > any > > >> >>> manner. > > >> >>> > > >> >>> > > >> >> > > >> >> > > >> >> -- > > >> >> > > >> >> Flavio Pompermaier > > >> >> *Development Department > > >> >> *_____________________________****__________________ > > >> >> *OKKAM**Srl **- www.okkam.it* > > >> >> > > >> >> *Phone:* +(39) 0461 283 702 > > >> >> *Fax:* + (39) 0461 186 6433 > > >> >> *Email:* f.pomperma...@okkam.it > > >> >> *Headquarters:* Trento (Italy), fraz. Villazzano, Salita dei > Molini 2 > > >> >> *Registered office:* Trento (Italy), via Segantini 23 > > >> >> > > >> >> Confidentially notice. This e-mail transmission may contain legally > > >> >> privileged and/or confidential information. Please do not read it > if > > >> >> you > > >> >> are not the intended recipient(S). Any use, distribution, > > reproduction > > >> or > > >> >> disclosure by any other person is strictly prohibited. If you have > > >> received > > >> >> this e-mail in error, please notify the sender and destroy the > > original > > >> >> transmission and its attachments without reading or saving it in > any > > >> >> manner. > > >> >> > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > Flavio Pompermaier > > >> > *Development Department > > >> > *_____________________________**__________________ > > >> > *OKKAM**Srl **- www.okkam.it* > > >> > > > >> > *Phone:* +(39) 0461 283 702 > > >> > *Fax:* + (39) 0461 186 6433 > > >> > *Email:* f.pomperma...@okkam.it > > >> > *Headquarters:* Trento (Italy), fraz. Villazzano, Salita dei Molini > 2 > > >> > *Registered office:* Trento (Italy), via Segantini 23 > > >> > > > >> > Confidentially notice. This e-mail transmission may contain legally > > >> > privileged and/or confidential information. Please do not read it if > > you > > >> > are not the intended recipient(S). Any use, distribution, > reproduction > > >> > or > > >> > disclosure by any other person is strictly prohibited. If you have > > >> received > > >> > this e-mail in error, please notify the sender and destroy the > > original > > >> > transmission and its attachments without reading or saving it in any > > >> manner. > > >> > > >> > > > > > >
