Re: Solr MultiValue Fields and adding values
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I was hoping that wasent going to be the case... I ended up querying for all unique IDs in the DB, and then querying for each unique ID and getting all names, and then inserting them that way... Seems a lot slower than in theory it really should be... Thanks. - --Tiernan On 18/10/2011 23:20, Otis Gospodnetic wrote: > Hi, > > You'll need to construct the whole document and index it as such. You can't append values to document fields. > > Otis > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Lucene ecosystem search :: http://search-lucene.com/ > > >> ____ >> From: Tiernan OToole >> To: solr-user@lucene.apache.org >> Sent: Tuesday, October 18, 2011 11:41 AM >> Subject: Solr MultiValue Fields and adding values >> >> Good morning. >> >> I asked this question on StackOverflow, but though this group may be >> able to help... the question is available on SO here: http://bit.ly/r6MAWU >> >> here goes: >> >> I am building a search engine, and have a not so unique ID for a lot of >> different names... So, for example, there could be an id of B0051QVF7A >> which would have multiple names like "Kindle" "Amazon Kindle" "Amazon >> Kindle 3G" "Kindle Ebook Reader" "New Kindle" etc. >> >> The problem, and question i have, is that i am trying to enter this data > >from a DB of 11 ish million rows. each is being read one at a time. So i >> dont have all the names of each ID. I am adding new documents to the >> list each time. >> >> What i am trying to find out is how do i add names to an existing >> Document? if i am reading documentation correctly, it seems to overwrite >> the whole document, not add extra info to the field... i just want to >> add an extra name to the document multivalue field... >> >> I know this could cause some weird and wonderful "issues" if a name is >> removed (in the example above, "New Kindle" could be removed when a >> newer Kindle gets released) but i am thinking of recreating the index >> every now and again, to clear out issues like that (once a month or so. >> Its taking about 45min currently to create the index). >> >> So, how do you add a value to a multivalue field in solr for an existing >> document? >> >> Thanks in advance. >> >> --Tiernan >> >> >> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6eohcACgkQW5AKVqf62MEDiACgrYRvLITHbR2fv//dokfRem1g gJcAoN0f8geuBJHHASRNGS4yDWc/RX2H =4exA -END PGP SIGNATURE-
Re: Solr MultiValue Fields and adding values
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks for the comment. Sounds like too much of a change in all fairness... I have actually made a tweak to my DB to allow multiple names, and storing them off the main table. my query then only needs to query the IDs, and then the second table to get the names. but i will keep the comments in mind and see how things go over the next while. As a side note, if i were to go down the "get doc from solr, modify, commit back to solr" is it really that simple? run a query on Solr, get the document, add the extra data, and insert back to solr? Thanks. - --Tiernan On 19/10/2011 15:26, Dyer, James wrote: > While Solr/Lucene can't support true document updates, there are 2 ways you might be able to work around this in your situation. > > 1. If you store all of the fields, you can write something that will read back everything already indexed to the document, append whatever data you want, then write it back. This will increase index size and possibly make indexing too slow. On the other hand, it might be more efficient than requiring the database to return everything in order. > > 2. You could store your data as multiple documents per id (pick something else as your unique id). Then use the grouping functionality to roll up on your unique id whenever you query. This will mean changes to your application, probably a bigger index, and likely somewhat slower querying. But the performance losses might be slight and this seems to me like it maybe would be a good solution in your case. Perhaps it would make it so you wouldn't have to entirely re-index each month or so. See http://wiki.apache.org/solr/FieldCollapsing for more information. > > James Dyer > E-Commerce Systems > Ingram Content Group > (615) 213-4311 > > > -Original Message- > From: Tiernan OToole [mailto:lsmart...@gmail.com] > Sent: Wednesday, October 19, 2011 5:11 AM > To: solr-user@lucene.apache.org > Cc: Otis Gospodnetic > Subject: Re: Solr MultiValue Fields and adding values > > > I was hoping that wasent going to be the case... I ended up querying for > all unique IDs in the DB, and then querying for each unique ID and > getting all names, and then inserting them that way... Seems a lot > slower than in theory it really should be... > > Thanks. > > --Tiernan > > On 18/10/2011 23:20, Otis Gospodnetic wrote: > > Hi, > > > You'll need to construct the whole document and index it as such. You > can't append values to document fields. > > > Otis > > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > > Lucene ecosystem search :: http://search-lucene.com/ > > > >> > >> From: Tiernan OToole > >> To: solr-user@lucene.apache.org > >> Sent: Tuesday, October 18, 2011 11:41 AM > >> Subject: Solr MultiValue Fields and adding values > >> > >> Good morning. > >> > >> I asked this question on StackOverflow, but though this group may be > >> able to help... the question is available on SO here: > http://bit.ly/r6MAWU > >> > >> here goes: > >> > >> I am building a search engine, and have a not so unique ID for a lot of > >> different names... So, for example, there could be an id of B0051QVF7A > >> which would have multiple names like "Kindle" "Amazon Kindle" "Amazon > >> Kindle 3G" "Kindle Ebook Reader" "New Kindle" etc. > >> > >> The problem, and question i have, is that i am trying to enter this data > > >from a DB of 11 ish million rows. each is being read one at a time. So i > >> dont have all the names of each ID. I am adding new documents to the > >> list each time. > >> > >> What i am trying to find out is how do i add names to an existing > >> Document? if i am reading documentation correctly, it seems to overwrite > >> the whole document, not add extra info to the field... i just want to > >> add an extra name to the document multivalue field... > >> > >> I know this could cause some weird and wonderful "issues" if a name is > >> removed (in the example above, "New Kindle" could be removed when a > >> newer Kindle gets released) but i am thinking of recreating the index > >> every now and again, to clear out issues like that (once a month or so. > >> Its taking about 45min currently to create the index). > >> > >> So, how do you add a value to a multivalue field in solr for an existing > >> document? > >> > >> Thanks in advance. > >> > >> --Tiernan > >> > >> > >> > > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6e5j4ACgkQW5AKVqf62MHcnACbBGtTs25FjGe8Rs7q9DyO0J5r VnEAnRiPe4KCe717i//aPFiAlYsLwELB =eqRg -END PGP SIGNATURE-
Re: Solr MultiValue Fields and adding values
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thats what i though too... we see what the speed difference actually is... running some tests now... Thanks for the info! - --Tiernan On 19/10/2011 16:07, Dyer, James wrote: > > Not that I am doing this with any of my indexes, but I'm pretty sure the "get doc from solr, modify, commit back to solr" approach really is that simple. Just be sure you are storing the exact raw data that came from your database (typically you would). The problem with this approach is it potentially could be very slow if you're updating lots of documents. > > > > James Dyer > > E-Commerce Systems > > Ingram Content Group > > (615) 213-4311 > > > > From: Tiernan OToole [mailto:lsmart...@gmail.com] > Sent: Wednesday, October 19, 2011 10:01 AM > To: solr-user@lucene.apache.org > Cc: Dyer, James > Subject: Re: Solr MultiValue Fields and adding values > > > > > Thanks for the comment. Sounds like too much of a change in all fairness... I have actually made a tweak to my DB to allow multiple names, and storing them off the main table. my query then only needs to query the IDs, and then the second table to get the names. but i will keep the comments in mind and see how things go over the next while. > > As a side note, if i were to go down the "get doc from solr, modify, commit back to solr" is it really that simple? run a query on Solr, get the document, add the extra data, and insert back to solr? > > Thanks. > > --Tiernan > > On 19/10/2011 15:26, Dyer, James wrote: > > While Solr/Lucene can't support > > true document updates, there are 2 ways you might be able to work > > around this in your situation. > > > > > > > > > 1. If you store all of the fields, you can write something > > that will read back everything already indexed to the document, > > append whatever data you want, then write it back. This will > > increase index size and possibly make indexing too slow. On the > > other hand, it might be more efficient than requiring the database > > to return everything in order. > > > > > > > > > 2. You could store your data as multiple documents per id > > (pick something else as your unique id). Then use the grouping > > functionality to roll up on your unique id whenever you query. > > This will mean changes to your application, probably a bigger > > index, and likely somewhat slower querying. But the performance > > losses might be slight and this seems to me like it maybe would be > > a good solution in your case. Perhaps it would make it so you > > wouldn't have to entirely re-index each month or so. See > > http://wiki.apache.org/solr/FieldCollapsing for more information. > > > > > > > > > James Dyer > > > > > E-Commerce Systems > > > > > Ingram Content Group > > > > > (615) 213-4311 > > > > > > > > > > > > > -Original Message- > > > > > From: Tiernan OToole [mailto:lsmart...@gmail.com] > > > > > Sent: Wednesday, October 19, 2011 5:11 AM > > > > > To: solr-user@lucene.apache.org > > > > > Cc: Otis Gospodnetic > > > > > Subject: Re: Solr MultiValue Fields and adding values > > > > > > > > > > > > > I was hoping that wasent going to be the case... I ended up > > querying for > > > > > all unique IDs in the DB, and then querying for each unique > > ID and > > > > > getting all names, and then inserting them that way... Seems > > a lot > > > > > slower than in theory it really should be... > > > > > > > > > Thanks. > > > > > > > > > --Tiernan > > > > > > > > > On 18/10/2011 23:20, Otis Gospodnetic wrote: > > > > > > Hi, > > > > > > > > > > You'll need to construct the whole document and index it > > as such. You > > > > > can't append values to document fields. > > > > > > > > > > Otis > > > > > > > > > > > > > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - > > Nutch > > > > > > Lucene ecosystem search :: http://search-lucene.com/ > > > > > > > > > > > > > >> > > > > > >> From: Tiernan OToole > > > > > >> To: solr-user@lucene.apache.org > > > > > >> Sent: Tuesday, October 18, 2011 1
MultiValued fields and Facets...
Good morning all.
I am currently indexing about 11 million records, and would like to add
facating to the results page. I have tweaked the query string to include
facating, but i am not getting anything back.
an Example Query string (slightly modified) is as follows:
http://localhost:8080/solr/select?indent=on&version=2.2&q=*:*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=json&explainOther=&hl.fl=&facet=on&facet.field=Category&facet.field=Warehouse
the Category and Warehouse fields are multivalue fields...
the results i get are as follows:
"facet_counts":{
"facet_queries":{},
"facet_fields":{
"Category":[],
"Warehouse":[]},
"facet_dates":{},
"facet_ranges":{}}}
the data i am sending in has mutliple values for Category and Warehouse. I
did read that this was not available in Solr 1.3 or 1.4... I am currently
running Solr 3.4, and its not working... Would it work if i went to solr 4,
or am i doing something wrong here?
Thanks in advance.
--
Tiernan O'Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
Re: MultiValued fields and Facets...
Ok, so now i am getting something back, but still getting "odd" results...
I actually made a mistake in the first question... Category is MultiValued,
but Warehouse is not... So, when i run the query, you sugested, Category
comes back with facets and counts, which is one step closer to where i want
to be, but Warehouse, which as i mentioned is a single valued result, is
not... If i just facet on either Category, i get the results, but just
faceting on Warehouse still gets me nothing...
Very confused now...
--Tiernan
On Wed, Oct 26, 2011 at 12:53 PM, Erik Hatcher wrote:
> That URL has several oddities to it... empty fq and qt parameters. Try
> simply ?q=*:*&facet=on&facet.field=Category&facet.field=Warehouse and see if
> that helps.
>
>Erik
>
> On Oct 26, 2011, at 07:08 , Tiernan OToole wrote:
>
> > Good morning all.
> >
> > I am currently indexing about 11 million records, and would like to add
> > facating to the results page. I have tweaked the query string to include
> > facating, but i am not getting anything back.
> >
> > an Example Query string (slightly modified) is as follows:
> >
> >
> http://localhost:8080/solr/select?indent=on&version=2.2&q=*:*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=json&explainOther=&hl.fl=&facet=on&facet.field=Category&facet.field=Warehouse
> >
> > the Category and Warehouse fields are multivalue fields...
> >
> > the results i get are as follows:
> >
> > "facet_counts":{
> >"facet_queries":{},
> >"facet_fields":{
> > "Category":[],
> > "Warehouse":[]},
> >"facet_dates":{},
> >"facet_ranges":{}}}
> >
> > the data i am sending in has mutliple values for Category and Warehouse.
> I
> > did read that this was not available in Solr 1.3 or 1.4... I am currently
> > running Solr 3.4, and its not working... Would it work if i went to solr
> 4,
> > or am i doing something wrong here?
> >
> > Thanks in advance.
> >
> > --
> > Tiernan O'Toole
> > blog.lotas-smartman.net
> > www.geekphotographer.com
> > www.tiernanotoole.ie
>
>
--
Tiernan O'Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
Re: MultiValued fields and Facets...
I have figured out what was wrong... The field Warehouse was not marked as
indexed... It was being stored, but not indexed... It is now working as
expected.
Thanks.
--Tiernan
On Wed, Oct 26, 2011 at 1:01 PM, Tiernan OToole wrote:
> Ok, so now i am getting something back, but still getting "odd" results...
>
> I actually made a mistake in the first question... Category is
> MultiValued, but Warehouse is not... So, when i run the query, you
> sugested, Category comes back with facets and counts, which is one step
> closer to where i want to be, but Warehouse, which as i mentioned is a
> single valued result, is not... If i just facet on either Category, i get
> the results, but just faceting on Warehouse still gets me nothing...
>
> Very confused now...
>
> --Tiernan
>
>
> On Wed, Oct 26, 2011 at 12:53 PM, Erik Hatcher wrote:
>
>> That URL has several oddities to it... empty fq and qt parameters. Try
>> simply ?q=*:*&facet=on&facet.field=Category&facet.field=Warehouse and see
>> if that helps.
>>
>>Erik
>>
>> On Oct 26, 2011, at 07:08 , Tiernan OToole wrote:
>>
>> > Good morning all.
>> >
>> > I am currently indexing about 11 million records, and would like to add
>> > facating to the results page. I have tweaked the query string to include
>> > facating, but i am not getting anything back.
>> >
>> > an Example Query string (slightly modified) is as follows:
>> >
>> >
>> http://localhost:8080/solr/select?indent=on&version=2.2&q=*:*&fq=&start=0&rows=10&fl=*%2Cscore&qt=&wt=json&explainOther=&hl.fl=&facet=on&facet.field=Category&facet.field=Warehouse
>> >
>> > the Category and Warehouse fields are multivalue fields...
>> >
>> > the results i get are as follows:
>> >
>> > "facet_counts":{
>> >"facet_queries":{},
>> >"facet_fields":{
>> > "Category":[],
>> > "Warehouse":[]},
>> >"facet_dates":{},
>> >"facet_ranges":{}}}
>> >
>> > the data i am sending in has mutliple values for Category and
>> Warehouse. I
>> > did read that this was not available in Solr 1.3 or 1.4... I am
>> currently
>> > running Solr 3.4, and its not working... Would it work if i went to
>> solr 4,
>> > or am i doing something wrong here?
>> >
>> > Thanks in advance.
>> >
>> > --
>> > Tiernan O'Toole
>> > blog.lotas-smartman.net
>> > www.geekphotographer.com
>> > www.tiernanotoole.ie
>>
>>
>
>
> --
> Tiernan O'Toole
> blog.lotas-smartman.net
> www.geekphotographer.com
> www.tiernanotoole.ie
>
--
Tiernan O'Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
Solr, MultiValues and links...
Right, not sure how to ask this question, what the terminology, but hopefully my explaination will help... We are chucking data into solr for queries. i cant mention the exact data, but the closest thing i can think of is as follows: - Unique ID for the solr record (DB ID in this case) - A not so unique ID (say an ISBN number) - the name of the book (multiple names for this, each have a ISBN and a unique DB ID) - A Status of the book (books arent the correct term, but, say, a book has 4 editions but keeps the ISBN, we would have 4 names in Solr, each queryable. So, searching for the first edition title will return the correct ISBN). Anyway, what i want to be able to do is search for a single title (say Solr or Dummies) and find all instances of that title in index. but for each name, i want to be able to link the status of each title with each one. there are other "statuses" for each item also... Anyway, i had 2 ways of doing this: - the first way was using multi values, storing the names in a multi value list and also the statuses, but the linking dont seem to be working correctly... - the second way is storing each record uniqley, but with this i would need to run a second query to get all records for the ISBN... Any ideas which one i should be using? any tips on this? Thanks. -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie
Re: Solr, MultiValues and links...
That looks promising... Will look into that a bit more. --Tiernan On Sat, Nov 5, 2011 at 4:07 PM, Erick Erickson wrote: > Hmmm, MultiValues are guaranteed to be returned in the order they were > inserted, so you might be able to do the linking yourself given the > results. > > But have you considered grouping (aka field collapsing) on the ISBN number? > If you indexed each record uniquely, that might do what you need. > > Best > Erick > > On Fri, Nov 4, 2011 at 5:41 AM, Tiernan OToole > wrote: > > Right, not sure how to ask this question, what the terminology, but > > hopefully my explaination will help... > > > > We are chucking data into solr for queries. i cant mention the exact > data, > > but the closest thing i can think of is as follows: > > > > > > - Unique ID for the solr record (DB ID in this case) > > - A not so unique ID (say an ISBN number) > > - the name of the book (multiple names for this, each have a ISBN and a > > unique DB ID) > > - A Status of the book (books arent the correct term, but, say, a book > > has 4 editions but keeps the ISBN, we would have 4 names in Solr, each > > queryable. So, searching for the first edition title will return the > > correct ISBN). > > > > Anyway, what i want to be able to do is search for a single title (say > Solr > > or Dummies) and find all instances of that title in index. but for each > > name, i want to be able to link the status of each title with each one. > > there are other "statuses" for each item also... > > > > Anyway, i had 2 ways of doing this: > > > > > > - the first way was using multi values, storing the names in a multi > > value list and also the statuses, but the linking dont seem to be > working > > correctly... > > - the second way is storing each record uniqley, but with this i would > > need to run a second query to get all records for the ISBN... > > > > Any ideas which one i should be using? any tips on this? > > > > Thanks. > > > > > > -- > > Tiernan O'Toole > > blog.lotas-smartman.net > > www.geekphotographer.com > > www.tiernanotoole.ie > > > -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie
getting solr to expand Acronym
Dont know if this is posible, but i need to ask anyway... Say we have a list of Acronyms in a database (CD, DVD, CPU) and also a list of their not so short names (Compact Disk, Digital Versitile Disk, Central Processing Unit) but they are not linked in any particular way (lost of items, some with full names, some using anronyms), is it posible for Solr to figure out CD is an Acronym of Compact Disk? I know CD could also mean Central Data, or anything that beings with C and D, but is there a way to tell solr to look for items that not only match CD, but have words next to each other that begin with C and D... Another example i can think of is IBM: It could be International Business Machines, or Irish Business Machines, or Irish Banking Machines... So, would that be posible? -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie
Re: getting solr to expand Acronym
thanks for the replies... the problem with Synonyms is that they would need to be tracked... there could be new words entered that will need to be added to the list on a regular basis... @Otis: As for the option of a custom TokenFilter, how would that work? i have not coded anything into Solr or any custom TokenFilters my self... I am sure theres documentation on this, but how would you think this should work? Thanks. --Tiernan On Fri, Nov 11, 2011 at 9:01 PM, Brandon Ramirez < brandon_rami...@elementk.com> wrote: > Could this be simulated through synonyms? Could you define "CD" as a > synonym of "Compact Disc" or vice versa? I'm not sure if that would work, > just brainstorming here... > > > Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 > Software Engineer II | Element K | www.elementk.com > > > -Original Message- > From: Tiernan OToole [mailto:lsmart...@gmail.com] > Sent: Friday, November 11, 2011 5:10 AM > To: solr-user@lucene.apache.org > Subject: getting solr to expand Acronym > > Dont know if this is posible, but i need to ask anyway... Say we have a > list of Acronyms in a database (CD, DVD, CPU) and also a list of their not > so short names (Compact Disk, Digital Versitile Disk, Central Processing > Unit) but they are not linked in any particular way (lost of items, some > with full names, some using anronyms), is it posible for Solr to figure out > CD is an Acronym of Compact Disk? I know CD could also mean Central Data, > or anything that beings with C and D, but is there a way to tell solr to > look for items that not only match CD, but have words next to each other > that begin with C and D... Another example i can think of is IBM: It could > be International Business Machines, or Irish Business Machines, or Irish > Banking Machines... > > So, would that be posible? > > -- > Tiernan O'Toole > blog.lotas-smartman.net > www.geekphotographer.com > www.tiernanotoole.ie > -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie
Re: Sharing and performance testing question.
Thanks for the tips! will check out those links and see what i can find! On Wed, Aug 29, 2012 at 9:44 AM, Alexey Serba wrote: > > Any tips on load testing solr? Ideally we would like caching to not > effect > > the result as much as possible. > > 1. Siege tool > This is probably the simplest option. You can generate urls.txt file > and pass it to the tool. You should also capture server performance > (CPU, memory, qps, etc) using tools like newrelic, zabbix, etc. > > 2. SolrMeter > http://code.google.com/p/solrmeter/ > > 3. Solr benchmark module (not committed yet) > You to run complex benchmarks using different algorithms > * https://issues.apache.org/jira/browse/SOLR-2646 > * > http://searchhub.org/dev/2011/07/11/benchmarking-the-new-solr-near-realtime-improvements/ > -- Tiernan O'Toole blog.lotas-smartman.net www.geekphotographer.com www.tiernanotoole.ie
