On Thu, Mar 18, 2010 at 8:45 AM, Moritz Maedler <m...@moritz-maedler.de>wrote:
> Hi guys! > > Thanks alot for your suggestions and help - I really appreciate that! > As we need e.g. the the price for sorting I think it must be in the index? > Thus, I'm not shure that a key-value-store is the thing we are looking for > as we > need a good searchengine. > Currently we are using serveral indices items, bids, etc. I think that ZOIE > looks > kind of interesting - I will check that out. > > BTW do/can you recommend any "newbe" literature or resources for getting a > bit > more familiar w/ Solr/Lucene ? > Probably the best way how to get your questions answered it participating in mail lists. Documentation and wiki pages as well. There are also some books: check Lucene in Action (http://www.manning.com/hatcher3/) and Solr 1.4 Enterprise Search Server ( http://www.packtpub.com/solr-1-4-enterprise-search-server). Other books about Lucene and Solr are WIP but unfortunately not released yet. > > Thanks a lot again guys! > > > -----Ursprüngliche Nachricht----- > Von: Lance Norskog [mailto:goks...@gmail.com] > Gesendet: Donnerstag, 18. März 2010 00:38 > An: solr-user@lucene.apache.org > Betreff: Re: Will Solr fit our needs? > > Another option is the ExternalFileField: > > > http://www.lucidimagination.com/search/document/CDRG_ch04_4.4.4?q=ExternalFileField > > This lets you store the current prices for all items in a separate > file. You can only use it in a function query, that is. But it does > allow you to maintain one Solr index, which is very very worthwhile. > > On Wed, Mar 17, 2010 at 4:19 AM, Geert-Jan Brits <gbr...@gmail.com> wrote: > > If you dont' plan on filtering/ sorting and/or faceting on fast-changing > > fields it would be better to store them outside of solr/lucene in my > > opinion. > > > > If you must: for indexing-performance reasons you will probably end up > with > > maintaining seperate indices (1 for slow-changing/static fields and 1 for > > fast-changing-fields) . > > You frequently commit the fast-changing -index to incorporate the changes > > in current_price. Afterwards you have 2 options I believe: > > > > 1. use parallelreader to query the seperate indices directly. Afaik, this > is > > not (completely) integrated in Solr... I wouldn't recommend it. > > 2. after you commit the fast-changing-index, merge with the static-index. > > You're left with 1 fresh index, which you can push to your slave-servers. > > (all this in regular interverals) > > > > Disadvatages: > > - In any way, you must be very careful with maintaining multiple parallel > > indexes with the purpose of treating them as one. For instance document > > inserts must be done exactly in the same order, otherwise the indices go > > 'out-of-sync' and are unusable. > > - higher maintenance > > - there is always a time-window in which the current_price values are > stale. > > If that's within reqs that's ok. > > > > The other path, which I recommend, would be to store the current_price > > outside of solr (like you're currently doing) but instead of using a > > relational db, try looking into persistent key-value stores. Many of them > > exist and a lot of progress has been made in the last couple of years. > For > > simple key-lookups (what you need as far as I can tell) they really blow > > every relational db out of the water (considering the same hardware of > > course) > > > > We're currently using Tokyo Cabinet with the server-frontend Tokyo Tyrant > > and seeing almost a 5x increased in lookup performance compared to our > > previous kv-store memcachedDB which is based on BerkelyDB. Memcachedb was > > already several times faster than our mysql-setup (although not optimally > > tuned) . > > > > to sum things up: use the best tools for what they were meant to do. > > > > - index/search --> solr/ lucene without a doubt. > > > > - kv-lookup --> consensus is still forming, and a lot of players (with a > lot > > of different types of functionality) but if all you need is simple > > key-value-lookup, I would go for Tokyo Cabinet (TC) / Tyrant at the > moment. > > Please note that TC and competitors aren't just some code/ hobby > projects > > but are usually born out of a real need at huge websites / social > networks > > such as TC which is born from mixi (big social network in Japan) . So at > > least you're in good company.. > > > > for kv-stores I would suggest to begin your research at: > > > http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ > > (beginning > > 2009) > > http://randomfoo.net/2009/04/20/some-notes-on-distributed-key-stores(half > > 2009) > > and get a feel of the kv-playing field. > > > > Hope this (pretty long) post helps, > > Geert-Jan > > > > > > 2010/3/17 Krzysztof Grodzicki <krzysztof.grodzi...@iterate.pl> > > > >> Hi Mortiz, > >> > >> You can take a look on the project ZOIE - > >> http://code.google.com/p/zoie/. I think it's that what are you looking > >> for. > >> > >> br > >> Krzysztof > >> > >> On Wed, Mar 17, 2010 at 9:49 AM, Moritz Mädler <m...@moritz-maedler.de> > >> wrote: > >> > Hi List, > >> > > >> > we are running a marketplace which has about a comparable > functionality > >> like ebay (auctions, fixed-price items etc). > >> > The items are placed on the market by users who want to sell their > goods. > >> > > >> > Currently we are using Sphinx as an indexing engine, but, as Sphinx > >> returns only document ids we have to make a > >> > database-query to fetch the data to display. This massively decreases > >> performance as we have to do two requests to > >> > display data. > >> > > >> > I heard that Solr is able to return a complete dataset and we hope a > >> switch to Solr can boost perfomance. > >> > A critical question is left and i was not able to find a solution for > it > >> in the docs: Is it possible to update attributes directly in the > >> > index? > >> > An example for better illustration: > >> > We have an index which holds all the auctions (containing auctionid, > >> auction title) with its current prices(field: current_price). When a > user > >> places a new bid, > >> > is it possible to update the attribute 'current_price' directly in the > >> index so that we can fetch the current_price from Solr and not from the > >> database? > >> > > >> > I hope you understood my problem. It would be kind if someone can > point > >> me to the right direction. > >> > > >> > Thanks alot! > >> > > >> > Moritz > >> > > > > > > -- > Lance Norskog > goks...@gmail.com > >
