Ephraim, I still can't view the document. Don't know if I'm doing something wrong, but I downloaded it and It appears to be empty.
Albert On 7 April 2011 09:32, Ephraim Ofir <ephra...@icq.com> wrote: > You can't view it online, but you should be able to download it from: > https://docs.google.com/leaf?id=0BwOEbnJ7oeOrNmU5ZThjODUtYzM5MS00YjRlLWI > 2OTktZTEzNDk1YmVmOWU4&hl=en&authkey=COGel4gP > > Enjoy, > Ephraim Ofir > > > -----Original Message----- > From: Jens Mueller [mailto:supidupi...@googlemail.com] > Sent: Thursday, April 07, 2011 8:30 AM > To: solr-user@lucene.apache.org > Subject: Re: Very very large scale Solr Deployment = how to do (Expert > Question)? > > Hello Ephraim, hello Lance, hello Walter, > > thanks for your replies: > > Ephraim, thanks very much for the further detailed explanation. I will > try > to setup a demo system in the next few days and use your advice. > LoadBalancers are an important aspect of your design. Can you recommend > one > LB specificallly? (I would be using haproxy.1wt.eu) . I think the Idea > with > uploading your document is very good. However Google-Docs seemed not be > be > working (at least for me with the docx format?), but maybe you can > simply > output the document as PDF and then I think Google Docs is working, so > all > the others can also have a look at your concept. The best approach would > be > if you could upload your advice directly somewhere to the solr wiki as > it is > really helpful.I found some other documents meanwhile, but yours is much > clearer and more complete, with the LBs and the Aggregators ( > http://lucene-eurocon.org/slides/Solr-In-The-Cloud_Mark-Miller.pdf) > > Lance, thanks I will have a look at what linkedin is doing. > > Walter, thanks for the advice: Well you are right, mentioning google. My > question was also to understand how such large systems like > google/facebook > are actually working. So my numbers are just theoretical and made up. My > system will be smaller, but I would be very happy to understand how > such > large systems are build and I think the approach Ephraim showd should be > working quite well at large scale. If you know a good documents (besides > the > bigtable research paper that I already know) that technically describes > how > google is working in detail that would be of great interest. You seem to > be > working for a company that handles large datasets. Does google use this > approach, sharing the index into N writers, and the procuded index is > then > replicated to N "read only searchers"? > > thank you all. > best regards > jens > > > > 2011/4/7 Walter Underwood <wun...@wunderwood.org> > >> The bigger answer is that you cannot get to this size by just > configuring >> Solr. You may have to invent a lot of stuff. Like all of Google. >> >> Where did you get these numbers? The proposed query rate is twice as > big as >> Google (Feb 2010 estimate, 34K qps). >> >> I work at MarkLogic, and we scale to 100's of terabytes, with fast > update >> and query rates. If you want a real system that handles that, you > might want >> to look at our product. >> >> wunder >> >> On Apr 6, 2011, at 8:06 PM, Lance Norskog wrote: >> >> > I would not use replication. LinkedIn consumer search is a flat > system >> > where one process indexes new entries and does queries > simultaneously. >> > It's a custom Lucene app called Zoie. Their stuff is on Github.. >> > >> > I would get documents to indexers via a multicast IP-based queueing >> > system. This scales very well and there's a lot of hardware support. >> > >> > The problem with distributed search is that it is a) inherently > slower >> > and b) has inherently more and longer jitter. The "airplane wing" >> > distribution of query times becomes longer and flatter. >> > >> > This is going to have to be a "federated" system, where the > front-end >> > app aggregates results rather than Solr. >> > >> > On Mon, Apr 4, 2011 at 6:25 PM, Jens Mueller > <supidupi...@googlemail.com> >> wrote: >> >> Hello Experts, >> >> >> >> >> >> >> >> I am a Solr newbie but read quite a lot of docs. I still do not >> understand >> >> what would be the best way to setup very large scale deployments: >> >> >> >> >> >> >> >> Goal (threoretical): >> >> >> >> A.) Index-Size: 1 Petabyte (1 Document is about 5 KB in Size) >> >> >> >> B) Queries: 100000 Queries/ per Second >> >> >> >> C) Updates: 100000 Updates / per Second >> >> >> >> >> >> >> >> >> >> Solr offers: >> >> >> >> 1.) Replication => Scales Well for B) BUT A) and C) are not >> satisfied >> >> >> >> >> >> 2.) Sharding => Scales well for A) BUT B) and C) are not > satisfied >> (=> As >> >> I understand the Sharding approach all goes through a central > server, >> that >> >> dispatches the updates and assembles the quries retrieved from the >> different >> >> shards. But this central server has also some capacity limits...) >> >> >> >> >> >> >> >> >> >> What is the right approach to handle such large deployments? I > would be >> >> thankfull for just a rough sketch of the concepts so I can >> experiment/search >> >> further... >> >> >> >> >> >> Maybe I am missing something very trivial as I think some of the > "Solr >> >> Users/Use Cases" on the homepage are that kind of large > deployments. How >> are >> >> they implemented? >> >> >> >> >> >> >> >> Thanky very much!!! >> >> >> >> Jens >> >> >> > >> >> >> >> >> > -- Albert Vila Puig <a...@imente.com> iMente.com <http://www.imente.com>
