We don't run one ourselves at Sematext, but know of people who do have large ES clusters, one with > 10B docs.
Otis -- Solr & ElasticSearch Support http://sematext.com/ On Mon, Feb 11, 2013 at 8:41 AM, Mikhail Khludnev < mkhlud...@griddynamics.com> wrote: > Otis, > Do you run 4bn docs SolrCloud or ElasticSearch or aware of somebody who do? > 10.02.2013 4:54 пользователь "Otis Gospodnetic" < > otis.gospodne...@gmail.com> > написал: > > > Exceeding 2B is no problem. But it won't happen in a single Lucene index > > any time soon, so... > > > > Otis > > Solr & ElasticSearch Support > > http://sematext.com/ > > On Feb 7, 2013 10:08 AM, "Mikhail Khludnev" <mkhlud...@griddynamics.com> > > wrote: > > > > > Actually, I have a dream to exceed those two billions. It seems > possible, > > > to move to Vint in fileformat and change int docnums to longs in Lucene > > > API. Does anyone know whether it's possible? > > > And this question is not so esoteric if we are talking about SolrCloud, > > > which can hold more that 2bn docs in few smaller shards. Any > experience? > > > > > > > > > On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote: > > > > > > > Hello! > > > > > > > > Right, my bad - ids are still using int32. However, that still > > > > gives us 2,147,483,648 possible identifiers right per single index, > > > > which is not close to the 13,5 millions mentioned in the first mail. > > > > > > > > -- > > > > Regards, > > > > Rafał Kuć > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - > > > ElasticSearch > > > > > > > > > Rafal, > > > > > > > > > What about docnums, don't they are limited by int32 ? > > > > > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал: > > > > > > > > >> Hello! > > > > >> > > > > >> Practically there is no limit in how many documents can be stored > > in a > > > > >> single index. In your case, as you are using Solr from 2011, there > > is > > > > >> a limitation regarding the number of unique terms per Lucene > segment > > > > >> ( > > > > >> > > > > > > > > > > http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations > > > > >> ). > > > > >> However I don't think you've hit that. Solr by itself doesn't > remove > > > > >> documents unless told to do so. > > > > >> > > > > >> Its hard to guess what can be the reason and as you said, you see > > > > >> updates coming to your handler. Maybe new documents have the same > > > > >> identifiers that the ones that are already indexed ? As I said, > this > > > > >> is only a guess and we would need to have more information. Are > > there > > > > >> any exceptions in the logs ? Do you run delete command ? Are your > > > > >> index files changed ? How do you run commit ? > > > > >> > > > > >> -- > > > > >> Regards, > > > > >> Rafał Kuć > > > > >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - > > > > ElasticSearch > > > > >> > > > > >> > I have searched this forum but not yet found a definitive > answer, > > I > > > > >> think the > > > > >> > answer is "There is No Limit depends on server specification". > But > > > > never > > > > >> the > > > > >> > less I will say what I have seen and then ask the questions. > > > > >> > > > > >> > From scratch (November 2011) I have set up our SOLR which > contains > > > > data > > > > >> from > > > > >> > various sources, since March 2012 , the number of indexed > records > > > > (unique > > > > >> > ID's) reached 13.5 million , which was to be expected. However > for > > > the > > > > >> last > > > > >> > 8 months the number of records in the index has not gone above > > 13.5 > > > > >> million, > > > > >> > yet looking at the request handler outputs I can safely say at > > least > > > > >> > anywhere from 50 thousand to 100 thousand records are being > > indexed > > > > >> daily. > > > > >> > So I am assuming that earlier records are being removed, and I > do > > > not > > > > >> want > > > > >> > that. > > > > >> > > > > >> > Question: If there is a limit to the number of records the index > > can > > > > >> store > > > > >> > where do I find this and change it? > > > > >> > Question: If there is no limit does anyone have any idea why for > > the > > > > last > > > > >> > months the number has not gone beyond 13.5 million, I can safely > > say > > > > >> that at > > > > >> > least 90% are new records. > > > > >> > > > > >> > thanks > > > > >> > > > > >> > macroman > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > > >> > View this message in context: > > > > >> > > > > > >> > > > > > > > > > > http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html > > > > >> > Sent from the Solr - User mailing list archive at Nabble.com. > > > > >> > > > > >> > > > > > > > > > > > > > > > > > -- > > > Sincerely yours > > > Mikhail Khludnev > > > Principal Engineer, > > > Grid Dynamics > > > > > > <http://www.griddynamics.com> > > > <mkhlud...@griddynamics.com> > > > > > >
