Otis, Do you run 4bn docs SolrCloud or ElasticSearch or aware of somebody who do? 10.02.2013 4:54 пользователь "Otis Gospodnetic" <otis.gospodne...@gmail.com> написал:
> Exceeding 2B is no problem. But it won't happen in a single Lucene index > any time soon, so... > > Otis > Solr & ElasticSearch Support > http://sematext.com/ > On Feb 7, 2013 10:08 AM, "Mikhail Khludnev" <mkhlud...@griddynamics.com> > wrote: > > > Actually, I have a dream to exceed those two billions. It seems possible, > > to move to Vint in fileformat and change int docnums to longs in Lucene > > API. Does anyone know whether it's possible? > > And this question is not so esoteric if we are talking about SolrCloud, > > which can hold more that 2bn docs in few smaller shards. Any experience? > > > > > > On Thu, Feb 7, 2013 at 5:46 PM, Rafał Kuć <r....@solr.pl> wrote: > > > > > Hello! > > > > > > Right, my bad - ids are still using int32. However, that still > > > gives us 2,147,483,648 possible identifiers right per single index, > > > which is not close to the 13,5 millions mentioned in the first mail. > > > > > > -- > > > Regards, > > > Rafał Kuć > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - > > ElasticSearch > > > > > > > Rafal, > > > > > > > What about docnums, don't they are limited by int32 ? > > > > 07.02.2013 15:33 пользователь "Rafał Kuć" <r....@solr.pl> написал: > > > > > > >> Hello! > > > >> > > > >> Practically there is no limit in how many documents can be stored > in a > > > >> single index. In your case, as you are using Solr from 2011, there > is > > > >> a limitation regarding the number of unique terms per Lucene segment > > > >> ( > > > >> > > > > > > http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html#Limitations > > > >> ). > > > >> However I don't think you've hit that. Solr by itself doesn't remove > > > >> documents unless told to do so. > > > >> > > > >> Its hard to guess what can be the reason and as you said, you see > > > >> updates coming to your handler. Maybe new documents have the same > > > >> identifiers that the ones that are already indexed ? As I said, this > > > >> is only a guess and we would need to have more information. Are > there > > > >> any exceptions in the logs ? Do you run delete command ? Are your > > > >> index files changed ? How do you run commit ? > > > >> > > > >> -- > > > >> Regards, > > > >> Rafał Kuć > > > >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - > > > ElasticSearch > > > >> > > > >> > I have searched this forum but not yet found a definitive answer, > I > > > >> think the > > > >> > answer is "There is No Limit depends on server specification". But > > > never > > > >> the > > > >> > less I will say what I have seen and then ask the questions. > > > >> > > > >> > From scratch (November 2011) I have set up our SOLR which contains > > > data > > > >> from > > > >> > various sources, since March 2012 , the number of indexed records > > > (unique > > > >> > ID's) reached 13.5 million , which was to be expected. However for > > the > > > >> last > > > >> > 8 months the number of records in the index has not gone above > 13.5 > > > >> million, > > > >> > yet looking at the request handler outputs I can safely say at > least > > > >> > anywhere from 50 thousand to 100 thousand records are being > indexed > > > >> daily. > > > >> > So I am assuming that earlier records are being removed, and I do > > not > > > >> want > > > >> > that. > > > >> > > > >> > Question: If there is a limit to the number of records the index > can > > > >> store > > > >> > where do I find this and change it? > > > >> > Question: If there is no limit does anyone have any idea why for > the > > > last > > > >> > months the number has not gone beyond 13.5 million, I can safely > say > > > >> that at > > > >> > least 90% are new records. > > > >> > > > >> > thanks > > > >> > > > >> > macroman > > > >> > > > >> > > > >> > > > >> > -- > > > >> > View this message in context: > > > >> > > > > >> > > > > > > http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961.html > > > >> > Sent from the Solr - User mailing list archive at Nabble.com. > > > >> > > > >> > > > > > > > > > > > > -- > > Sincerely yours > > Mikhail Khludnev > > Principal Engineer, > > Grid Dynamics > > > > <http://www.griddynamics.com> > > <mkhlud...@griddynamics.com> > > >
