Hi Dmitry, My index is not sharded, and since its size is so big, sharding won't help much on the paging issue. Do you know any API which can help read from lucene binary index directly? I will be nice if we can just scan through the docs directly.
Thanks! Ming- On Mon, May 6, 2013 at 3:33 AM, Dmitry Kan <solrexp...@gmail.com> wrote: > Are you doing it once? Is your index sharded? If so, can you ask each shard > individually? > Another way would be to do it on Lucene level, i.e. read from the binary > indices (API exists). > > Dmitry > > > On Mon, May 6, 2013 at 5:48 AM, Mingfeng Yang <mfy...@wisewindow.com> > wrote: > > > Dear Solr Users, > > > > Does anyone know what is the best way to iterate through each document > in a > > Solr index with billion entries? > > > > I tried to use select?q=*:*&start=xx&rows=500 to get 500 docs each time > > and then change start value, but it got very slow after getting through > > about 10 million docs. > > > > Thanks, > > Ming- > > >
