Yes, that is our approach. We did try deleting a day's worth of data at a time, and that resulted in OOM as well.
Thanks Vinay On 14 April 2014 00:27, Furkan KAMACI <furkankam...@gmail.com> wrote: > Hi; > > I mean you can divide the range (i.e. one week at each delete instead of > one month) and try to check whether you still get an OOM or not. > > Thanks; > Furkan KAMACI > > > 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <poth...@gmail.com>: > > > Aman, > > Yes - Will do! > > > > Furkan, > > How do you mean by 'bulk delete'? > > > > -Thanks > > Vinay > > > > > > On 12 April 2014 14:49, Furkan KAMACI <furkankam...@gmail.com> wrote: > > > > > Hi; > > > > > > Do you get any problems when you index your data? On the other hand > > > deleting as bulks and reducing the size of documents may help you not > to > > > hit OOM. > > > > > > Thanks; > > > Furkan KAMACI > > > > > > > > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <amantandon...@gmail.com>: > > > > > > > Vinay please share your experience after trying this solution. > > > > > > > > > > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <poth...@gmail.com> > > > wrote: > > > > > > > > > The query is something like this: > > > > > > > > > > > > > > > *curl -H 'Content-Type: text/xml' --data > '<delete><query>param1:(val1 > > > OR > > > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO > > > > > 1385164800000]</query></delete>' > > > > > 'http://host:port/solr/coll-name1/update?commit=true'* > > > > > > > > > > Trying to restrict the number of documents deleted via the date > > > > parameter. > > > > > > > > > > Had not tried the "distrib=false" option. I could give that a try. > > > Thanks > > > > > for the link! I will check on the cache sizes and autowarm values. > > Will > > > > try > > > > > and disable the caches when I am deleting and give that a try. > > > > > > > > > > Thanks Erick and Shawn for your inputs! > > > > > > > > > > -Vinay > > > > > > > > > > > > > > > > > > > > On 11 April 2014 15:28, Shawn Heisey <s...@elyograg.org> wrote: > > > > > > > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote: > > > > > > > > > > > >> When we tried to delete the data through a query - say 1 > > day/month's > > > > > worth > > > > > >> of data. But after deleting just 1 month's worth of data, the > > master > > > > > node > > > > > >> is going out of memory - heap space. > > > > > >> > > > > > >> Wondering is there any way to incrementally delete the data > > without > > > > > >> affecting the cluster adversely. > > > > > >> > > > > > > > > > > > > I'm curious about the actual query being used here. Can you > share > > > it, > > > > or > > > > > > a redacted version of it? Perhaps there might be a clue there? > > > > > > > > > > > > Is this a fully distributed delete request? One thing you might > > try, > > > > > > assuming Solr even supports it, is sending the same delete > request > > > > > directly > > > > > > to each shard core with distrib=false. > > > > > > > > > > > > Here's a very incomplete list about how you can reduce Solr heap > > > > > > requirements: > > > > > > > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems# > > > > > > Reducing_heap_requirements > > > > > > > > > > > > Thanks, > > > > > > Shawn > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > With Regards > > > > Aman Tandon > > > > > > > > > >
