Re: Keeping a rolling window of indexes around solr

2013-05-29 Thread Erick Erickson
I suspect you're worrying about something you don't need to. At 1 insert every 30 seconds, and assuming 30,000,000 records will fit on a machine (I've seen this), you're talking 1,000,000 seconds worth of data on a single box! Or roughly 10,000 day's worth of data. Test, of course, YMMV. Or I'm mi

Re: Keeping a rolling window of indexes around solr

2013-05-28 Thread Saikat Kanjilal
Volume of data: 1 log insert every 30 seconds, queries done sporadically asynchronously every so often at a much lower frequency every few days Also the majority of the requests are indeed going to be within a splice of time (typically hours or at most a few days) Type of queries: Keyword or te

Re: Keeping a rolling window of indexes around solr

2013-05-28 Thread Chris Hostetter
: This is kind of the approach used by elastic search , if I'm not using : solrcloud will I be able to use shard aliasing, also with this approach : how would replication work, is it even needed? you haven't said much about hte volume of data you expect to deal with, nor have you really explai

RE: Keeping a rolling window of indexes around solr

2013-05-28 Thread Saikat Kanjilal
afa...@gmail.com > Date: Mon, 27 May 2013 16:17:12 -0400 > Subject: Re: Keeping a rolling window of indexes around solr > To: solr-user@lucene.apache.org > > But how is Hourglass going to help Solr? Or is it a portable implementation? > > Regards, >Alex. > Personal blog:

Re: Keeping a rolling window of indexes around solr

2013-05-27 Thread Alexandre Rafalovitch
But how is Hourglass going to help Solr? Or is it a portable implementation? Regards, Alex. Personal blog: http://blog.outerthoughts.com/ LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch - Time is the quality of nature that keeps events from happening all at once. Lately, it doesn't se

Re: Keeping a rolling window of indexes around solr

2013-05-27 Thread Otis Gospodnetic
Hi, SolrCloud now has the same index aliasing as Elasticsearch. I can't lookup the link now but Zoie from LinkedIn has Hourglass, which is uses for circular buffer sort of index setup if I recall correctly. Otis Solr & ElasticSearch Support http://sematext.com/ On May 24, 2013 10:26 AM, "Saikat

Re: Keeping a rolling window of indexes around solr

2013-05-24 Thread Saikat Kanjilal
This is kind of the approach used by elastic search , if I'm not using solrcloud will I be able to use shard aliasing, also with this approach how would replication work, is it even needed? Sent from my iPhone On May 24, 2013, at 12:00 PM, Alexandre Rafalovitch wrote: > Would collection alias

Re: Keeping a rolling window of indexes around solr

2013-05-24 Thread Alexandre Rafalovitch
Would collection aliasing help here? From Solr 4.2 release notes: Collection Aliasing. Got time based data? Want to re-index in a temporary collection and then swap it into production? Done. Stay tuned for Shard Aliasing. Regards, Alex. Personal blog: http://blog.outerthoughts.com/ LinkedIn: htt

RE: Keeping a rolling window of indexes around solr

2013-05-24 Thread Saikat Kanjilal
window. > Date: Fri, 24 May 2013 09:07:38 -0600 > From: elyog...@elyograg.org > To: solr-user@lucene.apache.org > Subject: Re: Keeping a rolling window of indexes around solr > > On 5/24/2013 8:56 AM, Shawn Heisey wrote: > > On 5/24/2013 8:25 AM, Saikat Kanjilal wrote: &g

Re: Keeping a rolling window of indexes around solr

2013-05-24 Thread Shawn Heisey
On 5/24/2013 8:56 AM, Shawn Heisey wrote: > On 5/24/2013 8:25 AM, Saikat Kanjilal wrote: >> Anyways would love to hear thoughts and usecases that are similar from the >> community. > > Your use-case sounds a lot like what loggly was doing back in 2010. > > http://loggly.com/videos/lucene-revolut

Re: Keeping a rolling window of indexes around solr

2013-05-24 Thread Shawn Heisey
On 5/24/2013 8:25 AM, Saikat Kanjilal wrote: > Anyways would love to hear thoughts and usecases that are similar from the > community. Your use-case sounds a lot like what loggly was doing back in 2010. http://loggly.com/videos/lucene-revolution-2010/

Keeping a rolling window of indexes around solr

2013-05-24 Thread Saikat Kanjilal
Hello Solr community folks, I am doing some investigative work around how to roll and manage indexes inside our solr configuration, to date I've come up with an architecture that separates a set of masters that are focused on writes and get replicated periodically and a set of slave shards stric