Re: Dynamic collections in SolrCloud for log indexing

2012-12-27 Thread Otis Gospodnetic
Added https://issues.apache.org/jira/browse/SOLR-4237 Otis -- Performance Monitoring - http://sematext.com/spm/index.html Search Analytics - http://sematext.com/search-analytics/index.html On Tue, Dec 25, 2012 at 9:13 PM, Mark Miller wrote: > I've been thinking about aliases for a while as we

Re: Dynamic collections in SolrCloud for log indexing

2012-12-25 Thread Mark Miller
I've been thinking about aliases for a while as well. Seem very handy and fairly easy to implement. So far there has just always been higher priority things (need to finish collection api responses this week…) but this is something I'd def help work on. - Mark On Dec 25, 2012, at 1:49 AM, Otis

Re: Dynamic collections in SolrCloud for log indexing

2012-12-24 Thread Upayavira
This is precisely it. It is a 'collections alias', allowing you to group collections together into 'super-collections'. You add a new collection (made up of a core on n hosts) every day/week/month/whatever. When you do so, you add this collection to your super-collection. Many you do a quick audit

Re: Dynamic collections in SolrCloud for log indexing

2012-12-24 Thread Otis Gospodnetic
Hi, Right, this is not really about routing in ElasticSearch-sense. What's handy for indexing logs are index aliases which I thought I had added to JIRA a while back, but it looks like I have not. Index aliases would let you keep a "last 7 days" alias fixed while underneath you push and pop an

Re: Dynamic collections in SolrCloud for log indexing

2012-12-24 Thread Per Steffensen
I believe it is a misunderstandig to use custom routing (or sharding as Erick calls it) for this kind of stuff. Custom routing is nice if you want to control which slice/shard under a collection a specific document goes to - mainly to be able to control that two (or more) documents are indexed

Re: Dynamic collections in SolrCloud for log indexing

2012-12-23 Thread Erick Erickson
I think this is one of the primary use-cases for custom sharding. Solr 4.0 doesn't really lend itself to this scenario, but I _believe_ that the patch for custom sharding has been committed... That said, I'm not quite sure how you drop off the old shard if you don't need to keep old data. I'd gues

Dynamic collections in SolrCloud for log indexing

2012-12-21 Thread Upayavira
I'm working on a system for indexing logs. We're probably looking at filling one core every month. We'll maintain a short term index containing the last 7 days - that one is easy to handle. For the longer term stuff, we'd like to maintain a collection that will query across all the historic data,