Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-22 Thread oakstream
Hi Otis, Yes, My data is in HBASE and I just need a fast Spatial Index where I can do lookups and then take the ID's back to HBASE to retrieve the results. HBASE doesn't support polygon searches that I'm aware of. You can do bounding box queries but that doesn't meet my requirements. I thought a

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-22 Thread Otis Gospodnetic
Hi david, I think (s)he is simply using HBase for storing the actual data. So after getting IDs from Solr (s)he will use them as keys to look up data in HBase. I think that is all. Otis -- Solr & ElasticSearch Support http://sematext.com/ On Mon, Jan 21, 2013 at 1:46 PM, David Smiley (@MIT

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-21 Thread oakstream
David, I appreciate your time. I'm going to take a crack at the Lucene sharded index approach and will let you know how I fare. Thanks again -- View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-Spatial-in-conjunction-with-HBASE-Hadoop-tp4034307p4035211.html Sent from

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-21 Thread David Smiley (@MITRE.org)
What good is a key-value store in the context of oakstream's question? ~ David - Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-Spatial-in-conjunction-with-HBASE-Hadoop-tp4034307p40

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-20 Thread ashok joshi
Have you looked at Oracle NoSQL Database http://www.oracle.com/us/products/database/nosql/overview/index.html, a scalable key-value store? Can Solr be integrated with it? Thanks and warm regards. ashok joshi oracle -- View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-19 Thread David Smiley (@MITRE.org)
oakstream wrote > Thanks guys! > David, > > In general and in your opinion would Lucene Spatial be the way to go to > index hundreds of terabytes of spatial data that continually grows. > Mostly point data, mostly structured, however, could be polygons. The > searches would be within or contains

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-18 Thread oakstream
Thanks guys! David, In general and in your opinion would Lucene Spatial be the way to go to index hundreds of terabytes of spatial data that continually grows. Mostly point data, mostly structured, however, could be polygons. The searches would be within or contains in a polygon. Do you have a

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-17 Thread David Smiley (@MITRE.org)
Hi Oakstream, Coincidentally I've been thinking of porting the geohash prefixtree intersection algorithm in Lucene 4 spatial to Accumulo (another big-table system like HBase). There's a decent chance it'll happen this year, I think. That doesn't help your need right now of course so go with Otis

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-17 Thread Otis Gospodnetic
You'd want to do your Solr spatial query, get IDs from the index, and then *after* that do a multi get against your HBase table with top N IDs from Solr's response and get thus get the data back to the caller. I don't know how fast multi gets are, what the limitations are, etc. Maybe somebody els

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-17 Thread oakstream
Thanks for your response! I appreciate it. There will be cases where I want to "AND or OR" the query between HBASE and Lucene. Would it make sense to custom code querying both repositories at the same time or sequentiallyOr are there any tools out there to do this? Basically I'm thinking

Re: Using Solr Spatial in conjunction with HBASE/Hadoop

2013-01-17 Thread Otis Gospodnetic
Hi, You certainly can do that, but you'll need to suck all data out of HBase and index it in Solr first. And then presumably you'll want to keep the 2 more or less in sync via incremental indexing. Maybe Lily project can help? If not, you'll have to write something that scans HBase and indexes,