-- Jack Krupansky
-----Original Message----- From: Ali Nazemian
Sent: Wednesday, July 30, 2014 2:53 AM To: solr-user@lucene.apache.org Subject: Re: integrating Accumulo with solr Sure, Thank you very much for your guide. I think I am not that kind of gunslinger and probably I will go for another NoSQL that can be integrated with solr/elastic search much easier:) Best regards. On Sun, Jul 27, 2014 at 5:02 PM, Jack Krupansky <j...@basetechnology.com> wrote:
Right, and that's exactly what DataStax Enterprise provides (at great engineering effort!) - synchronization of database updates and searchindexing. Sure, you can do it as well, but that's a significant engineering challenge with both sides of the equation, and not a simple "plug and play"configuration setting by writing a simple "connector." But, hey, if you consider yourself one of those "true hard-core gunslingers" then you'll be able to code that up in a weekend without any of our assistance, right? In short, synchronizing two data stores is a real challenge. Yes, it isdoable, but... it is non-trivial. Especially if both stores are distributedclusters. Maybe now you can guess why the Sqrrl guys went the Lucene route instead of Solr. I'm certainly not suggesting that it can't be done. Just highlighting the challenge of such a task. Just to be clear, you are referring to "sync mode" and not mere "ETL", which people do all the time with batch scripts, Java extraction and ingestion connectors, and cron jobs. Give it a shot and let us know how it works out. -- Jack Krupansky -----Original Message----- From: Ali Nazemian Sent: Sunday, July 27, 2014 1:20 AM To: solr-user@lucene.apache.org Subject: Re: integrating Accumulo with solr Dear Jack, Hi, One more thing to mention: I dont want to use solr or lucence for indexing accumulo or full text search inside that. I am looking for have both in a sync mode. I mean import some parts of data to solr for indexing. For this purpose probably I need something like trigger in RDBMS, I have to define something (probably with accumulo iterator) to import to solr on inserting new data. Regards. On Fri, Jul 25, 2014 at 12:59 PM, Ali Nazemian <alinazem...@gmail.com> wrote: Dear Jack,Actually I am going to do benefit-cost analysis for in-house developement or going for sqrrl support. Best regards. On Thu, Jul 24, 2014 at 11:48 PM, Jack Krupansky <j...@basetechnology.com > wrote: Like I said, you're going to have to be a real, hard-core gunslinger todo that well. Sqrrl uses Lucene directly, BTW: "Full-Text Search: Utilizing open-source Lucene and custom indexing methods, Sqrrl Enterprise users can conduct real-time, full-text search across data in Sqrrl Enterprise." See: http://sqrrl.com/product/search/Out of curiosity, why are you not using that integrated Lucene support ofSqrrl Enterprise? -- Jack Krupansky -----Original Message----- From: Ali Nazemian Sent: Thursday, July 24, 2014 3:07 PM To: solr-user@lucene.apache.org Subject: Re: integrating Accumulo with solr Dear Jack, Thank you. I am aware of datastax but I am looking for integrating accumulo with solr. This is something like what sqrrl guys offer. Regards. On Thu, Jul 24, 2014 at 7:27 PM, Jack Krupansky <j...@basetechnology.com > wrote: If you are not a "true hard-core gunslinger" who is willing to dive inand integrate the code yourself, instead you should give serious consideration to a product such as DataStax Enterprise that fully integrates and packages a NoSQL database (Cassandra) and Solr for search. The security aspects are still a work in progress, but certainly headed in the right direction. And it has Hadoop and Spark integration as well. See: http://www.datastax.com/what-we-offer/products-services/ datastax-enterprise -- Jack Krupansky -----Original Message----- From: Ali Nazemian Sent: Thursday, July 24, 2014 10:30 AM To: solr-user@lucene.apache.org Subject: Re: integrating Accumulo with solr Thank you very much. Nice Idea but how can Solr and Accumulo can be synchronized in this way? I know that Solr can be integrated with HDFS and also Accumulo works on the top of HDFS. So can I use HDFS as integration point? I mean set Solr to use HDFS as a source of documents as well as the destination of documents. Regards. On Thu, Jul 24, 2014 at 4:33 PM, Joe Gresock <jgres...@gmail.com> wrote: Ali,Sounds like a good choice. It's pretty standard to store the primary storage id as a field in Solr so that you can search the full text in Solr and then retrieve the full document elsewhere. I would recommend creating a document structure in Solr with whatever fields you want indexed (most likely as text_en, etc.), and then store a "string" field named "content_id", which would be the Accumulo row id that you look up with a scan. One caveat -- Accumulo will be protected at the cell level, but if you need your Solr search results to be protected by complex authorization strings similar to Accumulo, you will need to write your own QParserPlugin and use post filtering: http://java.dzone.com/articles/custom-security-filtering-solr The code you see in that article is written for an earlier version of Solr, but it's not too difficult to adjust it for the latest (we've done so in our project). Once you've implemented this, you would store an "authorizations" string field in each Solr document, and pass in theauthorizations that the user has access to in the fq parameter of everyquery. It's also not too bad to write something that parses the Accumuloauthorizations string (like A&B&(C|D|E|F)) and interpret it accordinglyin the QParserPlugin.This will give you true row level security in Solr and Accumulo, and itperforms quite well in Solr. Let me know if you have any other questions. Joe On Thu, Jul 24, 2014 at 4:07 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > Dear Joe, > Hi, > I am going to store the crawl web pages in accumulo as the main storage> part of my project and I need to give these data to solr for > indexing> and > user searches. I need to do some social and web analysis on my data > as well> as having some security features. Therefore accumulo is my choice > for> the > database part and for index and search I am going to use Solr. Would > you > please guide me through that? > > > > On Thu, Jul 24, 2014 at 1:28 AM, Joe Gresock <jgres...@gmail.com> wrote: >> > We store data in both Solr and Accumulo -- do you have more > > detailsabout > > what kind of data and indexing you want? Is there a reason you're > thinking > > of using both databases in particular? > > > > > > On Wed, Jul 23, 2014 at 5:17 AM, Ali Nazemian < alinazem...@gmail.com> > > wrote: > > > > > Dear All, > > > Hi, > > > I was wondering is there anybody out there that tried to > > > integrate Solr> > > with Accumulo? I was thinking about using Accumulo on top of > > > HDFS> > > and > > using > > > Solr to index data inside Accumulo? Do you have any idea how can > > > I > > > do > > such > > > integration? > > > > > > Best regards. > > > > > > -- > > > A.Nazemian > > > > > > > > > > > -- > > I know what it is to be in need, and I know what it is to have > > plenty. I > > have learned the secret of being content in any and every > > situation,> > whether well fed or hungry, whether living in plenty or in want. > > I> > can > do > > all this through him who gives me strength. *-Philippians 4:12-13* > > > > > > -- > A.Nazemian > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation,whether well fed or hungry, whether living in plenty or in want. I cando all this through him who gives me strength. *-Philippians 4:12-13*-- A.Nazemian-- A.Nazemian-- A.Nazemian-- A.Nazemian
--A.Nazemian
