Mohit, just curious. Have you considered just encrypting the volume where there index is stored? It will be done at the OS level and performance wise it will be much better and easier to manage. Would it address your requirement? Or is it purely academic exercise for you.
Bosco From: "Sidana, Mohit" <mohit.sid...@wolterskluwer.com> Reply-To: <solr-user@lucene.apache.org> Date: Monday, June 20, 2016 at 7:42 AM To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> Subject: Encryption to Solr indexes – Using Custom Codec Hello, As Part of my studies I am exploring the solutions which can be used for Lucene/Solr Index encryption. I found the patch open on Apache JIRA - Codec for index-level encryption (LUCENE-6966). https://issues.apache.org/jira/browse/LUCENE-6966 and I am currently trying to test this Custom codec with Solr to perform secure search over some sensitive records. I've decided to follow the path described in Solr wiki, setting up Simple Text Codec and further tried to use Encrypted codec Source. Here are the additional details. I've created a basic jar file out of this source code (Build it as a jar from Eclipse using Maven Plugin). The Solr installation I'm using to test this is the Solr 6.0.0 unzipped, and started via its embedded Jetty server and using the single core. I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib In: [$SolrDir]\Solr\ My_Core \conf\solrconfig.xml I've added the following lines: <lib dir="./lib/" regex=".*\.jar"/> <codecFactory class="solr.SchemaCodecFactory"/> Then in the schema.xml file, I've declared some field and field Types that should use this codec: <field name="City" type="strings"/> <field name="Country" type="strings"/> <field name="Name" type="strings"/> <field name="Payment" type="strings"/> <field name="Price" type="tlongs"/> <field name="Product" type="strings"/> <field name="State" type="strings"/> <!-- set the per field posting format as below --> <fieldType name="strings" class="solr.StrField" sortMissingLast="true" docValues="true" multiValued="true" postingsFormat="EncryptedLucene50"/> <fieldType name="tlongs" class="solr.TrieLongField" positionIncrementGap="0" docValues="true" multiValued="true" precisionStep="8" postingsFormat="EncryptedLucene50"/> I'm pretty sure I've followed all the steps described in Solr Wiki; however, when I actually try to use custom codec implementation (named "Encrypted Codec") to index some sample CSV data using simple post tool java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update -jar post.jar Sales.csv and I have also tried doing the same with SolrJ but I have faced the same error. SolrClient server = new HttpSolrClient("http://localhost:8983/solr/My_Core "); SolrInputDocument doc = new SolrInputDocument(); doc.addField("id", "1234"); doc.addField("name", "A lovely summer holiday"); try { server.add(doc); server.commit(); System.out.println("Document added!"); } catch (SolrServerException | IOException e) { e.printStackTrace(); } } } I get the attached errors in Solr log. org.apache.solr.common.SolrException: Exception writing document id b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3 to the index; possible analysis error. at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:936) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1091) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:714) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48) at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93) at org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:391) at org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:43) at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:360) at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2033) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1488) at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169) ... 58 more Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/codecs/encrypted/DummyCipherFactory at org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEncryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60Codec.java:67) at org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fieldsConsumer(EncryptedLucene50PostingsFormat.java:479) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196) at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:423) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2815) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:619) at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1453) at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Caused by: java.lang.ClassNotFoundException: org.apache.lucene.codecs.encrypted.DummyCipherFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 22 more >From the output I assume that Solr is not picking up the jar with this custom >codec, and I don't get why? What I find strange is that, I have also tried to run this codec with Lucene and eclipse by writing some code for indexing and searching and same codec jar is working just fine. System.out.println("Indexing to directory '" + indexPath + "'..."); Directory dir = FSDirectory.open(Paths.get(indexPath)); Analyzer analyzer = new StandardAnalyzer(); IndexWriterConfig iwc = new IndexWriterConfig(analyzer); // set encrypted codec //This encrypted Codec is being used for indexing iwc.setCodec(new DummyEncryptedLucene60Codec()); if (create) { // Create a new index in the directory, removing any // previously indexed documents: iwc.setOpenMode(OpenMode.CREATE); } else { // Add new documents to an existing index: iwc.setOpenMode(OpenMode.CREATE_OR_APPEND); } IndexWriter writer = new IndexWriter(dir, iwc); indexDocs(writer, docDir); writer.close(); I'm guessing that As Solr uses Java SPI options to load this codec which loads things differently, and there's something missing...? My question is, what might I be doing wrong here or what's missing so that codec jar can be picked up by Solr? Thanks.
