Why not use replication? Call it inexperience... We're really early into working with and fully understanding Solr and the best way to approach various issues. I did mention that this was a prototype and non-production code, so I'm covered, though :)
We'll take a look at the replication feature... Thanks! Bob Sandiford | Lead Software Engineer | SirsiDynix P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com www.sirsidynix.com > -----Original Message----- > From: Jonathan Rochkind [mailto:rochk...@jhu.edu] > Sent: Wednesday, November 10, 2010 3:26 PM > To: solr-user@lucene.apache.org > Subject: Re: Dynamic creating of cores in solr > > You could use the actual built-in Solr replication feature to > accomplish > that same function -- complete re-index to a 'master', and then when > finished, trigger replication to the 'slave', with the 'slave' being > the > live index that actually serves your applications. > > I am curious if there was any reason you chose to roll your own > solution > using JSolr and dynamic creation of cores, instead of simply using the > replication feature. Were there any downsides of using the replication > feature for this purpose that you amerliorated through your solution? > > Jonathan > > Bob Sandiford wrote: > > We also use SolrJ, and have a dynamically created Core capability - > where we don't know in advance what the Cores will be that we require. > > > > We almost always do a complete index build, and if there's a previous > instance of that index, it needs to be available during a complete > index build, so we have two cores per index, and switch them as > required at the end of an indexing run. > > > > Here's a summary of how we do it (we're in an early prototype / > implementation right now - this isn't production quality code - as you > can tell from our voluminous javadocs on the methods...) > > > > 1) Identify if the core exists, and if not, create it: > > > > /** > > * This method instantiates two SolrServer objects, solr and > indexCore. It requires that > > * indexName be set before calling. > > */ > > private void initSolrServer() throws IOException > > { > > String baseUrl = "http://localhost:8983/solr/";; > > solr = new CommonsHttpSolrServer(baseUrl); > > > > String indexCoreName = indexName + > SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX" > > String indexCoreUrl = baseUrl + indexCoreName; > > > > // Here we create two cores for the indexName, if they don't > already exist - the live core used > > // for searching and a second core used for indexing. After > indexing, the two will be switched so the > > // just-indexed core will become the live core. The way that > core swapping works, the live core will always > > // be named [indexName] and the indexing core will always be > named [indexname]_INDEX, but the > > // dataDir of each core will alternate between [indexName]_1 > and [indexName]_2. > > createCoreIfNeeded(indexName, indexName + "_1", solr); > > createCoreIfNeeded(indexCoreName, indexName + "_2", solr); > > indexCore = new CommonsHttpSolrServer(indexCoreUrl); > > } > > > > > > /** > > * Create a core if it does not already exists. Returns true if a > new core was created, false otherwise. > > */ > > private boolean createCoreIfNeeded(String coreName, String > dataDir, SolrServer server) throws IOException > > { > > boolean coreExists = true; > > try > > { > > // SolrJ provides no direct method to check if a core > exists, but getStatus will > > // return an empty list for any core that doesn't. > > CoreAdminResponse statusResponse = > CoreAdminRequest.getStatus(coreName, server); > > coreExists = > statusResponse.getCoreStatus(coreName).size() > 0; > > if(!coreExists) > > { > > // Create the core > > LOG.info("Creating Solr core: " + coreName); > > CoreAdminRequest.Create create = new > CoreAdminRequest.Create(); > > create.setCoreName(coreName); > > create.setInstanceDir("."); > > create.setDataDir(dataDir); > > create.process(server); > > } > > } > > catch (SolrServerException e) > > { > > e.printStackTrace(); > > } > > return !coreExists; > > } > > > > > > 2) Do the index, clearing it first if it's a complete rebuild: > > > > [snip] > > if (fullIndex) > > { > > try > > { > > indexCore.deleteByQuery("*:*"); > > } > > catch (SolrServerException e) > > { > > e.printStackTrace(); //To change body of catch > statement use File | Settings | File Templates. > > } > > } > > [snip] > > > > various logic, then (we submit batches of 100 : > > > > [snip] > > List<SolrInputDocument> docList = > b.getSolrInputDocumentList(); > > UpdateResponse rsp; > > try > > { > > rsp = indexCore.add(docList); > > rsp = indexCore.commit(); > > } > > catch (IOException e) > > { > > LOG.warn("Error commiting documents", e); > > } > > catch (SolrServerException e) > > { > > LOG.warn("Error commiting documents", e); > > } > > [snip] > > > > 3) optimize, then swap cores: > > > > private void optimizeCore() > > { > > try > > { > > indexCore.optimize(); > > } > > catch(SolrServerException e) > > { > > LOG.warn("Error while optimizing core", e); > > } > > catch(IOException e) > > { > > LOG.warn("Error while optimizing core", e); > > } > > } > > > > private void swapCores() > > { > > String liveCore = indexName; > > String indexCore = indexName + SolrConstants.SUFFIX_INDEX; // > SUFFIX_INDEX = "_INDEX" > > LOG.info("Swapping Solr cores: " + indexCore + ", " + > liveCore); > > CoreAdminRequest request = new CoreAdminRequest(); > > request.setAction(CoreAdminAction.SWAP); > > request.setCoreName(indexCore); > > request.setOtherCoreName(liveCore); > > try > > { > > request.process(solr); > > } > > catch (SolrServerException e) > > { > > e.printStackTrace(); > > } > > catch (IOException e) > > { > > e.printStackTrace(); > > } > > } > > > > > > And that's about it. > > > > You could adjust the above so there's only one core per index that > you want - if you don't do complete reindexes, and don't need the index > to always be searchable. > > > > Hope that helps... > > > > > > Bob Sandiford | Lead Software Engineer | SirsiDynix > > P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com > > www.sirsidynix.com > > > > > > > >> -----Original Message----- > >> From: Nizan Grauer [mailto:niz...@yahoo-inc.com] > >> Sent: Tuesday, November 09, 2010 3:36 AM > >> To: solr-user@lucene.apache.org > >> Subject: Dynamic creating of cores in solr > >> > >> Hi, > >> > >> I'm not sure this is the right mail to write to, hopefully you can > help > >> or direct me to the right person > >> > >> I'm using solr - one master with 17 slaves in the server and using > >> solrj as the java client > >> > >> Currently there's only one core in all of them (master and slaves) - > >> only the cpaCore. > >> > >> I thought about using multi-cores solr, but I have some problems > with > >> that. > >> > >> I don't know in advance which cores I'd need - > >> > >> When my java program runs, I call for documents to be index to a > >> certain url, which contains the core name, and I might create a url > >> based on core that is not yet created. For example: > >> > >> Calling to index - http://localhost:8080/cpaCore - existing core, > >> everything as usual > >> Calling to index - http://localhost:8080/newCore - server realizes > >> there's no core "newCore", creates it and indexes to it. After that > - > >> also creates the new core in the slaves > >> Calling to index - http://localhost:8080/newCore - existing core, > >> everything as usual > >> > >> What I'd like to have on the server side to do is realize by itself > if > >> the cores exists or not, and if not - create it > >> > >> One other restriction - I can't change anything in the client side - > >> calling to the server can only make the calls it's doing now - for > >> index and search, and cannot make calls for cores creation via the > >> CoreAdminHandler. All I can do is something in the server itself > >> > >> What can I do to get it done? Write some RequestHandler? > >> REquestProcessor? Any other option? > >> > >> Thanks, nizan > >> > > > > > >
