Upgrading Solr Cloud from 4.7.1 to 4.10
Hi, we are planning for a Solr Cloud upgrade from 4.7.1 to 4.10. I have identified the following steps for the upgrade 1. Update solrconfig.xml to be compliant with 4.10. 2. Update the required jar files and war files from Solr 4.10 distribution. 3. Verified we are using Java 1.7. Now comes the question of data - index, do I need to recreate the indexes from scratch, basically trunk and load or is there some other way to upgrade the indexes? Any guidance you can provide will be very helpful. Nitin
Re: Upgrading Solr Cloud from 4.7.1 to 4.10
Thanks Shawn, will try and it out without re-indexing and see how that goes. Nitin On Tue, Sep 23, 2014 at 9:00 AM, Shawn Heisey wrote: > On 9/23/2014 8:40 AM, Nitin Agarwal wrote: > > Hi, we are planning for a Solr Cloud upgrade from 4.7.1 to 4.10. I have > > identified the following steps for the upgrade > > > > 1. Update solrconfig.xml to be compliant with 4.10. > > 2. Update the required jar files and war files from Solr 4.10 > distribution. > > 3. Verified we are using Java 1.7. > > > > Now comes the question of data - index, do I need to recreate the indexes > > from scratch, basically trunk and load or is there some other way to > > upgrade the indexes? > > All Solr 4.x versions will read the index from previous 4.x and 3.x > versions. It is recommended that you completely reindex for any > upgrade, but it should not be absolutely necessary. If that doesn't > work, it's either a bug or a result of config/schema changes. > > Thanks, > Shawn > >
SOLR query - restrict access to user documents
Hi, I have a question around SOLR query, I am trying to restrict access to SOLR data. We are running SOLR 4.7.1, and wish to expose the query capabilities to our customers for the data that belongs to them. Specifically "/select", with default configuration is the only Request Handler that customers can access. explicit 10 text The custom API that fronts SOLR, will inject appropriate restriction into the "q" param e.g. q=customerNumber:123 or append to "q" param q= AND customerNumber:123, before sending the request to the "/select" handler. This works fine, however, I want to know if there is a way customer can override these restrictions? If so what can I do to prevent that? So far I have come across facet.mincount as one potential concern where by customer can see data that they should not, e.g. /select?q= AND customerNumber:123&facet=true&facet.field=customerName&rows=0&*facet.mincount=0* will return those customer names as well that do not belong to customerNumber 123. Are there any other gotchas that I should know? Thanks for your time and help, Nitin
Re: SOLR query - restrict access to user documents
Thanks for the info Jorge, I will look into invariants, good pointer. My API, forces the rows to be a max of 500. If the user specifies more than 500 docs, then we modify the rows param to be 500. On Tue, Oct 7, 2014 at 3:31 PM, Jorge Luis Betancourt Gonzalez < jlbetanco...@uci.cu> wrote: > I see you’re defining a default value for “rows” this could be overridden > on the request, and requesting a lot of documents from solr can stress out > your server/cluster, of course if the client in question has that many > documents. if this is a fixed value and the clients can’t request more > documents, then I’ll consider moving this into the invariants section > ensuring that no matter what this value can’t be changed by the request. > Some time ago I had a similar use case, we wanted to expose Solr to the > clients and eventually we faced problems where some clients requested “all > of his documents” in one request stressing out our cluster in the end we > wrote a custom SearchComponent to set max values (instead of a fixed value > specified on invariants) for the rows and start parameters (actually this > component those a little more as we add some limitations to each type of > client, defining some constrains as how many documents. i.e. data points > can be requested, etc.). > > Hope it helps, > > On Oct 7, 2014, at 11:37 AM, Nitin Agarwal <2nitinagar...@gmail.com> > wrote: > > > Hi, I have a question around SOLR query, I am trying to restrict access > to > > SOLR data. > > > > We are running SOLR 4.7.1, and wish to expose the query capabilities to > our > > customers for the data that belongs to them. Specifically "/select", with > > default configuration is the only Request Handler that customers can > > access. > > > > > > > > explicit > > 10 > > text > > > > > > > > The custom API that fronts SOLR, will inject appropriate restriction > > into the "q" param e.g. q=customerNumber:123 or > > append to "q" param q= AND customerNumber:123, before > > sending the request to the "/select" handler. > > > > This works fine, however, > > > > I want to know if there is a way customer can override these > restrictions? > > > > If so what can I do to prevent that? > > > > So far I have come across facet.mincount as one potential concern > > where by customer can see data that they should not, e.g. > > > > /select?q= AND > > > customerNumber:123&facet=true&facet.field=customerName&rows=0&*facet.mincount=0* > > > > will return those customer names as well that do not belong to > > customerNumber 123. > > > > Are there any other gotchas that I should know? > > > > Thanks for your time and help, > > > > Nitin > > Concurso "Mi selfie por los 5". Detalles en > http://justiciaparaloscinco.wordpress.com >
Solr 4.3.1 - query does not return documents, just numFounds, 2 shards, replication Factor 1
Hi, I am using Solr 4.3.1 with 2 Shards and replication factor of 1, running on apache tomcat 7.0.42 with external zookeeper 3.4.5. When I query "select?q=*:*" I only get the number of documents found, but no actual document. When I query with rows=0, I do get correct count of documents in the index. Faceting queries as well as group by queries also work with rows=0. However, when rows is not equal to 0 I do not get any documents. When I query the index I see that a query is being sent to both shards, and subsequently I see a query being sent with just ids, however, after that query returns I do not see any documents back. Not sure what do I need to change, please help. Thanks, Nitin
Re: Solr 4.3.1 - query does not return documents, just numFounds, 2 shards, replication Factor 1
Jason, all my fields are set with stored=ture and indexed = true, and I used select?q=*:*&fl=*,score but still I get the same response * 0 138 *,score *:* * Here is what my schema looks like * ** * Nitin On Mon, Jul 29, 2013 at 9:38 AM, Jason Hellman < jhell...@innoventsolutions.com> wrote: > Nitin, > > You need to ensure the fields you wish to see are marked stored="true" in > your schema.xml file, and you should include fields in your fl= parameter > (fl=*,score is a good place to start). > > Jason > > On Jul 29, 2013, at 8:08 AM, Nitin Agarwal <2nitinagar...@gmail.com> > wrote: > > > Hi, I am using Solr 4.3.1 with 2 Shards and replication factor of 1, > > running on apache tomcat 7.0.42 with external zookeeper 3.4.5. > > > > When I query "select?q=*:*" > > > > I only get the number of documents found, but no actual document. When I > > query with rows=0, I do get correct count of documents in the index. > > Faceting queries as well as group by queries also work with rows=0. > > However, when rows is not equal to 0 I do not get any documents. > > > > When I query the index I see that a query is being sent to both shards, > and > > subsequently I see a query being sent with just ids, however, after that > > query returns I do not see any documents back. > > > > Not sure what do I need to change, please help. > > > > Thanks, > > Nitin > >
Re: Solr 4.3.1 - query does not return documents, just numFounds, 2 shards, replication Factor 1
Jack, I checked my solrconfig.xml for "/select" RequestHandler, it had rows=10, I changed it 20, uploaded the configs to zookeeper, restarted Tomcat, didn't work. Then removed rows all together from defaults, uploaded the configs to zookeeper and restarted Tomcat, but it still does not work. Here is "/select" RequestHandler snippet explicit tn_lookup_key_id when I ran the query /select?q=*:*&fl=*,score I still get 0 72 *,score *:* On Mon, Jul 29, 2013 at 10:41 AM, Jack Krupansky wrote: > Check the "/select" request handler in solrconfig. See if it defaults > "start" or "rows". start is the initial document number (e.g., 1), and rows > is the number of rows to actually return in the response (nothing to do > with numFound). The internal Solr default is rows=10, but you can set it to > 20, 50, 100, or whatever, but DO NOT set it to 0 unless you just want the > header without any actual documents. > > -- Jack Krupansky > > -Original Message- From: Nitin Agarwal > Sent: Monday, July 29, 2013 11:49 AM > To: solr-user@lucene.apache.org > Subject: Re: Solr 4.3.1 - query does not return documents, just numFounds, > 2 shards, replication Factor 1 > > > Jason, all my fields are set with stored=ture and indexed = true, and I > used > > select?q=*:*&fl=*,score > > but still I get the same response > > * > > > 0 > 138 > > *,score > *:* > > > > * > > > Here is what my schema looks like > > * > > multiValued="false" /> > stored="true" required="false" /> > stored="true" required="false" /> > required="false" /> >** > indexed="true" stored="true" required="true" /> > * > > > > > Nitin > > > On Mon, Jul 29, 2013 at 9:38 AM, Jason Hellman < > jhell...@innoventsolutions.com**> wrote: > > Nitin, >> >> You need to ensure the fields you wish to see are marked stored="true" in >> your schema.xml file, and you should include fields in your fl= parameter >> (fl=*,score is a good place to start). >> >> Jason >> >> On Jul 29, 2013, at 8:08 AM, Nitin Agarwal <2nitinagar...@gmail.com> >> wrote: >> >> > Hi, I am using Solr 4.3.1 with 2 Shards and replication factor of 1, >> > running on apache tomcat 7.0.42 with external zookeeper 3.4.5. >> > >> > When I query "select?q=*:*" >> > >> > I only get the number of documents found, but no actual document. When I >> > query with rows=0, I do get correct count of documents in the index. >> > Faceting queries as well as group by queries also work with rows=0. >> > However, when rows is not equal to 0 I do not get any documents. >> > >> > When I query the index I see that a query is being sent to both shards, >> and >> > subsequently I see a query being sent with just ids, however, after that >> > query returns I do not see any documents back. >> > >> > Not sure what do I need to change, please help. >> > >> > Thanks, >> > Nitin >> >> >> >
Re: Solr 4.3.1 - query does not return documents, just numFounds, 2 shards, replication Factor 1
Hoss, you rock! That was the issue, I changed tn_lookup_key_id, which was my unique key field, to string and reloaded the index and it works. Jason, Jack and Hoss, thanks for your help. Nitin On Mon, Jul 29, 2013 at 12:22 PM, Chris Hostetter wrote: > > : Here is what my schema looks like > > what is your uniqueKey field? > > I'm going to bet it's "tn_lookup_key_id" and i'm going to bet your > "lowercase" fieldType has an interesting analyzer on it. > > you are probably hitting a situation where the analyzer you have on your > uniqueKey field is munging the values in such a way that when the > coordinator node decides which N docs to include in the response, > and then asks the various shards to give it those specific N > docs, those subsequent field fetching queries fail because of an > analysis mismatch. > > you need to keep your uniqueKeyField simple -- i strongly recommend a > basic StrField. If you also want to do "lowercase" lookups on your key > field, index it redundently in a second field. > > > : * > :: multiValued="false" /> > :: stored="true" required="false" /> > :: stored="true" required="false" /> > :stored="true" > : required="false" /> > : ** : indexed="true" stored="true" required="true" /> > : * > > -Hoss >
Re: Solr 4.3.1 - query does not return documents, just numFounds, 2 shards, replication Factor 1
Erick, I had typed tn_lookup_key_id as lowercase and it was defined as Nitin On Mon, Jul 29, 2013 at 1:23 PM, Erick Erickson wrote: > Nitin: > > What was your tn_lookup_key_id field definition when things didn't work? > The stock lowercase is KeywordTokenizerFactory+LowerCaseFilterFactory > and if this leads to mis-matches as Hoss outlined, it'd surprise me so I > need > to file it away in my list of things not to do. > > Thanks, > Erick > > On Mon, Jul 29, 2013 at 3:01 PM, Nitin Agarwal <2nitinagar...@gmail.com> > wrote: > > Hoss, you rock! > > > > That was the issue, I changed tn_lookup_key_id, which was my unique key > > field, to string and reloaded the index and it works. > > > > Jason, Jack and Hoss, thanks for your help. > > > > Nitin > > > > > > On Mon, Jul 29, 2013 at 12:22 PM, Chris Hostetter > > wrote: > > > >> > >> : Here is what my schema looks like > >> > >> what is your uniqueKey field? > >> > >> I'm going to bet it's "tn_lookup_key_id" and i'm going to bet your > >> "lowercase" fieldType has an interesting analyzer on it. > >> > >> you are probably hitting a situation where the analyzer you have on your > >> uniqueKey field is munging the values in such a way that when the > >> coordinator node decides which N docs to include in the response, > >> and then asks the various shards to give it those specific N > >> docs, those subsequent field fetching queries fail because of an > >> analysis mismatch. > >> > >> you need to keep your uniqueKeyField simple -- i strongly recommend a > >> basic StrField. If you also want to do "lowercase" lookups on your key > >> field, index it redundently in a second field. > >> > >> > >> : * > >> :>> : multiValued="false" /> > >> :>> : stored="true" required="false" /> > >> :>> : stored="true" required="false" /> > >> :>> stored="true" > >> : required="false" /> > >> : ** >> : indexed="true" stored="true" required="true" /> > >> : * > >> > >> -Hoss > >> >
