Re: Different number of replicas for different shards

2019-06-30 Thread Nawab Zada Asad Iqbal
@Erick Actually, i thought further and realized what you were saying. I am hoping to rely on the murmur3 hash of the routing key to find the destination shard. On Sun, Jun 30, 2019 at 3:32 AM Nawab Zada Asad Iqbal wrote: > Hi Erick, > > I plan to use the composite-id routing. And

Re: Different number of replicas for different shards

2019-06-30 Thread Nawab Zada Asad Iqbal
ve more _replicas_ for those shards with the ADDREPLICA > collections API command. > > > > On Jun 29, 2019, at 7:00 AM, Shawn Heisey wrote: > > > > On 6/29/2019 12:23 AM, Nawab Zada Asad Iqbal wrote: > >> is it possible to specify different number of replicas for di

Different number of replicas for different shards

2019-06-28 Thread Nawab Zada Asad Iqbal
Hi, is it possible to specify different number of replicas for different shards? i.e if I expect some shard to get more queries , i can add more replicas to that shard alone, instead of adding replicas for all the shards. Thanks Nawab

Re: Solr metric for finding number of merges in progress

2018-09-07 Thread Nawab Zada Asad Iqbal
Actually, i found this after posting; (still if someone has more to offer, please reply) https://lucene.apache.org/solr/guide/7_0/metrics-reporting.html#index-merge-metrics On Fri, Sep 7, 2018 at 12:13 PM Nawab Zada Asad Iqbal wrote: > Hi, > > Does Solr expose a metric to find the

Solr metric for finding number of merges in progress

2018-09-07 Thread Nawab Zada Asad Iqbal
Hi, Does Solr expose a metric to find the number of segment merges which are in progress? Thanks Nawab

Re: Internal details on how ADDREPLICA executes?

2018-08-02 Thread Nawab Zada Asad Iqbal
impacts on > your particular installation by issuing that collections API > command. > > Best, > Erick > > On Thu, Aug 2, 2018 at 12:20 PM, Nawab Zada Asad Iqbal > wrote: > > Hi, > > > > I am considering using SolrCloud and enable autoADDREPLICA. > >

Internal details on how ADDREPLICA executes?

2018-08-02 Thread Nawab Zada Asad Iqbal
Hi, I am considering using SolrCloud and enable autoADDREPLICA. I am curious on how long does it take for SolrCloud to setup the replica. Before the new replica can start to serve queries, it needs to copy all the documents in the current leader and also index whatever new traffic is arriving. How

Re: SolrCloud: Different replicationFactor for different shards in same collection

2018-07-31 Thread Nawab Zada Asad Iqbal
x27;re using the default composite id routing having one shard get > substantially more queries than the others is unexpected. > > If you're using implicit routing then it's perfectly understandable. > > Best, > Erick > > On Tue, Jul 31, 2018 at 4:12 PM, Nawab Za

SolrCloud: Different replicationFactor for different shards in same collection

2018-07-31 Thread Nawab Zada Asad Iqbal
Hi, I am looking at Solr 7.x and couldn't find an answer in the documentation. Is it possibly to specify different replicationFactor for different shards in same collection? E.g. if a certain shard is receiving more queries than rest of the collection I would like to add more replicas for it to h

Re: Mysterious Solr crash

2018-06-04 Thread Nawab Zada Asad Iqbal
any other meaningful part and will copy here. On Sun, Jun 3, 2018 at 6:07 PM, Shawn Heisey wrote: > On 6/3/2018 7:52 AM, Nawab Zada Asad Iqbal wrote: > >> I am running a batch indexing job and Solr core mysteriously shut down >> without any particular error. How can I inve

Mysterious Solr crash

2018-06-03 Thread Nawab Zada Asad Iqbal
Good morning I am running a batch indexing job and Solr core mysteriously shut down without any particular error. How can I investigate this? I am focusing on the line which mentions "Shutting down CoreContainer instance". There are errors soon after that, but they seemed to be caused by the core

Query logs when query reached the solr server

2018-05-31 Thread Nawab Zada Asad Iqbal
Hi, 1. Is there a way to enable query log when the query first reached the solr server? 2. In recent Solr versions, there is a NOW value in the query log. Is it correct to assume that this is the time when query arrived on that server? Thanks Nawab

Solr admin Segments page legend

2018-05-17 Thread Nawab Zada Asad Iqbal
Hi, Solr has a nice segments visualization at [core_name]/segments , but I am wondering what the colors mean? Gray color is probably deleted documents., But I couldn't guess the significance of pink color: Thanks Nawab

Re: Solr 7.3 debug/explain with boost applied

2018-04-24 Thread Nawab Zada Asad Iqbal
I didn't know you can add boosts like that (&boost=2 ). Are you boosting on a field or document by using that syntax? On Sun, Apr 22, 2018 at 10:51 PM, Ryan Yacyshyn wrote: > Hi all, > > When viewing the explain under debug=true in Solr 7.3.0 using > the edismax query parser with a boost, I only

Re: Solr document routing using composite key

2018-04-07 Thread Nawab Zada Asad Iqbal
e number of docs on various shards be off by > more than 2-3% when enough docs are indexed to be statistically valid. > > Best, > Erick > > On Fri, Mar 16, 2018 at 5:34 AM, Shawn Heisey wrote: > > On 3/6/2018 11:53 AM, Nawab Zada Asad Iqbal wrote: > >> > >> I

Solr Metrics mismatch the query logs

2018-03-28 Thread Nawab Zada Asad Iqbal
Hi, I gather solr metrics from the following URL for my request handler. [solr-host]/solr/core1/admin/mbeans?stats=true&wt=json Specifically, there is a requests value in the stats map for each handler. I use this value to calculate the requests which arrived between two successive intervals

Solr document routing using composite key

2018-03-06 Thread Nawab Zada Asad Iqbal
Hi solr community: I have been thinking to use composite key for my next project iteration and tried it today to see how it distributes the documents. Here is a gist of my code: https://gist.github.com/niqbal/3e293e2bcb800d6912a250d914c9d478 I have 117 shards and i tried to use document ids fro

Re: Are flushed (but not committed yet) segments mutable?

2018-02-10 Thread Nawab Zada Asad Iqbal
ogs are around, then Solr replays the documents > since that point. > > Best, > Erick > > On Fri, Feb 9, 2018 at 8:07 PM, Nawab Zada Asad Iqbal > wrote: > > Hi, > > > > When a segment is flushed to disk because it is exceeding available > memory, > > i

Are flushed (but not committed yet) segments mutable?

2018-02-09 Thread Nawab Zada Asad Iqbal
Hi, When a segment is flushed to disk because it is exceeding available memory, is it sill updated when new documents are added? I also read somewhere that a segment is not committed even if it is flushed. How is a flushed-but-not-committed segment different from a committed segment? For example,

Using 'learning to rank' with user specific features

2018-01-23 Thread Nawab Zada Asad Iqbal
hi, I am going through learning to rank examples in Solr7. In the examples, the features are part of the searched document. Can I use solr's learning to rank system if my features are user specific? e.g., if searching for products, i want to rank some products higher if they have been used by cur

Substitute for SmartChineseWordTokenFilterFactory in Solr7

2018-01-23 Thread Nawab Zada Asad Iqbal
Hi, I used to use SmartChineseSentenceTokenizerFactory and SmartChineseWordTokenFilterFactory in Solr4 for analyzing Chinese text. In solr7, I found that SmartChineseSentenceTokenizerFactory is replaced with HMMChineseTokenizerFactory, but i cannot locate a Chinese Word Tokenizer. Is that not need

Re: trivia question: why q=*:* doesn't return same result as q.alt=*:*

2018-01-17 Thread Nawab Zada Asad Iqbal
Chris / Hoss Thanks for the detailed explanation. Erick Erickson's explanation made sense to me but it didn't explain the part why the fields are different for 'hello' vs '*:*' . I had never paid much attention the parser part of query handling and so far focused only on the field definitions. I

Re: trivia question: why q=*:* doesn't return same result as q.alt=*:*

2018-01-06 Thread Nawab Zada Asad Iqbal
Thanks everyone, that was a very informative thread. One more curiosity: why are different set of fields being used based on the query string:- http://localhost:8983/solr/filesearch/select?fq=id:1193&; q=*:*&debugQuery=true - parsedquery: "+DisjunctionMaxQuery((user_email:*:* | user_name:*:

Re: CommonGramsFilter

2018-01-05 Thread Nawab Zada Asad Iqbal
Actually, i have found that it is *not* mandatory to use phrase search with CommonGramsFilter . PS: i had some other code change (which is unnecessary) which was causing the above behavior. On Thu, Jan 4, 2018 at 6:56 PM, Nawab Zada Asad Iqbal wrote: > After some debugging, it seems that

Re: trivia question: why q=*:* doesn't return same result as q.alt=*:*

2018-01-05 Thread Nawab Zada Asad Iqbal
HI Erik Hatcher Yes, i am using dismax. But dismax allows *:* for q.alt ,which also seems like inconsistency. On Thu, Jan 4, 2018 at 5:53 PM, Erik Hatcher wrote: > defType=??? Probably dismax. It doesn’t do *:* like edismax or lucene. > > > On Jan 4, 2018, at 20:39, Nawab Zad

Re: CommonGramsFilter

2018-01-04 Thread Nawab Zada Asad Iqbal
erly pass the stopwords to the CommonGramsFilter? On Thu, Jan 4, 2018 at 6:08 PM, Nawab Zada Asad Iqbal wrote: > Hi, > > I am looking at this documentation and wondering if it would be better to > optionally skip indexing of original stopwords. > > https://lucene.apache.org

CommonGramsFilter

2018-01-04 Thread Nawab Zada Asad Iqbal
Hi, I am looking at this documentation and wondering if it would be better to optionally skip indexing of original stopwords. https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html# FilterDescriptions-CommonGramsFilter http://localhost:8983/solr/filesearch/select?q=not%20to%20or%20be&

Re: trivia question: why q=*:* doesn't return same result as q.alt=*:*

2018-01-04 Thread Nawab Zada Asad Iqbal
ou attach &debug=query? I'm curious how > the parsed queries differ. > > On Jan 4, 2018 15:14, "Nawab Zada Asad Iqbal" wrote: > > > Hi, > > > > In my SearchHandler solrconfig, i have q.alt=*:* . This allows me to run > > queries which only have `fq`

trivia question: why q=*:* doesn't return same result as q.alt=*:*

2018-01-04 Thread Nawab Zada Asad Iqbal
Hi, In my SearchHandler solrconfig, i have q.alt=*:* . This allows me to run queries which only have `fq` filters and no `q`. If I remove q.alt from the solrconfig and specify `q=*:*` in the query parameters, it does not give any results. I also tried `q=*` but of no avail. Is there some good re

Re: Small Tokenization issue

2018-01-03 Thread Nawab Zada Asad Iqbal
t; Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 3 Jan 2018, at 21:04, Nawab Zada Asad Iqbal wrote: > > > > Hi, > > > > So, I have a string for indexing: >

Small Tokenization issue

2018-01-03 Thread Nawab Zada Asad Iqbal
Hi, So, I have a string for indexing: abc - def (notice the space on either side of hyphen) which is being processed with this filter-list:- I get two shingle tokens at the e

Re: fq: OR operator (sometimes) not working

2017-12-27 Thread Nawab Zada Asad Iqbal
ions and associated types. > > Because it worked for me just fine in the simple case, so some > non-obvious combination of things is causing this. Since neither of us > know _what_, include everything ;).. > > Best, > Erick > > On Wed, Dec 27, 2017 at 4:45 PM, Nawab Zada

Re: fq: OR operator (sometimes) not working

2017-12-27 Thread Nawab Zada Asad Iqbal
not what I wanted, you just reiterated the _input_. > I asked for the results when adding &debug=query to the string so you > can see the parsed query. > You should see something similar to: > > "parsed_filter_queries":["file_type:jpg file_type:jpeg"]}

Re: fq: OR operator (sometimes) not working

2017-12-27 Thread Nawab Zada Asad Iqbal
ates from 7.0.1). Regards Nawab On Wed, Dec 27, 2017 at 3:54 PM, Erick Erickson wrote: > What does adding &debug=query show in the two cases? > > Best, > Erick > > On Wed, Dec 27, 2017 at 3:40 PM, Nawab Zada Asad Iqbal > wrote: > > Hi, > > > > Are the fol

fq: OR operator (sometimes) not working

2017-12-27 Thread Nawab Zada Asad Iqbal
Hi, Are the following two queries equal: In my understanding, I can specify the arguments the operator once in the {} local parameter syntax (example 1) or I can interleave OR between different clauses (example 2). But I am getting my result in the second case only. What am I doing wrong? This

Re: Do i need to reindex after changing similarity setting

2017-11-30 Thread Nawab Zada Asad Iqbal
This JIRA also throws some light. There is a discussion of encoding norm during indexing. The contributor eventually comments that "norms" encoded by different similarity are compatible to each other. On Thu, Nov 30, 2017 at 5:12 PM, Nawab Zada Asad Iqbal wrote: > Hi Walter, &

Re: Do i need to reindex after changing similarity setting

2017-11-30 Thread Nawab Zada Asad Iqbal
the collection allows it. " On Wed, Nov 22, 2017 at 9:11 AM, Nawab Zada Asad Iqbal wrote: > Thanks Walter > > On Mon, Nov 20, 2017 at 4:59 PM Walter Underwood > wrote: > >> Similarity is query time. >> >> wunder >> Walter Underwood >> w

Re: Do i need to reindex after changing similarity setting

2017-11-22 Thread Nawab Zada Asad Iqbal
Thanks Walter On Mon, Nov 20, 2017 at 4:59 PM Walter Underwood wrote: > Similarity is query time. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > > On Nov 20, 2017, at 4:57 PM, Nawab Zada Asad Iqbal > wro

Re: Solr7: Very High number of threads on aggregator node

2017-11-22 Thread Nawab Zada Asad Iqbal
Rick Your suspicion is correct. I mostly reused my config from solr4 except where it was deprecated or obsoleted and I switched to the newer configs: Having said that I couldn't find any new query related settings which can impact us, since most of our queries dont use fancy new features. I could

Do i need to reindex after changing similarity setting

2017-11-20 Thread Nawab Zada Asad Iqbal
Hi, I want to switch to Classic similarity instead of BM25 (default in solr7). Do I need to reindex all cores after this? Or is it only a query time setting? Thanks Nawab

Re: Solr7: Very High number of threads on aggregator node

2017-11-20 Thread Nawab Zada Asad Iqbal
t for the process, rather it is somewhere else. Thanks Nawab On Sat, Nov 18, 2017 at 10:22 AM, Rick Leir wrote: > Nawab > You probably need to share the relevant config to get an answer to this. > Cheers -- Rick > > On November 17, 2017 2:19:03 PM EST, Nawab Zada Asad Iqbal

Solr7: Very High number of threads on aggregator node

2017-11-17 Thread Nawab Zada Asad Iqbal
Hi, I have a sharded solr7 cluster and I am using an aggregator node (which has no data/index of its own) to distribute queries and aggregate results from the shards. I am puzzled that when I use solr7 on the aggregator node, then number of threads shoots up to 32000 on that host and then the proc

Re: Solr7: Bad query throughput around commit time

2017-11-12 Thread Nawab Zada Asad Iqbal
tened by the fact that enabling docValues will > cause your index size on disk to grow. Paradoxically that will > actually _lower_ the size of the JVM heap requirements. Essentially > the additional size on disk is the serialized structure that would > have to be built in the JVM. S

Re: Solr7: Bad query throughput around commit time

2017-11-11 Thread Nawab Zada Asad Iqbal
~248 gb Nawab On Sat, Nov 11, 2017 at 2:41 PM Kevin Risden wrote: > > One machine runs with a 3TB drive, running 3 solr processes (each with > one core as described above). > > How much total memory on the machine? > > Kevin Risden > > On Sat, Nov 11, 2017 at 1:0

Re: Solr7: Bad query throughput around commit time

2017-11-11 Thread Nawab Zada Asad Iqbal
_ as > you can demonstrate improvements. You've changed enough things here > that untangling which one is the culprit will be hard. You want the > JVM to have as little memory as possible, unfortunately that's > something you figure out by experimentation. > > Best, &

Solr7: Bad query throughput around commit time

2017-11-09 Thread Nawab Zada Asad Iqbal
Hi, I am committing every 5 minutes using a periodic cron job "curl http://localhost:8984/solr/core1/update?commit=true";. Besides this, my app doesn't do any soft or hard commits. With Solr 7 upgrade, I am noticing that query throughput plummets every 5 minutes - probably when the commit happens

Re: update document stuck on: java.net.SocketInputStream.socketRead0

2017-11-03 Thread Nawab Zada Asad Iqbal
to at least improve my performance. PS: I have also posted this question here: https://stackoverflow.com/questions/47098816/solr-jetty-9-webserver-sending-a-ton-of-socket-timeouts Thanks Nawab On Thu, Oct 26, 2017 at 7:03 PM, Nawab Zada Asad Iqbal wrote: > Hi, > > After Solr 7 upgrade,

Re: SOLR-11504: Provide a config to restrict number of indexing threads

2017-11-01 Thread Nawab Zada Asad Iqbal
ting on my client side - on the other side of the wire. Thanks Nawab On Wed, Nov 1, 2017 at 7:11 AM, Shawn Heisey wrote: > On 10/31/2017 4:57 PM, Nawab Zada Asad Iqbal wrote: > >> I hit this issue https://issues.apache.org/jira/browse/SOLR-11504 while >> migrating to solr6 and

SOLR-11504: Provide a config to restrict number of indexing threads

2017-10-31 Thread Nawab Zada Asad Iqbal
Hi, I hit this issue https://issues.apache.org/jira/browse/SOLR-11504 while migrating to solr6 and locally working around it in Lucene code. I am thinking to fix it properly and hopefully patch back to Solr. Since, Lucene code does not want to keep any such config, I am thinking to use a counting

update document stuck on: java.net.SocketInputStream.socketRead0

2017-10-26 Thread Nawab Zada Asad Iqbal
Hi, After Solr 7 upgrade, I am realizing that my '/update' request is sometimes getting stuck on this:- - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.read(byte[], int, in

Measuring time spent in analysis and writing to index

2017-10-19 Thread Nawab Zada Asad Iqbal
Hi, I want to analyze the time spent in different stages during add/update document request. E.g., I want to compare time spend in analysis vs writing to Lucene index. Does Solr provide any such thing? I have looked at [core/admin/mbeans?stats=true&wt=json&indent=true] which provides overall stat

Re: 3 color jvm memory usage bar

2017-10-19 Thread Nawab Zada Asad Iqbal
lable resource and the > darker shade is used. > > Best, > Erick > > On Thu, Oct 19, 2017 at 7:27 AM, Nawab Zada Asad Iqbal > wrote: > > Good morning, > > > > > > What do the 3 colors mean in this bar on Solr dashboard page? (please see > > attached) : > > > > > > Regards > > Nawab >

3 color jvm memory usage bar

2017-10-19 Thread Nawab Zada Asad Iqbal
Good morning, What do the 3 colors mean in this bar on Solr dashboard page? (please see attached) : Regards Nawab

Re: [EXTERNAL] Re: OOM during indexing with 24G heap - Solr 6.5.1

2017-10-17 Thread Nawab Zada Asad Iqbal
Randy That is one issue, i don't know if it fixes everything for you or not. However, Lucene doesn't put a limit on number of incoming requests and after https://issues.apache.org/jira/browse/LUCENE-6659 , solr has no way (i don't know at least) to limit threads. So if you have ton of parallel upd

Re: solr 7.0: What causes the segment to flush

2017-10-17 Thread Nawab Zada Asad Iqbal
5-589-9269 > www.lucidworks.com > Twitter http://twitter.com/lucidworks > LinkedIn: https://www.linkedin.com/in/sarkaramrit2 > > On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal > wrote: > > > Hi, > > > > I have tuned (or tried to tune) my settings to onl

solr 7.0: What causes the segment to flush

2017-10-17 Thread Nawab Zada Asad Iqbal
Hi, I have tuned (or tried to tune) my settings to only flush the segment when it has reached its maximum size. At the moment,I am using my application with only a couple of threads (i have limited to one thread for analyzing this scenario) and my ramBufferSizeMB=2 (i.e. ~20GB). With this, I

Re: Solr test runs: test skipping logic

2017-10-06 Thread Nawab Zada Asad Iqbal
Thanks Chris, That very likely is the reason. I had noticed the seed and realized that it will be controlling the random input generation for the tests to make failures reproducible. However, i didn't consider that it can also cause test skipping. Thanks! Nawab On Thu, Oct 5, 2017 at 3:13 PM, C

Re: Maven build error (Was: Jenkins setup for continuous build)

2017-10-04 Thread Nawab Zada Asad Iqbal
Hi Steve, I did this: ant get-maven-poms cd maven-build/ mvn -DskipTests install On Wed, Oct 4, 2017 at 4:56 PM, Steve Rowe wrote: > Hi Nawab, > > > On Oct 4, 2017, at 7:39 PM, Nawab Zada Asad Iqbal > wrote: > > > > I am hitting following error with maven

Re: Jenkins setup for continuous build

2017-10-04 Thread Nawab Zada Asad Iqbal
] Is that expected? Can someone share me the details about how https://builds.apache.org/job/Lucene-Solr-Maven-master is configured On Wed, Oct 4, 2017 at 9:14 AM, Nawab Zada Asad Iqbal wrote: > Hi, > > I have some custom code in solr

Maven build error (Was: Jenkins setup for continuous build)

2017-10-04 Thread Nawab Zada Asad Iqbal
(TestCustomDocTransformer.java:78) [ERROR] Scanned 1290 (and 2112 related) class file(s) for forbidden API invocations (in 2.74s), 1 error(s). On Wed, Oct 4, 2017 at 9:14 AM, Nawab Zada Asad Iqbal wrote: > Hi, > > I have some custom code in solr (which is not of good quality for > contributing back) so I n

Jenkins setup for continuous build

2017-10-04 Thread Nawab Zada Asad Iqbal
Hi, I have some custom code in solr (which is not of good quality for contributing back) so I need to setup my own continuous build solution. I tried jenkins and was hoping that ant build (ant clean compile) in Execute Shell textbox will work, but I am stuck at this ivy-fail error: To work around

Solr test runs: test skipping logic

2017-10-04 Thread Nawab Zada Asad Iqbal
Hi, I am seeing that in different test runs (e.g., by executing 'ant test' on the root folder in 'lucene-solr') a different subset of tests are skipped. Where can I find more about it? I am trying to create parity between test successes before and after my changes and this is causing confusion.

solr 7.0: mbeans stats missing for many keys

2017-10-01 Thread Nawab Zada Asad Iqbal
hi, When upgrading from solr4.5 to solr 7.0, I noticed that many key names in solr-mbeans ( http://localhost:8983/solr/tcore/admin/mbeans?stats=true&wt=json) have changed. Mostly, "parent key" name is also appended to the stats keyname e.g., instead of cumulative_evictions, I now have to look for

Re: how to recover from OpenSearcher called on closed core

2017-09-28 Thread Nawab Zada Asad Iqbal
Hi Are you upgrading from an earlier version? If not, I am curious why not try SolrCloud instead of Master/Slave. Is there any other error before this error in the logs? Did the core close after a crash? Regards Nawab On Thu, Sep 28, 2017 at 2:57 AM, rubi.hali wrote: > Hi > > we are using So

Re: Solr Beginner!!

2017-09-28 Thread Nawab Zada Asad Iqbal
Hi Jaya Text extraction is a step before you put data into solr. Say, you have pdf or doc type documents, you will extract the text (minus unnecessary formatting details etc.) and store in solr. Later you can query it as you said. i have not worked in extraction area, but look at this for an idea:

Re: Solr cloud most stable version

2017-09-28 Thread Nawab Zada Asad Iqbal
Hi Lars Although, that doesn't really answer of whether 6.6.1 is the most stable one or not, but there has been a recent security fix, so definitely go to 6.6.1 . Copied the detail below:- CVE-2017-9803: Security vulnerability in kerberos delegation token functionality Severity: Important Ve

Re: solr 7.0: possible analysis error: startOffset must be non-negative

2017-09-27 Thread Nawab Zada Asad Iqbal
, 2017 at 3:12 PM, Nawab Zada Asad Iqbal wrote: > Hi, > > I upgraded to solr 7 today and i am seeing tonnes of following errors for > various fields. > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: > Exception writing document id file_3881549 to the index

solr 7.0: possible analysis error: startOffset must be non-negative

2017-09-27 Thread Nawab Zada Asad Iqbal
Hi, I upgraded to solr 7 today and i am seeing tonnes of following errors for various fields. o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id file_3881549 to the index; possible analysis error: startOffset must be non-negative, and endOffset must

Re: When will be solr 7.1 released?

2017-09-26 Thread Nawab Zada Asad Iqbal
, at 8:04 PM, Nawab Zada Asad Iqbal > wrote: > > > > Thanks , another question(s): > > > > why is this released marked 'unreleased' ? > > https://issues.apache.org/jira/projects/SOLR/versions/12335718 > > The 7.0 release manager hasn’t gotten around

Re: When will be solr 7.1 released?

2017-09-26 Thread Nawab Zada Asad Iqbal
t; try to compile you may see instructions to execute an ant target that > >> > downloads ivy. > >> > > >> > One note, there was a comment recently that you may have to get > >> > ivy-2.4.0.jar to have the "ant package" complete successfully. > >>

Re: When will be solr 7.1 released?

2017-09-26 Thread Nawab Zada Asad Iqbal
release manager > role. > >> > >> Some info here about the release process: <https://wiki.apache.org/ > lucene-java/ReleaseTodo> > >> > >> -- > >> Steve > >> www.lucidworks.com > >> > >>> On Sep 26, 2017, a

Re: When will be solr 7.1 released?

2017-09-26 Thread Nawab Zada Asad Iqbal
t spinning a new version, they volunteer. > Nobody has stepped up to do that yet, although I expect it to be in > the next 2-3 months, but that's only a guess. > > Best, > Erick > > On Mon, Sep 25, 2017 at 5:21 PM, Nawab Zada Asad Iqbal > wrote: > > Hi, > > &g

When will be solr 7.1 released?

2017-09-25 Thread Nawab Zada Asad Iqbal
Hi, How are the release dates decided for new versions, are they known in advance? Thanks Nawab

Re: solr 6.6.1: Lock held by this virtual machine

2017-08-25 Thread Nawab Zada Asad Iqbal
h identical setup (except without haproxy traffic), I was able to make start and use (index 100s GB and queries for many hours) this solr version (6.6.1) without any problem. Thanks Nawab On Fri, Aug 25, 2017 at 3:38 PM, Nawab Zada Asad Iqbal wrote: > Thanks Erik > I expected that but it

Re: solr 6.6.1: Lock held by this virtual machine

2017-08-25 Thread Nawab Zada Asad Iqbal
ing to the > _same_ data directory. Examine your core.properties files and see if > any dataDir variables are set > > Best, > Erick > > On Fri, Aug 25, 2017 at 1:12 PM, Nawab Zada Asad Iqbal > wrote: > > Ok, after looking at the logs for some more time, i found

Re: solr 6.6.1: Lock held by this virtual machine

2017-08-25 Thread Nawab Zada Asad Iqbal
.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) On Fri, Aug 25, 2017 at 10:47 AM, Nawab Zada Asad Iqbal wrote: > Hi, > > I am getting this error. i have deleted the file and restart

solr 6.6.1: Lock held by this virtual machine

2017-08-25 Thread Nawab Zada Asad Iqbal
Hi, I am getting this error. i have deleted the file and restarted the server, but this error doesn't go away. What should I do to fix it? Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.(SolrCore.java:977) at org.apa

Re: Request Highlighting only for the final set of rows

2017-08-18 Thread Nawab Zada Asad Iqbal
s asking for only ids. > > > Nawab > > On Fri, Aug 18, 2017 at 8:33 AM, Nawab Zada Asad Iqbal > wrote: > >> Thanks Erick for the pointing to better option. I will explore that. >> After your email, I found that if i have specified 'fl=*' in the query then >

Re: Request Highlighting only for the final set of rows

2017-08-18 Thread Nawab Zada Asad Iqbal
Fri, Aug 18, 2017 at 8:33 AM, Nawab Zada Asad Iqbal wrote: > Thanks Erick for the pointing to better option. I will explore that. After > your email, I found that if i have specified 'fl=*' in the query then it is > doing the right thing (a 2 pass process). However, my queries

Re: Request Highlighting only for the final set of rows

2017-08-18 Thread Nawab Zada Asad Iqbal
d, and those should be the only ones highlighted. > > Do you have any evidence to the contrary that they're all being > highlighted? Or are you misinterpreting the log message for the first > pass? > > Best, > Erick > > On Thu, Aug 17, 2017 at 5:43 PM, Nawab Zada Asad

Request Highlighting only for the final set of rows

2017-08-17 Thread Nawab Zada Asad Iqbal
Hi, In a multi-node solr installation (without SolrCloud), during a paging scenario (e.g., start=1000, rows=200), the primary node asks for 1200 rows from each shard. If highlighting is ON, then the primary node is asking for highlighting all the 1200 results from each shard, which doesn't scale w

Re: solr asks more and more rows from shards

2017-08-17 Thread Nawab Zada Asad Iqbal
So, it turned out that i was not paying attention to the start parameter. I found that the 'primary' node is asking for very large count of rows from shard nodes when the start= is a large value. On Thu, Aug 17, 2017 at 11:28 AM, Nawab Zada Asad Iqbal wrote: > Hi solr community &g

Re: Unable to write response, client closed connection or we are shutting down

2017-08-17 Thread Nawab Zada Asad Iqbal
> On 8/12/2017 11:48 AM, Nawab Zada Asad Iqbal wrote: > > I am executing a query performance test against my solr 6.6 setup and I > > noticed following exception every now and then. What do I need to do? > > > > Aug 11, 2017 08:40:07 AM INFO (qtp761960786-250) [ x:

solr asks more and more rows from shards

2017-08-17 Thread Nawab Zada Asad Iqbal
Hi solr community I am having performance issues after solr6 upgrade. I have multiple nodes in the cluster and direct queries to one of them with `shards=[list of hosts]` which takes care of submitting queries to all the shards and aggregating the results. All the original queries have rows=200.

Re: Solr query help

2017-08-17 Thread Nawab Zada Asad Iqbal
Hi Krishna I haven't used date range queries myself. But if Solr only supports a particular date format, you can write a thin client for queries, which will convert the date to solr's format and query solr. Nawab On Thu, Aug 17, 2017 at 7:36 AM, chiru s wrote: > Hello guys > > I am working on

Re: Unable to write response, client closed connection or we are shutting down

2017-08-16 Thread Nawab Zada Asad Iqbal
:09 PM, Nawab Zada Asad Iqbal wrote: > Hi Rick > > My software is not very sophisticated. I have picked some queries from > production logs, which I am replaying against this solr installation. It is > not a SolrCloud but i specify "shards=" in the query to gather results

Breaking down QTime for debugging performance

2017-08-15 Thread Nawab Zada Asad Iqbal
Hi all For a given solr host and shard, is there any way to get a breakdown on QTime to see where is the time being spent? Thanks Nawab

Re: Unable to write response, client closed connection or we are shutting down

2017-08-15 Thread Nawab Zada Asad Iqbal
e do you use? What else is happening when the exception > occurs? > Cheers -- Rick > > On August 12, 2017 1:48:19 PM EDT, Nawab Zada Asad Iqbal > wrote: > >Hi, > > > >I am executing a query performance test against my solr 6.6 setup and I > >noticed following

Unable to write response, client closed connection or we are shutting down

2017-08-12 Thread Nawab Zada Asad Iqbal
Hi, I am executing a query performance test against my solr 6.6 setup and I noticed following exception every now and then. What do I need to do? Aug 11, 2017 08:40:07 AM INFO (qtp761960786-250) [ x:filesearch] o.a.s.s.HttpSolrCall Unable to write response, client closed connection or we are s

which class is: o.a.s.c.S.Request

2017-08-10 Thread Nawab Zada Asad Iqbal
Hi I see logs from this class 'o.a.s.c.S.Request', and I am able to tune this log by going to the logging webpage (Solr -> Request), but I cannot find the full class name in code. What should I put in the log properties file to disable this log? Thanks Nawab

Solr 6.6: Configure number of indexing threads

2017-08-06 Thread Nawab Zada Asad Iqbal
: https://issues.apache.org/jira/plugins/servlet/mobile#issue/LUCENE-6659 Is there any config in solr 6.6 to control this ? If not , why was the current config considered useless ? Thanks Nawab -- Forwarded message - From: Nawab Zada Asad Iqbal Date: Sun, Aug 6, 2017 at 8:25 AM

Re: Commit takes very long with NoSuchFileException

2017-08-03 Thread Nawab Zada Asad Iqbal
increase number of merge threads and see how it goes. Regards Nawab On Thu, Aug 3, 2017 at 12:52 PM, Nawab Zada Asad Iqbal wrote: > Hi, > > I have a host with 3 solr processes running, each with one shard only; > there are no replicas. I am reindexing some 100 GB of data per

Commit takes very long with NoSuchFileException

2017-08-03 Thread Nawab Zada Asad Iqbal
Hi, I have a host with 3 solr processes running, each with one shard only; there are no replicas. I am reindexing some 100 GB of data per solr (or per shard since each solr has one shard). After about 3 hours, I manually committed once. I was able to get through 40 GB in each shard, and the commi

Re: logging support in Lucene code

2017-07-31 Thread Nawab Zada Asad Iqbal
lucene, as it is included in other projects too. Regards Nawab On Fri, Jul 28, 2017 at 6:57 AM, Shawn Heisey wrote: > On 7/27/2017 10:57 AM, Nawab Zada Asad Iqbal wrote: > > I see a lot of discussion on this topic from almost 10 years ago: e.g., > > https://issues.apache.org/jir

Re: logging support in Lucene code

2017-07-31 Thread Nawab Zada Asad Iqbal
then you > can route the infoStream logging to Solr's log files by setting an option > in the solrconfig.xml. See > http://lucene.apache.org/solr/guide/6_6/indexconfig-in-solrconfig.html# > IndexConfiginSolrConfig-OtherIndexingSettings > > On Fri, Jul 28, 2017 at 11:13 AM

master replication: what are 'master (searching)' and 'master (replicable)' fields

2017-07-31 Thread Nawab Zada Asad Iqbal
Hi, >From the solr console, I see 'master (searching)' and 'master (replicable)' fields on `host/solr/#/core_1/replication` page and wondering how does it impact me given that I don't have any replicas. If I don't have any replicas, does it make any impact on performance by enabling or disablin

Re: logging support in Lucene code

2017-07-27 Thread Nawab Zada Asad Iqbal
Any doughnut for me ? Regards Nawab On Thu, Jul 27, 2017 at 9:57 AM Nawab Zada Asad Iqbal wrote: > Hi, > > I see a lot of discussion on this topic from almost 10 years ago: e.g., > https://issues.apache.org/jira/browse/LUCENE-1482 > > For 4.5, I relied on 'System.o

Re: Unable to create core [collection] Caused by: null

2017-07-27 Thread Nawab Zada Asad Iqbal
Lucas may be hitting this issue: https://stackoverflow.com/questions/4659151/recurring-exception-without-a-stack-trace-how-to-reset Could you try running your server with jvm value: -XX:-OmitStackTraceInFastThrow ? Nawab On Wed, Jul 26, 2017 at 11:42 AM, Anshum Gupta wrote: > Hi Lucas, > >

logging support in Lucene code

2017-07-27 Thread Nawab Zada Asad Iqbal
Hi, I see a lot of discussion on this topic from almost 10 years ago: e.g., https://issues.apache.org/jira/browse/LUCENE-1482 For 4.5, I relied on 'System.out.println' for writing information for debugging in production. In 6.6, I notice that some classes in Lucene are instantiating a Logger, sh

Re: How to use javacc with QueryParser.jj

2017-07-24 Thread Nawab Zada Asad Iqbal
om lucene/ or lucene/solr/ don't accomplish this. So to do this, run 'ant javacc' from lucene/solr/core/. That's it. " On Mon, Jul 24, 2017 at 7:31 AM, Nawab Zada Asad Iqbal wrote: > [Subject changed for reposting] > > Good morning, > > If I want to change s

  1   2   >