Re: Urgent help on solr optimisation issue !!

David: Some of this still matters even with 7.5+. Prior to 7.5, you could easily have 50% of your index consist of deleted docs. With 7.5, this ceiling is reduced. expungeDeletes will reduce the size to no more than 10% while still respecting the default max segment size of 5G. Optimizing and s

Re: Solr Heap Usage

+1 for eclipse mat. Yourkit is another option. Heap dumps are invaluable but a pain. If you’re just interested in overall heap and gc analysis I use gc-viewer, which is usually all you need to know. I do heap dumps when there are for large deviations from expectations and it is non obvious why Gre

NPE in DelegationTokenHttpSolrClient

Hi, Receiving NPE when trying to index into solr collection. Initializing the HTTPSolrClient like this.. HttpSolrClient client = new HttpSolrClient.Builder() .withKerberosDelegationToken(token) .withHttpClient(httpClient)

Re: Urgent help on solr optimisation issue !!

I use the same algorithm and for me, initialMaxSegments is always the number of segments currently in the index (seen, e.g, in the SOLR admin UI). finalMaxSegments depends on what kind of updates have happened. If I know that "older" documents are untouched, then I'll usually use -60% or even -7

Re: Configure mutual TLS 1.2 to secure SOLR

(On the server side there is AFAIK anyway only 1.2 possible) > Am 07.06.2019 um 21:42 schrieb Jörn Franke : > > Configure SSL according to the reference guide. > > Then start each Solr node with the option -Dhttps.protocols=TLSv1.2 > >> Am 07.06.2019 um 17:02 schrieb Paul : >> >> Hi, >> >> Ca

NullPointerException in QueryComponent.unmarshalSortValues

Hi, I'm doing a simple *:* search on an empty multi sharded collection using Solr 7.6 and am getting this exception: NullPointerException     at org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)     at org.apache.solr.handler.component.QueryComponent.

RE: Solr Heap Usage

Hello, We use VisualVM for making observations. But use Eclipse MAT for in-depth analysis, usually only when there is a suspected memory leak. Regards, Markus -Original message- > From:John Davis > Sent: Friday 7th June 2019 20:30 > To: solr-user@lucene.apache.org > Subject: Re: So

Re: Configure mutual TLS 1.2 to secure SOLR

Configure SSL according to the reference guide. Then start each Solr node with the option -Dhttps.protocols=TLSv1.2 > Am 07.06.2019 um 17:02 schrieb Paul : > > Hi, > > Can someone please outline how to use mutual TLS 1.2 with SOLR. Or, point me > at docs/tutorials/other where I can read up furt

Re: Solr Heap Usage

What would be the best way to understand where heap is being used? On Tue, Jun 4, 2019 at 9:31 PM Greg Harris wrote: > Just a couple of points I’d make here. I did some testing a while back in > which if no commit is made, (hard or soft) there are internal memory > structures holding tlogs and i

Re: Urgent help on solr optimisation issue !!

Thanks @Michael Joyner, how did you decide initialmax segment to 256 ? Or it is some random number i can use for my case ? Can you guuide me how to decide the initial & final max segments ? Michael Joyner wrote > That is the way we do it here - also helps a lot with not needing x2 or > x3 disk

Re: Configure mutual TLS 1.2 to secure SOLR

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Paul, On 6/7/19 11:02, Paul wrote: > Can someone please outline how to use mutual TLS 1.2 with SOLR. Or, > point me at docs/tutorials/other where I can read up further on > this (version currently onsite is SOLR 7.6). Here's a copy/paste from our i

Re: searching only within a date range

Disregard my previous response. When I reindexed, something went wrong and so my Lucene database was empty, which explains the immediate results and 0 results. I reindexed again (properly) and all is working find now. Thanks for the help. Mark On Fri, Jun 7, 2019 at 10:40 AM Erick Erickson wro

Re: searching only within a date range

I added "posttime" to the schema first thing this morning, but your message reminded me that I needed to re-index the table, which I did. My schema entry: But my SQL contains "SELECT posttime as id" as so I tried both "posttime" and "id" in my setParam() function, namely, query.setParam("fq", "

Re: Urgent help on solr optimisation issue !!

> On Jun 7, 2019, at 7:53 AM, David Santamauro > wrote: > > So is this new optimize maxSegments / commit expungeDeletes behavior in 7.5? > My experience, and I watch the my optimize process very closely, is that > using maxSgements does not touch every segment with a deleted document. > ex

Re: Query takes a long time Solr 6.1.0

On 6/6/2019 5:45 AM, vishal patel wrote: One server(256GB RAM) has two below Solr instance and other application also 1) shards1 (80GB heap ,790GB Storage, 449GB Indexed data) 2) replica of shard2 (80GB heap, 895GB Storage, 337GB Indexed data) The second server(256GB RAM and 1 TB storage) has tw

Re: Re: Query takes a long time Solr 6.1.0

There isnt anything wrong aside from your query is poorly thought out. On Fri, Jun 7, 2019 at 11:04 AM vishal patel wrote: > Any one is looking my issue?? > > Get Outlook for Android > > > From: vishal patel > Sent: Thursday, June 6, 2019 5

Fwd: Re: Query takes a long time Solr 6.1.0

Any one is looking my issue?? Get Outlook for Android From: vishal patel Sent: Thursday, June 6, 2019 5:15:15 PM To: solr-user@lucene.apache.org Subject: Re: Query takes a long time Solr 6.1.0 Thanks for your reply. > How much index data i

Configure mutual TLS 1.2 to secure SOLR

Hi, Can someone please outline how to use mutual TLS 1.2 with SOLR. Or, point me at docs/tutorials/other where I can read up further on this (version currently onsite is SOLR 7.6). Thanks Paul -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Custom cache for Solr Cloud mode

On 6/7/2019 8:49 AM, Erick Erickson wrote: Yes. ZooKeeper has a “blob store”. See the Blob Store API in the ref guide. Minor nit. You will be creating a jar file, and configuring your collection to be able to find the new jar file. Then you _upload_ both to ZooKeeper and reload your collecti

Re: Urgent help on solr optimisation issue !!

So is this new optimize maxSegments / commit expungeDeletes behavior in 7.5? My experience, and I watch the my optimize process very closely, is that using maxSgements does not touch every segment with a deleted document. expungeDeletes merges all segments that have deleted documents that have b

Re: Custom cache for Solr Cloud mode

Yes. ZooKeeper has a “blob store”. See the Blob Store API in the ref guide. Minor nit. You will be creating a jar file, and configuring your collection to be able to find the new jar file. Then you _upload_ both to ZooKeeper and reload your collection. The rest should be automatic, Solr shoul

Re: searching only within a date range

Yeah, it can be opaque… My first guess is that you may not have a field “posttime” defined in your schema and/or documents. For searching it needs “indexed=true” and for faceting/grouping/sorting it should have “docValues=true”. That’s what your original facet query was telling you, the field i

Re: Urgent help on solr optimisation issue !!

Optimizing guarantees that there will be _no_ deleted documents in an index when done. If a segment has even one deleted document, it’s merged, no matter what you specify for maxSegments. Segments are write-once, so to remove deleted data from a segment it must be at least rewritten into a new

Re: Urgent help on solr optimisation issue !!

Thanks @Erick for the suggestions. That looks so bad, yes your assumptions are right, we have lot of delete & index documents as well. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: searching only within a date range

So, instead of addDateRangeFacet(), I used: query.setParam("fq", "posttime:[2010-01-01T00:00:00Z TO 2015-01-01T00:00:00Z]"); I didn't get any errors, but the query returned immediately with 0 results. Without this contraint, it searches 13,000 records and takes 1 to 2 minutes and returns 356 reco

Re: Urgent help on solr optimisation issue !!

Thanks Shawn for suggestions. Interesting to know deleteByQuery has some impact, will try to change it as you have suggested. Thabks -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr cloud setup

First of all, do not shard unless necessary to handle your QPS requirements. Sharding adds overhead and has some functionality limitations. How to define “necessary”? Load test a single shard (or even stand-alone with a single core) until it falls over. See: https://lucidworks.com/2012/07/23/si

Re: Urgent help on solr optimisation issue !!

/clarification/ ... expungeDeletes will merge every segment *touched by the current commit* that has a deleted document. On 6/7/19, 10:07 AM, "David Santamauro" wrote: Erick, on 6.0.1, optimize with maxSegments only merges down to the specified number. E.g., given an index with 75 segme

Re: Urgent help on solr optimisation issue !!

Erick, on 6.0.1, optimize with maxSegments only merges down to the specified number. E.g., given an index with 75 segments, optimize with maxSegments=74 will only merge 2 segments leaving 74 segments. It will choose a segment to merge that has deleted documents, but does not merge every segment

Re: Urgent help on solr optimisation issue !!

That is the way we do it here - also helps a lot with not needing x2 or x3 disk space to handle the merge: public void solrOptimize() {         int initialMaxSegments = 256;         int finalMaxSegments = 4;         if (isShowSegmentCounter()) {             log.info("Optimizing ...");         }

Re: Urgent help on solr optimisation issue !!

This isn’t quite right. Solr will rewrite _all_ segments that have _any_ deleted documents in them when optimizing, even one. Given your description, I’d guess that all your segments will have deleted documents, so even if you do specify maxSegments on the optimize command, the entire index will

Re: searching only within a date range

Hi Mark, you are using a "range facet" which is a "query-shape" feature, it doesn't have any constraint on the results (i.e. it doesn't filter at all). You need to add a filter query [1] with a date range clause (e.g. fq=field:[ TO or *>]). Best, Andrea [1] https://lucene.apache.org/solr/gui

Re: Urgent help on solr optimisation issue !!

On 6/6/2019 11:27 PM, jena wrote: Because of heavy indexing & deletion, we optimise solr instance everyday, because of that our solr cloud getting unstable , every solr instance go on recovery mode & our search is getting affected & very slow because of that. Optimisation takes around 1hr 30minut

searching only within a date range

Hello! I have a search setup and it works fine. I search a text field called "logtext" in a database table. My Java code is like this: SolrQuery query - new SolrQuery(); query.setQuery(searchWord); query.setParam("df", "logtext"); Then I execute the search... and it works just great. But now

Issues with calculating metrics and sorting on a float field in a stream

Hi guys! I bumped into a couple of issues when trying to sort a stream or calculate metrics on a Float field which contains values without the decimal part (e.g 1.0, 0.0, etc.). 1. Issues with sorting. Consider this expression: > sort( > list( >tuple(a=val(1.0)), >tuple(a=val

Re: Urgent help on solr optimisation issue !!

Thanks @Nicolas Franck for reply, i don't see any any segment info for 4.4 version. Is there any API i can use to get my segment information ? Will try to use maxSegments and see if it can help us during optimization. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr slave core corrupted and not replicating.

Hi team, Please help us in the issue mentioned above. If this is not the right place to look please direct us to the correct team. Thanks & Regards, Y Mahesh Varma On Thu, 6 Jun, 2019, 1:21 AM varma mahesh, wrote: > ++solr-user@lucene.apache.org > > On Thu 6 Jun, 2019, 1:19 AM varma mahesh, w

RE: query parsed in different ways in two identical solr instances

any thoughts on that difference in the solr parsing? is it correct that the first looks like an AND while the second looks like and OR? Thank you Danilo Tomasoni Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI) Piazza Manifattura 1,

Re: Urgent help on solr optimisation issue !!

In that case, hard optimisation like that is out the question. Resort to automatic merge policies, specifying a maximum amount of segments. Solr is created with multiple segments in mind. Hard optimisation seems like not worth the problem. The problem is this: the less segments you specify during

Re: Solr cloud setup

Hi Abhishek, Here is a nice blog post about migrating to SolrCloud: https://sematext.com/blog/solr-master-slave-solrcloud-migration/ Re number of shards - there is no definite answer - it depends on your indexing/search latency

Urgent help on solr optimisation issue !!

Hello guys, We have 4 solr(version 4.4) instance on production environment, which are linked/associated with zookeeper for replication. We do heavy deleted & add operations. We have around 26million records and the index size is around 70GB. We serve 100k+ requests per day. Because of heavy inde

Solr cloud setup

Hi , Currently we are in master slave architechture we want to move in solr cloud architechture . how i should decide shard number in solr cloud ? My current solr in version 6 and index size is 300 GB. Regards, Abhishek Tiwari