Solr NRT Replicas Out of Sync

2021-03-03 Thread Anshuman Singh
Hi, In our Solr 7.4 cluster, we have noticed that some replicas of some of our Collections are out of sync, the slave replica has more number of records than the leader. This is resulting in different number of records on subsequent queries on the same Collection. Commit is also not helping in thi

Re: NRT - Indexing

2021-02-02 Thread Dominique Bejean
Hi, The issue was buildOnCommit=true on a SuggestComponent. Dominique Le mar. 2 févr. 2021 à 00:54, Shawn Heisey a écrit : > On 2/1/2021 12:08 AM, haris.k...@vnc.biz wrote: > > Hope you're doing good. I am trying to configure NRT - Indexing in my > > project. For

Re: NRT - Indexing

2021-02-01 Thread Shawn Heisey
On 2/1/2021 12:08 AM, haris.k...@vnc.biz wrote: Hope you're doing good. I am trying to configure NRT - Indexing in my project. For this reason, I have configured *autoSoftCommit* to execute every second and *autoCommit* to execute every 5 minutes. Everything works as expected on the de

Re: NRT - Indexing

2021-02-01 Thread Dominique Bejean
u grep your solr logs on with the "commit' pattern in order to see > > hard and soft commit occurrences ? > > How are you pushing new docs or updates in the collection ? > > > Regards. > > > Dominique > > > > > > Le lun. 1 févr. 2021 à 0

Re: NRT - Indexing

2021-02-01 Thread haris . khan
p your solr logs on with the "commit' pattern in order to seehard and soft commit occurrences ?How are you pushing new docs or updates in the collection ?Regards.DominiqueLe lun. 1 févr. 2021 à 08:08, <haris.k...@vnc.biz> a écrit :> Hello,>> Hope you're doin

Re: NRT - Indexing

2021-02-01 Thread Dominique Bejean
Le lun. 1 févr. 2021 à 08:08, a écrit : > Hello, > > Hope you're doing good. I am trying to configure NRT - Indexing in my > project. For this reason, I have configured *autoSoftCommit* to execute > every second and *autoCommit* to execute every 5 minutes. Everything > work

Re: NRT - Indexing

2021-02-01 Thread Mr Havercamp
so seem to have no effect (from what I understand of the docs, Soft commit makes items viewable but I'm not seeing them until well after the autoCommit period has passed. On Mon, 1 Feb 2021 at 15:08, wrote: > Hello, > > Hope you're doing good. I am trying to configure NRT -

NRT - Indexing

2021-01-31 Thread haris . khan
Hello, Hope you're doing good. I am trying to configure NRT - Indexing in my project. For this reason, I have configured autoSoftCommit to execute every second and autoCommit to execute every 5 minutes. Everything works as expected on the dev and test server. But on the production s

NRT - Indexing

2021-01-29 Thread haris . khan
Hello, Hope you're doing good. I am trying to configure NRT - Indexing in my project. For this reason, I have configured autoSoftCommit to execute every second and autoCommit to execute every 5 minutes. Everything works as expected on the dev and test server. But on the production s

Re: Suggestion or recommendation for NRT

2020-07-08 Thread ramyogi
Hi Team, Any suggestion or recommendation for the above approach which we are doing to have better search performance. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Suggestion or recommendation for NRT

2020-07-02 Thread ramyogi
Thanks a lot for your time to respond for my clarifications. We are having two environment, ENV A and ENV B ( Both same capacity of RAM ( r5.2xlarge and same number of shards and replicas type (NRT) for the collection) ENV A - it is having a collection which is optimized ( segment count 1 and

Re: Suggestion or recommendation for NRT

2020-07-01 Thread Erick Erickson
That seems high. It can be tricky to get tests. Are you running with some kind of test runner? Do you have, say, 3-4 thousand queries you run? Are you running the tests after warming the searchers? Also, if you have indexed down to one segment, _then_ tried adding docs and measuring you are not ge

Re: Suggestion or recommendation for NRT

2020-07-01 Thread ramyogi
Thanks Erick for the details and reference to understand better about merging segment stuff. When I compare performance of uninterrupted/optimized ( segment count 1) collection for search request vs (indexing + search) in parallel going on collection performance is 3 times higher, for example

Re: Suggestion or recommendation for NRT

2020-07-01 Thread Erick Erickson
Updated documents are marked as deleted in the old segment and added to a new segment. When commits happen, merges occur and only then is the space occupied by the deleted document reclaimed. Which segments are merged on commit depends on a number of factors. Unless you can prove the extra space

Re: Suggestion or recommendation for NRT

2020-07-01 Thread ramyogi
Even though same document indexed over and over again due to incremental update. Index size is being increased. Do I miss any configuration to make optimization occur by internally ? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Suggestion or recommendation for NRT

2020-06-29 Thread ramyogi
Hi, We are using SOLR 7.5.0 version, We are testing one collection for both Search and Index. Our collection created with below indexerconfig, We are using indexing process KAFKA connect plugin with every 5 min commit (cloud SOLRJ) as below https://github.com/jcustenborder/kafka-connect-solr Our

Re: NRT Real time Get with documentCache

2020-02-03 Thread Karl Stoney
Great stuff thank you Erick On 04/02/2020, 00:17, "Erick Erickson" wrote: The documentCache shouldn’t matter at all. RTG should return the latest doc by maintaining a pointer into the tlogs and returning that version. > On Feb 3, 2020, at 6:43 PM, Karl Stoney wrote: > > Hi,

Re: NRT Real time Get with documentCache

2020-02-03 Thread Erick Erickson
The documentCache shouldn’t matter at all. RTG should return the latest doc by maintaining a pointer into the tlogs and returning that version. > On Feb 3, 2020, at 6:43 PM, Karl Stoney > wrote: > > Hi, > Could anyone let me know if a real time get would return a cached, up to date > version

NRT Real time Get with documentCache

2020-02-03 Thread Karl Stoney
Hi, Could anyone let me know if a real time get would return a cached, up to date version of a document if we enabled documentCache? Thanks Karl This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England

Re: Convert TLOG collection to NRT

2019-12-10 Thread Dominique Bejean
) from TLOG to > > NRT. > > > > The only solution I imagine is something like : > > * with collection API, remove replicas in order to keep only 1 replica > per > > 3 shard > > * update the collection state.json in zookeer > > * with collection API, reload t

Re: Convert TLOG collection to NRT

2019-12-10 Thread Shawn Heisey
On 12/10/2019 11:25 AM, Dominique Bejean wrote: I would like to convert a collection (3 shards x 3 replicas) from TLOG to NRT. The only solution I imagine is something like : * with collection API, remove replicas in order to keep only 1 replica per 3 shard * update the collection state.json in

Convert TLOG collection to NRT

2019-12-10 Thread Dominique Bejean
Hi, I would like to convert a collection (3 shards x 3 replicas) from TLOG to NRT. The only solution I imagine is something like : * with collection API, remove replicas in order to keep only 1 replica per 3 shard * update the collection state.json in zookeer * with collection API, reload the

Re: NRT vs TLOG bulk indexing performances

2019-10-30 Thread Dominique Bejean
tests : - One shard leader only NRT or TLOG : 36 minutes - All NRT timing is between 23 and 27 minutes - All TLOG timing is between 28 and 34 minutes I also changed the autoCommit maxtime from 15000 et 3 in order to get the 28 minutes in TLOG mode. With one shard and no replica,

Re: NRT vs TLOG bulk indexing performances

2019-10-26 Thread Erick Erickson
"I understand that while non leader TLOG is copying the index from leader, the leader stop indexing” This _better_ not be happening. If you can demonstrate this let’s open a JIRA. > On Oct 25, 2019, at 8:28 AM, Dominique Bejean > wrote: > > I understand that while non leader TLOG is copying th

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Erick Erickson
I’m also surpised that you see a slowdown, it’s worth investigating. Let’s take the NRT case with only a leader. I’ve seen the NRT indexing time increase when even a single follower was added (30-40% in this case). We believed that the issue was the time the leader sat waiting around for the

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Ere Maijala
Shawn Heisey kirjoitti 25.10.2019 klo 14.54: > With newer Solr versions, you can ask SolrCloud to prefer PULL replicas > for querying, so queries will be targeted to those replicas, unless they > all go down, in which case it will go to non-preferred replica types.  I > do not know how to do this,

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Dominique Bejean
10/25/2019 1:16 AM, Dominique Bejean wrote: > > For collection created with all replicas as NRT > > > > * Indexing time : 22 minutes > > > > > For collection created with all replicas as TLOG > > > > * Indexing time : 34 minutes > > NRT indexes sim

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Shawn Heisey
On 10/25/2019 1:16 AM, Dominique Bejean wrote: For collection created with all replicas as NRT * Indexing time : 22 minutes For collection created with all replicas as TLOG * Indexing time : 34 minutes NRT indexes simultaneously on all replicas. So when indexing is done on one, it is

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Dominique Bejean
est? > > > Am 25.10.2019 um 09:16 schrieb Dominique Bejean < > dominique.bej...@eolya.fr>: > > > > Hi, > > > > I made some benchmarks for bulk indexing in order to compare performances > > and ressources usage for NRT versus TLOG replica. > > >

Re: NRT vs TLOG bulk indexing performances

2019-10-25 Thread Jörn Franke
Which Solr version are you using and how often you repeated the test? > Am 25.10.2019 um 09:16 schrieb Dominique Bejean : > > Hi, > > I made some benchmarks for bulk indexing in order to compare performances > and ressources usage for NRT versus TLOG replica. > > Env

NRT vs TLOG bulk indexing performances

2019-10-25 Thread Dominique Bejean
Hi, I made some benchmarks for bulk indexing in order to compare performances and ressources usage for NRT versus TLOG replica. Environnent : * Solrcloud with 4 Solr nodes (8 Gb RAM, 4 Gb Heap) * 1 collection with 2 shards x 2 replicas (all NRT or all TLOG) * 1 core per Solr Server Indexing of

Re: NRT for new items in index

2019-08-08 Thread Updates Profimedia
NRT times about one minute are acceptable Thank you On 2019/08/06 19:59:49, Shawn Heisey wrote: > On 7/31/2019 6:47 AM, profiuser wrote: > > we have something about 400 000 000 items in a solr collection. > > We have set up auto commit property for this collection to 15 minute

Re: NRT for new items in index

2019-08-06 Thread Anthony Gatlin
; we have something about 400 000 000 items in a solr collection. > We have set up auto commit property for this collection to 15 minutes. > Is a big collection and we using some caches etc. Therefore we have big > autocommit value. > > This have disadvantage that we haven't NR

Re: NRT for new items in index

2019-08-06 Thread Shawn Heisey
. Start with a value between two and five minutes. This have disadvantage that we haven't NRT searches. We would like to have NRT at least for searching for the newly added items. We read about new functionality "Category routed alilases" in a solr version 8.1. And we got an idea

Re: NRT for new items in index

2019-08-06 Thread Updates Profimedia
Furkan KAMACI wrote: > >> Hi, > >> > >> First of all, could you check here: > >> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > >> to > >> better understand hard commits, soft commits and transa

Re: NRT for new items in index

2019-08-05 Thread Jörn Franke
ing-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> to >> better understand hard commits, soft commits and transaction logs to >> achieve NRT search. >> >> Kind Regards, >> Furkan KAMACI >> >>> On Wed, Jul 31, 2019 at 3:47 PM profiuser

Re: NRT for new items in index

2019-08-05 Thread Updates Profimedia
On 2019/08/03 18:00:28, Furkan KAMACI wrote: > Hi, > > First of all, could you check here: > https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > to > better understand hard commits, soft commits and transaction logs to &g

Re: NRT for new items in index

2019-08-03 Thread Furkan KAMACI
Hi, First of all, could you check here: https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ to better understand hard commits, soft commits and transaction logs to achieve NRT search. Kind Regards, Furkan KAMACI On Wed, Jul 31, 2019 at 3:47 PM

NRT for new items in index

2019-07-31 Thread profiuser
Hi, we have something about 400 000 000 items in a solr collection. We have set up auto commit property for this collection to 15 minutes. Is a big collection and we using some caches etc. Therefore we have big autocommit value. This have disadvantage that we haven't NRT searches. We would

Re: Regarding Shards - Composite / Implicit , Replica Type - NRT / TLOG

2019-01-04 Thread Shawn Heisey
On 1/3/2019 11:26 PM, Doss wrote: We are planning to setup a SOLR cloud with 6 nodes for 3 million records (expected to grow to 5 million in a year), with 150 fields and over all index would come around 120GB. We plan to use NRT with 5 sec soft commit and 1 min hard commit. Five seconds is

Re: Regarding Shards - Composite / Implicit , Replica Type - NRT / TLOG

2019-01-04 Thread Erick Erickson
rect leader, avoiding one extra hop. Given your soft commit setting of 5 seconds, I infer that the allowable time for updates to be searchable is quite small, indicating that NRT replicas are the way to go. I'll also say that this commit rate is pretty aggressive given your volume, is it really nec

Regarding Shards - Composite / Implicit , Replica Type - NRT / TLOG

2019-01-03 Thread Doss
Hi, We are planning to setup a SOLR cloud with 6 nodes for 3 million records (expected to grow to 5 million in a year), with 150 fields and over all index would come around 120GB. We plan to use NRT with 5 sec soft commit and 1 min hard commit. Expected query volume would be 5000 select hits

Re: Solr Setup using NRT and PULL replicas

2018-12-02 Thread Daniel Carrasco
don't know why), and then the entire cluster goes down (all replicas are NRT). Looks like was related with hugepages of that nodes and for now we're investigateing why. Greetings! El dom., 2 dic. 2018 a las 18:48, Edward Ribeiro () escribió: > To mix NRT and TLOG/PULL replicas i

Re: Solr Setup using NRT and PULL replicas

2018-12-02 Thread Edward Ribeiro
To mix NRT and TLOG/PULL replicas is not recommended. It is all NRT nodes or TLOG nodes mixed (or not) with PULL replicas. As you know, all PULL replicas is not possible. According to the talk below, one of the reasons is that if you have NRT mixed with TLOG and PULL replicas then a leadership

Solr Setup using NRT and PULL replicas

2018-11-30 Thread Daniel Carrasco
Hello, We've a cluster consisting in 7 to 10 NRT nodes serving data to a webpage (products, categories,...), but every time a leader node fails importing data (connection lost, broken pipe...), the entire cluster goes to recovering mode and then is not working for about 15-30 minutes. Tha

Issues with new replicas in solrcloud cluster being created as NRT by autoscaling and not respecting cluster policy

2018-11-05 Thread Matt Goward [Contractor]
I am working on an autoscaling kubernetes cluster for solrcloud running 7.5. I have most of this up and working, but ran into a few issues when I got to the point of testing. The core of it is that when solr replaces a replica it is doing so as NRT rather than TLOG and it is not respecting

Re: Handshake for NRT?

2018-08-21 Thread Walter Underwood
t;>> On Aug 20, 2018, at 7:02 PM, Shalin Shekhar Mangar >>>>> wrote: >>>>> >>>>> There are a single persistent HTTP connection open from the leader to each >>>>> replica in the shard. All updates coming to the leader are expanded

Re: Handshake for NRT?

2018-08-21 Thread Erick Erickson
gt;>> atomic updates) and streamed over that single connection. When using >>>> in-place docvalues updates, there is a possibility of the replica making a >>>> request to the leader if updates has been re-ordered and the replica does >>>> not have enough context t

Re: Handshake for NRT?

2018-08-21 Thread Walter Underwood
the replica making a >>> request to the leader if updates has been re-ordered and the replica does >>> not have enough context to process the update. >>> >>> Can you quantify the "tons of internal traffic"? Are you seeing higher >>> number of op

Re: Handshake for NRT?

2018-08-20 Thread Erick Erickson
ffic"? Are you seeing higher >> number of open connections as well? >> >> On Fri, Aug 17, 2018 at 11:17 PM Walter Underwood >> wrote: >> >>> How many messages are sent back and forth between a leader and replica >>> with NRT? >>> >>> W

Re: Handshake for NRT?

2018-08-20 Thread Walter Underwood
not have enough context to process the update. > > Can you quantify the "tons of internal traffic"? Are you seeing higher > number of open connections as well? > > On Fri, Aug 17, 2018 at 11:17 PM Walter Underwood > wrote: > >> How many messages are sent ba

Re: Handshake for NRT?

2018-08-20 Thread Shalin Shekhar Mangar
> How many messages are sent back and forth between a leader and replica > with NRT? > > We have a collection that gets frequent updates and we are seeing a ton of > internal > cluster traffic. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwo

Handshake for NRT?

2018-08-17 Thread Walter Underwood
How many messages are sent back and forth between a leader and replica with NRT? We have a collection that gets frequent updates and we are seeing a ton of internal cluster traffic. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog)

Re: FilterQueries with NRT

2018-05-12 Thread Mikhail Khludnev
You can experiment with the backdoor NRT filters: - replace COLOR:Red to {!parent which='COLOR:Red' v=''} - add *NoOpRegenerator **into perSegFilter* *- **monitor cache stats. * On Fri, May 11, 2018 at 10:46 PM, root23 wrote: > Hi all, > We have a requirement for NRT s

Re: FilterQueries with NRT

2018-05-11 Thread Erick Erickson
ee: http://yonik.com/advanced-filter-caching-in-solr/. And I'll spare you the rant about 500 ms being unreasonable as you seem to already get that ;) Best, Erick On Fri, May 11, 2018 at 12:46 PM, root23 wrote: > Hi all, > We have a requirement for NRT search. Our autosoft commit tim

FilterQueries with NRT

2018-05-11 Thread root23
Hi all, We have a requirement for NRT search. Our autosoft commit time is set to 500 ms(i know its low.But that's another story). We use filter queries extensively for most of our queries. But i am trying to understand how filter query caching works with NRT. Now as i understand we use f

Re: Solr 7.2.1 DELETEREPLICA automatically NRT replica appears

2018-03-07 Thread Greg Roodt
Thu, 8 Mar 2018 at 15:49, Tomas Fernandez Lobbe wrote: > This shouldn’t be happening. Did you see anything related in the logs? > Does the new NRT replica ever becomes active? Is there a new core created > or do you just see the replica in the clusterstate? > > Tomas > > Sen

Re: Solr 7.2.1 DELETEREPLICA automatically NRT replica appears

2018-03-07 Thread Tomas Fernandez Lobbe
This shouldn’t be happening. Did you see anything related in the logs? Does the new NRT replica ever becomes active? Is there a new core created or do you just see the replica in the clusterstate? Tomas Sent from my iPhone > On Mar 7, 2018, at 8:18 PM, Greg Roodt wrote: > > Hi

Solr 7.2.1 DELETEREPLICA automatically NRT replica appears

2018-03-07 Thread Greg Roodt
Hi I am running a cluster of TLOG and PULL replicas. When I call the DELETEREPLICA api to remove a replica, the replica is removed, however, a new NRT replica pops up in a down state in the cluster. Any ideas why? Greg

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-03-02 Thread Webster Homer
essage- > From: Webster Homer [mailto:webster.ho...@sial.com] > Sent: Friday, March 2, 2018 10:29 AM > To: solr-user@lucene.apache.org > Subject: Re: NRT replicas miss hits and return duplicate hits when paging > solrcloud searches > > I am trying to test if enabling stats

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-03-02 Thread Webster Homer
NRT replicas. So using TLOG replicas still looks like the best work around for the NRT issue On Fri, Mar 2, 2018 at 10:44 AM, Shawn Heisey wrote: > On 3/2/2018 9:28 AM, Webster Homer wrote: > >> I've never disabled this before. I edited the solrconfig.xml setting the >> size

RE: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-03-02 Thread Becky Bonner
pplication they belong to. It wasn’t a problem until we had 2 collections for one of the apps. -Original Message- From: Webster Homer [mailto:webster.ho...@sial.com] Sent: Friday, March 2, 2018 10:29 AM To: solr-user@lucene.apache.org Subject: Re: NRT replicas miss hits and return

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-03-02 Thread Shawn Heisey
On 3/2/2018 9:28 AM, Webster Homer wrote: I've never disabled this before. I edited the solrconfig.xml setting the sizes to 0. I'm not sure if this is how to disable the cache or not. To completely disable a cache, either comment it out or remove it from the config.  I do not know wheth

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-03-02 Thread Webster Homer
s cache will make the other searches > more stable. I think we will eventually move to favoring tlog replicas. We > have a couple of collections where NRT makes sense, but those collections > don't need to return data in relevancy order. I think NRT should be > considered a nich

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-27 Thread Webster Homer
from Endeca which has very consistent results. I'm hoping that using the global stats cache will make the other searches more stable. I think we will eventually move to favoring tlog replicas. We have a couple of collections where NRT makes sense, but those collections don't need to retu

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-27 Thread Emir Arnautović
sort would defeat that purpose. >> Since >>> we wanted to be able to return all the results for a query, I originally >>> looked at using the Streaming API, but that doesn't support returning >>> results sorted by relevancy >>> >>> I disagree

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-26 Thread Webster Homer
defeat that purpose. > Since > > we wanted to be able to return all the results for a query, I originally > > looked at using the Streaming API, but that doesn't support returning > > results sorted by relevancy > > > > I disagree with you about NRT replicas tho

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-26 Thread Erick Erickson
I originally > looked at using the Streaming API, but that doesn't support returning > results sorted by relevancy > > I disagree with you about NRT replicas though. They may function as > designed, but since they cannot guarantee consistent results their design > is buggy,

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-26 Thread Webster Homer
Streaming API, but that doesn't support returning results sorted by relevancy I disagree with you about NRT replicas though. They may function as designed, but since they cannot guarantee consistent results their design is buggy, at least it is for a search engine. On Mon, Feb 26, 2018 at 12:

Re: NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-26 Thread Shawn Heisey
On 2/26/2018 10:26 AM, Webster Homer wrote: > We need the results by relevancy so the application sorts the results by > score desc, and the unique id ascending as the tie breaker This is the reason for the discrepancy, and why the different replica types don't have the same issue

NRT replicas miss hits and return duplicate hits when paging solrcloud searches

2018-02-26 Thread Webster Homer
I have an application which implements several different searches against a solrcloud collection. We are using Solr 7.2 and Solr 6.1 The collection b2b-catalog-material is created with the default Near Real Time (NRT) replicas. The collection has 2 shards each with 2 replicas. The application

Sharding strategy for optimal NRT performance

2017-04-15 Thread Chris Troullis
Hi! I am looking for some advice on an sharding strategy that will produce optimal performance in the NRT search case for my setup. I have come up with a strategy that I think will work based on my experience, testing, and reading of similar questions on the mailing list, but I was hoping to run

Re: NRT updates

2016-06-06 Thread Chris Vizisa
Hi, Any pointers, suggestions, experiences ... please.. Thanks! Chris. On Mon, Jun 6, 2016 at 10:27 AM, Chris Vizisa wrote: > Hi, > > Does number of fields in a document affect NRT updates? > I have around 1.6 million products. Each product can be available in about > 30

NRT updates

2016-06-06 Thread Chris Vizisa
Hi, Does number of fields in a document affect NRT updates? I have around 1.6 million products. Each product can be available in about 3000 stores. In addition to around 50 fields related to a product I am storing product_store info in each product document like: 1. Quantity of that product in

Re: NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-15 Thread Charlie Hull
n the function queries for boosting, sorting, etc. Anyone has experience with this? Option 2) SOLR NRT with Soft Commits We would depend on the in-built NRT features. Let's say we do soft-commits every second and hard-commits every 10 seconds. Suppose huge amount of dynamic data is created o

Re: NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-15 Thread Vikram Parmar
> > > The plugin looks good, but not sure if the plugin can be used to fetch > the > > data stored in Redis as part of the solr result set, i.e. in docs. The > > description looks more like the Redis data can be used in the function > > queries for boosting, sorting, et

Re: NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-15 Thread Mikhail Khludnev
The plugin looks good, but not sure if the plugin can be used to fetch the > data stored in Redis as part of the solr result set, i.e. in docs. The > description looks more like the Redis data can be used in the function > queries for boosting, sorting, etc. Anyone has experience with this?

Re: NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-11 Thread Jack Krupansky
atext/solr-redis > > > > The plugin looks good, but not sure if the plugin can be used to fetch > the > > data stored in Redis as part of the solr result set, i.e. in docs. The > > description looks more like the Redis data can be used in the function > > queries for

Re: NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-11 Thread Andrea Gazzarini
part of the solr result set, i.e. in docs. The > description looks more like the Redis data can be used in the function > queries for boosting, sorting, etc. Anyone has experience with this? > > > Option 2) SOLR NRT with Soft Commits > > We would depend on the in-built NRT features

NRT vs Redis for Dynamic Data in SOLR (like counts, viewcounts, etc) -

2015-12-11 Thread Vikram Parmar
. in docs. The description looks more like the Redis data can be used in the function queries for boosting, sorting, etc. Anyone has experience with this? Option 2) SOLR NRT with Soft Commits We would depend on the in-built NRT features. Let's say we do soft-commits every second and ha

Re: Replication and soft commits for NRT searches

2015-10-15 Thread Erick Erickson
bq: the background for my question is that one of the requirements for our injection tool is that it should report that a new document has been successfully enrolled to the cluster only if it is available on all replicas Frankly, this is the tail wagging the dog. SolrCloud is designed to guarantee

Re: Replication and soft commits for NRT searches

2015-10-14 Thread MOIS Martin (MORPHO)
Hello, the background for my question is that one of the requirements for our injection tool is that it should report that a new document has been successfully enrolled to the cluster only if it is available on all replicas. The automated integration test for this feature will submit a document

Re: Replication and soft commits for NRT searches

2015-10-14 Thread Erick Erickson
bq: If a timeout between shard leader and replica can lead to a smaller rf value (because replication has timed out), is it possible to increase this timeout in the configuration? Why do you care? If it timed out, then the follower will no longer be active and will not serve queries. The Cloud vie

Re: Replication and soft commits for NRT searches

2015-10-13 Thread MOIS Martin (MORPHO)
Hello, thank you for the detailed answer. If a timeout between shard leader and replica can lead to a smaller rf value (because replication has timed out), is it possible to increase this timeout in the configuration? Best Regards, Martin Mois Comments inline: On Mon, Oct 12, 2015 at 1:31 PM

Re: Replication and soft commits for NRT searches

2015-10-13 Thread Shalin Shekhar Mangar
Comments inline: On Mon, Oct 12, 2015 at 1:31 PM, MOIS Martin (MORPHO) wrote: > Hello, > > I am running Solr 5.2.1 in a cluster with 6 nodes. My collections have been > created with replicationFactor=2, i.e. I have one replica for each shard. > Beyond that I am using autoCommit/maxDocs=1 an

Re: Replication and soft commits for NRT searches

2015-10-12 Thread Erick Erickson
First of all, setting soft commit with maxDocs=1 is almost (but not quite) guaranteed to lead to problems. For _every_ document you add to Solr, all your top-level caches (i.e. the ones configured in solrconrig.xml) will be thrown away, all autowarming will be performed etc. Essentially assuming a

Replication and soft commits for NRT searches

2015-10-12 Thread MOIS Martin (MORPHO)
Hello, I am running Solr 5.2.1 in a cluster with 6 nodes. My collections have been created with replicationFactor=2, i.e. I have one replica for each shard. Beyond that I am using autoCommit/maxDocs=1 and autoSoftCommits/maxDocs=1 in order to achieve near realtime search behavior. As far a

Re: SOLR nrt read writes

2015-07-21 Thread Alessandro Benedetti
> > Could this be due to caching? I have tried to disable all in my solrconfig. If you mean Solr caches ? NO . Solr caches live the life of the searcher. So new searcher, new caches ( possibly warmed with updated results) . If you mean your application caching or browser caching, you should veri

Re: SOLR nrt read writes

2015-07-21 Thread Upayavira
Bhawna, I think you need to reconcile yourself to the fact that what you want to achieve is not going to be possible. Solr (and Lucene underneath it) is HEAVILY optimised for high read/low write situations, and that leads to some latency in content reaching the index. If you wanted to change this

Re: SOLR nrt read writes

2015-07-20 Thread Bhawna Asnani
Thanks, I tried turning off auto softCommits but that didn't help much. Still seeing stale results every now and then. Also load on the server very light. We are running this just on a test server with one or two users. I don't see any warning in logs whole doing softCommits and it says it succe

Re: SOLR nrt read writes

2015-07-20 Thread Shawn Heisey
On 7/20/2015 9:29 AM, Bhawna Asnani wrote: > Thanks for your suggestions. The requirement is still the same , to be > able to make a change to some solr documents and be able to see it on > subsequent search/facet calls. > I am using softCommit with waitSearcher=true. > > Also I am sending reads/wr

Re: SOLR nrt read writes

2015-07-20 Thread Bhawna Asnani
ed > >> > frequently under application control (vs. auto commit) and we turned > off > >> > all auto-warming and most of the caching. > >> > > >> > There is scant documentation in the new Solr reference ( > cwiki.apache.org > >> ), > >> &

Re: SOLR nrt read writes

2015-07-15 Thread Erick Erickson
lrCaching >> > >> > Just a thought: would true be helpful >> > here? >> > >> > Also, since you have just inserted the documents, it sounds like you >> > probably could search by ID ... >> > >> > -Original Message- &

Re: SOLR nrt read writes

2015-07-15 Thread Bhawna Asnani
ps://wiki.apache.org/solr/SolrCaching > > > > Just a thought: would true be helpful > > here? > > > > Also, since you have just inserted the documents, it sounds like you > > probably could search by ID ... > > > > -Original Message- > > Fr

Re: SOLR nrt read writes

2015-07-15 Thread Daniel Collins
inserted the documents, it sounds like you > probably could search by ID ... > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Tuesday, July 14, 2015 6:04 PM > To: solr-user@lucene.apache.org > Subject: Re: SOLR nrt read writes > > On

RE: SOLR nrt read writes

2015-07-15 Thread Reitzel, Charles
? Also, since you have just inserted the documents, it sounds like you probably could search by ID ... -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Tuesday, July 14, 2015 6:04 PM To: solr-user@lucene.apache.org Subject: Re: SOLR nrt read writes On 7/14/2015 12:19

Re: SOLR nrt read writes

2015-07-14 Thread Shawn Heisey
On 7/14/2015 12:19 PM, Bhawna Asnani wrote: > I have a use case where we have to write data into solr and immediately > read it back. > The read is not get by Id but a search call. > > I am doing a softCommit after every such write which needs to be visible > immediately. > However sometimes the ch

Re: SOLR nrt read writes

2015-07-14 Thread Erick Erickson
anks once again. > > On Tue, Jul 14, 2015 at 3:09 PM, Erick Erickson > wrote: > >> bq: I have a use case where we have to write data into solr and immediately >> read it back. >> >> This is simply not going to work with frequent updates. Solr >> promises

Re: SOLR nrt read writes

2015-07-14 Thread Bhawna Asnani
> This is simply not going to work with frequent updates. Solr > promises Near in NRT, not "real time". > > If nothing else, if you fire the query before autowarming is completed. In > this > case you'll sometimes get the doc and sometimes not because you'll > g

Re: SOLR nrt read writes

2015-07-14 Thread Erick Erickson
bq: I have a use case where we have to write data into solr and immediately read it back. This is simply not going to work with frequent updates. Solr promises Near in NRT, not "real time". If nothing else, if you fire the query before autowarming is completed. In this case you'l

  1   2   3   4   >