There was nothing like that year ago. Patches are welcome.
On Mon, May 21, 2018 at 6:35 PM, Matteo Grolla
wrote:
> Hi everyone,
> I'm evaluating suggesters that can can be in near real time and I came
> across
> https://issues.apache.org/jira/browse/LUCENE-5477.
> I
Hi everyone,
I'm evaluating suggesters that can can be in near real time and I came
across
https://issues.apache.org/jira/browse/LUCENE-5477.
Is there a way to use this functionality from solr?
Thanks very much
Matteo Grolla
ur and approx 3 Lakh's
of data of would be seached in one hour.This is just the rough value though
nearby,so how can we achieve near -real-Time search in solr ? and how much
percent of real time search would be possible on this large data?
Can we even achieve this doing indexing at certain
inter
I want to use solr for enterprise level search on a large scale of data in
TB, where in Lakh's of data will be update in an hour and approx 3 Lakh's
of data of would be seached in one hour.This is just the rough value though
nearby,so how can we achieve near -real-Time search in solr
>>
>> Otis
>> Solr & ElasticSearch Support
>> http://sematext.com/
>>
>> On May 2, 2013 5:34 AM, "Furkan KAMACI" wrote:
>> >
>> > Does Near Real Time get not supported at SolrCloud?
>> >
>> > I mean when a soft commi
Does soft commits distributes into nodes of SolrCloud?
2013/5/3 Otis Gospodnetic
> NRT works with SolrCloud.
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
>
> On May 2, 2013 5:34 AM, "Furkan KAMACI" wrote:
> >
> > Does Near Real
NRT works with SolrCloud.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On May 2, 2013 5:34 AM, "Furkan KAMACI" wrote:
>
> Does Near Real Time get not supported at SolrCloud?
>
> I mean when a soft commit occurs at a leader I think that it doesn't
> di
Does Near Real Time get not supported at SolrCloud?
I mean when a soft commit occurs at a leader I think that it doesn't
distribute it to replicas(because it is not at storage, does indexes at RAM
distributes to replicas too?) and a search query comes what happens?
most important, "string" facets are much faster
>>than
>> "Date Trie" ranges.
>>
>>
>>
>> 2. Our index is overs 100 millions (from social networks) and rapidly
>>grows
>> (millions a day); cache warm up takes few minutes; Near-R
a day); cache warm up takes few minutes; Near-Real-Time does not
work with faceting.
HoweverS( another workaround: we can have Daily Core (optimized at midnight),
plus Current Core (only today's data, optimized), plus Last Hour Core (near
real time)
"Last Hour Data" is small enough and
gt; index(es)
> for each timezone. And most important, "string" facets are much faster than
> "Date Trie" ranges.
>
>
>
> 2. Our index is overs 100 millions (from social networks) and rapidly grows
> (millions a day); cache warm up takes few minutes; Near-Real-Ti
a day); cache warm up takes few minutes; Near-Real-Time does not
work with faceting.
However another workaround: we can have Daily Core (optimized at midnight),
plus Current Core (only today's data, optimized), plus Last Hour Core (near
real time)
"Last Hour Data" is small enough and
Hi,
You might want to take a look at Solr's trunk (very soon to be 4.0.0
alpha release), which already has a near-real-time solution (using
Lucene's near-real-time APIs).
Lucene has NRTCachingDirectory (to use RAM for small / recently
flushed segments), but I don't think Solr use
Hi,
As part of my bachelor thesis I'm trying to archive NRT with Solr 3.6.
I've came up with a basic concept and would be trilled if I could get
some feedback.
The main idea is to use two different Indexes. One persistent on disc
and one in RAM. The plan is to route every added and modified
r will be used
> primarily for indexing and the queries will be load balanced across to the
> replicated slave servers. I would like to know if, with the current support
> for Near Real Time search in 4.0, there's support for Faceted Search. Keeping
> in mind that the searches
with the current support for
Near Real Time search in 4.0, there's support for Faceted Search. Keeping in
mind that the searches will be performed against the Slave servers and not the
Master (indexing) server.
If it's not supported, will we need to use SolrCloud to gain the benefits of
[?], thanks. Try in a minute.
Spark
2011/12/9 Mark Miller
> pseudo code:
>
>SolrServer client = ...
>
>UpdateRequest ureq = new UpdateRequest();
>ureq.add(doc);
>ureq.setParam("param", "value");
>ureq.setAction( UpdateRequest.ACTION.COMMIT, waitFlush, waitSearcher);
>ure
pseudo code:
SolrServer client = ...
UpdateRequest ureq = new UpdateRequest();
ureq.add(doc);
ureq.setParam("param", "value");
ureq.setAction( UpdateRequest.ACTION.COMMIT, waitFlush, waitSearcher);
ureq.process(client);
It will say setAction is deprecated, but internal So
I will try to use your methods.
2011/12/9 Sharath Jagannath
> Kinda off topic why not directly use /solr/update/json?commitWithin=3000,
> ofcourse with autocommit.maxtime and autoSoftCommit in configs.
> I am not using the java client but send the docs using the http client.
> It is doing fairly
Hi Mark,
My question inline.
2011/12/9 Mark Miller
> I don't know about DataImportHandler.
>
> But the whole point of AutoCommit and AutoSoftCommit is that they happen
> 'Auto'matically. You couldn't measure how long they take from solrj, and if
> you are using solrj to do the commit, it has no
Kinda off topic why not directly use /solr/update/json?commitWithin=3000,
ofcourse with autocommit.maxtime and autoSoftCommit in configs.
I am not using the java client but send the docs using the http client.
It is doing fairly good for me.
Zoie is another option worth looking at. Had very good e
I don't know about DataImportHandler.
But the whole point of AutoCommit and AutoSoftCommit is that they happen
'Auto'matically. You couldn't measure how long they take from solrj, and if
you are using solrj to do the commit, it has nothing to do with auto commit.
If you want to commit from solrj
That sounds like a good idea. I will check my schema configuration, and see
what I can do.
Thanks Mark and Siva for all the information, and see what I can do, and
update the result here.
Spark
2011/12/9 Siva Kommuri
> Hi Spark,
>
> Is it possible to store a modified time on the database recor
Hi Spark,
Is it possible to store a modified time on the database record which would help
you to get the updated documents?
That way you can avoid reindexing the documents that have not been updated.
Best wishes,
Siva on 3GS
On Dec 8, 2011, at 15:40, yu shen wrote:
> Hi Mark,
>
> I did not
Hi Mark,
I did not fully catch you.
I have a dataImportHandler which import data from a database. Are you
suggesting autoCommit/autoSoftCommit can automaticall pull data from
database using dataImportHandler?
Otherwise, how can I commit user modification of search result from my web
page back to
Lets please keep the conversation on list.
If you are using auto commit and soft auto commit, it makes no sense to be
committing with solrj. The commits happen automatically. You couldn't measure
them with solrj.
- mark
On Dec 8, 2011, at 12:44 AM, yu shen wrote:
> Thanks for the help.
>
> F
Hmmm...that sounds pretty odd...
How are you measuring the commit time?
You likely want to turn off any caches, as they will be expired every second,
but that should not cause this...
I can try and duplicate your setup tomorrow and see what i can spot.
- Mark
On Dec 7, 2011, at 8:13 PM, yu sh
Hi Mark, and all
I now use commit configuration exactly as below:
10
1000
But the commit time takes about 60 seconds.
I have around 120 - 130 documents in my server. And each day, the
number will increase about 6000. My symptom is if solr server is just
s
Thanks for the correction, I did not notice that [?]
Spark
2011/12/7 Mark Miller
> Well, if that is exactly what you put, it's wrong. That second one should
> be softAutoCommit.
>
> On Wednesday, December 7, 2011, yu shen wrote:
> > Hi All,
> >
> > I tried using solr 4 nightly build: apache-s
Well, if that is exactly what you put, it's wrong. That second one should
be softAutoCommit.
On Wednesday, December 7, 2011, yu shen wrote:
> Hi All,
>
> I tried using solr 4 nightly build: apache-solr-4.0-2011-12-06_08-52-46.
> And try to enable autoSoftCommit like below in solrconfig.xml
>
>
Hi All,
I tried using solr 4 nightly build: apache-solr-4.0-2011-12-06_08-52-46.
And try to enable autoSoftCommit like below in solrconfig.xml
10
1000
I try to add a document to this solr instance using solrj client in the
nightly build. I do saw a commit time boost. Single docume
This is great! I guess, there is nothing left to worry about for a while.
Erick & Yonik, thank you again for your great responses.
Bests,
Jak
On Thu, Nov 17, 2011 at 4:01 PM, Yonik Seeley wrote:
> On Thu, Nov 17, 2011 at 3:56 PM, Jak Akdemir wrote:
> > Is it ok to see soft committed records af
On Thu, Nov 17, 2011 at 3:56 PM, Jak Akdemir wrote:
> Is it ok to see soft committed records after server restart, too?
Yes... we currently have Jetty configured to call some cleanups on
exit (such as closing the index writer).
-Yonik
http://www.lucidimagination.com
Yonik,
Is it ok to see soft committed records after server restart, too? If it is,
there is no problem left at all.
I added changing files and 1 sec of log at the end of the e-mail. One
significant line says softCommit=true, so Solr recognizes our softCommit
request.
INFO: start
commit(optimize=fa
On Thu, Nov 17, 2011 at 1:34 PM, Erick Erickson wrote:
> Hmmm. It is suspicious that your index files change every
> second.
Why is this suspicious?
A soft commit still writes out some files currently... it just doesn't
fsync them.
-Yonik
http://www.lucidimagination.com
1- There is an improvement on the issue. I add 10 seconds time interval
into the delta of data-config.xml, which will cover records that already
indexed.
"revision_time > DATE_SUB('${dataimporter.last_index_time}', INTERVAL 10
SECOND);"
In this case 1369 new records inserted with 7 records per sec
Hmmm. It is suspicious that your index files change every
second. If you change our cron task to update every 10
seconds, do the index files change every 10 seconds?
Regarding your question about
"After a server restart last query results reserved. (In NRT they would
disappear, right?)"
not necess
Yonik,
I updated my solrconfig time based only as follows.
30
1000
And changed my soft commit script to the first case.
while [ 1 ]; do
echo "Soft commit applied!"
wget -O /dev/null '
http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=fa
On Thu, Nov 17, 2011 at 11:48 AM, Jak Akdemir wrote:
> 2) I am sure about delta-queries configured well. Full-Import is completed
> in 40 secs for 40 docs. And delta's are in 1 sec for 15 new records.
> Also I checked it. There is no problem in it.
That's 10,000 docs/sec. If you configure a
re-indexing *everything* every time? There's an interactive
> debugging console you can use that may help, try:
> http://localhost:8983/solr/admin/dataimport.jsp
>
> Best
> Erick
>
> On Thu, Nov 17, 2011 at 3:19 AM, Jak Akdemir wrote:
> > Hi,
> >
> > I wa
every time? There's an interactive
debugging console you can use that may help, try:
http://localhost:8983/solr/admin/dataimport.jsp
Best
Erick
On Thu, Nov 17, 2011 at 3:19 AM, Jak Akdemir wrote:
> Hi,
>
> I was trying to configure a Solr instance with the near real-time search
>
Hi,
I was trying to configure a Solr instance with the near real-time search
and auto-complete capabilities. I stuck in the NRT feature. There are
15 new records per second that inserted into the database (mysql) and I
indexed them with DIH. First, I tried to manage autoCommits from
yya wrote:
> From: Nagendra Nagarajayya
> Subject: [Announce[ White paper describing Near Real Time Implementation with
> Solr and RankingAlgorithm
> To: solr-user@lucene.apache.org
> Date: Thursday, May 19, 2011, 9:18 AM
> Hi!
>
> I would like to announce a white paper that
Hi!
I would like to announce a white paper that describes the technical
details of Near Real Time implementation with Solr and the
RankingAlgorithm. The paper discusses the modifications made to enable NRT.
You can download the white paper from here:
http://solr-ra.tgels.com/papers
Hi!
I would like to announce Solr with RankingAlgorithm has Near Real Time
functionality now. The NRT functionality allows you to add documents
without the IndexSearchers being closed or caches being cleared. A
commit is not needed with the document update. Searches can run
concurrently with
> no, I only thought you use one day :-)
> so you don't or do you have 31 shards?
>
No, we use 1 shard per month - e.g. 7 shards will hold 7 month's of data.
It can be set to 1 day, but you would need to have a huge amount of
data in a single day to warrant doing that.
On Thu, Nov 18, 2010 at 8
Does yours need to be once a day?
no, I only thought you use one day :-)
so you don't or do you have 31 shards?
having a look at Solr Cloud or Katta - could be useful
here in dynamically allocating shards.
ah, thx! I will take a look at it (after trying solr4)!
Regards,
Peter.
May
> Maybe I didn't fully understood what you explained: but doesn't this mean
> that you'll have one index per day?
> Or are you overwriting, via replicating, every shard and the number of shard
> is fixed?
> And why are you replicating from the local replica to the next shard? (why
> not directly fr
Hi Peter!
* I believe the NRT patches are included in the 4.x trunk. I don't
think there's any support as yet in 3x (uses features in Lucene 3.0).
I'll investage how much effort it is to update to solr4
* For merging, I'm talking about commits/writes. If you merge while
commits are going on
* I believe the NRT patches are included in the 4.x trunk. I don't
think there's any support as yet in 3x (uses features in Lucene 3.0).
* For merging, I'm talking about commits/writes. If you merge while
commits are going on, things can get a bit messy (maybe on source
cores this is ok, but I hav
Hi Peter,
thanks for your response. I will dig into the sharding stuff asap :-)
This may have changed recently, but the NRT stuff - e.g. per-segment
commits etc. is for the latest Solr 4 trunk only.
Do I need to turn something 'on'?
Or do you know wether the NRT patches are documented some
Hi Peter,
First off, many thanks for putting together the NRT Wiki page!
This may have changed recently, but the NRT stuff - e.g. per-segment
commits etc. is for the latest Solr 4 trunk only.
If your setup uses the 3x Solr code branch, then there's a bit of work
to do to move to the new version.
Hi,
I wanted to provide my indexed docs (tweets) relative fast: so 1 to 10
sec or even 30 sec would be ok.
At the moment I am using the read only core scenario described here
(point 5)*
with a commit frequency of 180 seconds which was fine until some days.
(I am using solr1.4.1)
Now the time
On Wed, Oct 21, 2009 at 10:19 PM, George Aroush wrote:
>> Depends a lot on the nature of the requests and the size of the index,
>> but one minute is often doable.
>> On a large index that facets on many fields per request, one minute is
>> probably still out of reach.
>
> With no facets, what ind
> > Further without the NRT features present what's the closest I can
> > expect to real time for the typical use case (obviously this will vary
> > but the average deploy). One hour? One Minute? It seems like there are
> > a few hacks to get somewhat close. Thanks so much.
>
> Depends a lot o
Thanks, that's great to know.
-Matt
--- On Wed, 10/21/09, Yonik Seeley wrote:
From: Yonik Seeley
Subject: Re: Near Real Time
To: solr-user@lucene.apache.org
Date: Wednesday, October 21, 2009, 6:57 PM
On Wed, Oct 21, 2009 at 6:35 PM, Matthew Rushton wrote:
> I'm investigating s
On Wed, Oct 21, 2009 at 6:35 PM, Matthew Rushton wrote:
> I'm investigating several search indexing options and solr looks great but
> I have a few questions I couldn't find answers to. To begin I don't think the
> near real time features of Lucene are a requirement
Hi,
I'm investigating several search indexing options and solr looks great but I
have a few questions I couldn't find answers to. To begin I don't think the
near real time features of Lucene are a requirement to start but I do think it
will be a requirement moving forward and I
running into memory issues?
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > - Original Message
> >> From: Mark Ferguson
> >> To: solr-user@lucene.apache.org
> >> Sent:
t.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message
>> From: Mark Ferguson
>> To: solr-user@lucene.apache.org
>> Sent: Friday, February 20, 2009 1:14:15 AM
>> Subject: Near real-time search of user data
>>
>> Hi,
>>
>> I a
http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
> From: Mark Ferguson
> To: solr-user@lucene.apache.org
> Sent: Friday, February 20, 2009 1:14:15 AM
> Subject: Near real-time search of user data
>
> Hi,
>
> I am trying to come up with a strateg
Hi,
I am trying to come up with a strategy for a solr setup in which a user's
indexed data can be nearly immediately available to them for search. My
current strategy (which is starting to cause problems) is as follows:
- each user has their own personal index (core), which gets committed
after
62 matches
Mail list logo
