On 2/15/2018 2:00 AM, Srinivas Kashyap wrote:
> I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the
> child entities in data-config.xml. And i'm using the same for full-import
> only. And in the beginning of my implementation, i had written delta-import
> query to index th
Srinivas:
Not an answer to your question, but when DIH starts getting this
complicated, I start to seriously think about SolrJ, see:
https://lucidworks.com/2012/02/14/indexing-with-solrj/
IN particular, it moves the heavy lifting of acquiring the data from a
Solr node (which I'm assuming also has
Hi,
I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the
child entities in data-config.xml. And i'm using the same for full-import only.
And in the beginning of my implementation, i had written delta-import query to
index the modified changes. But my requirement grew and i
Hi Erick,
As suggested, I did try nonHDFS solr cloud instance and it response looks to
be really better. From the configuration side to, I am mostly using default
configurations and with block.cache.direct.memory.allocation as false. On
analysis of hdfs cache, evictions seems to be on higher sid
Hi Arun,
It is hard to measure something without affecting it, but we could use debug
results and combine with QTime without debug: If we ignore merging results, it
seems that majority of time is spent for retrieving docs (~500ms). You should
consider reducing number of rows if you want better r
Hi Emir,
Please find the response without bq parameter and debugQuery set to true.
Also it was noted that Qtime comes down drastically without the debug
parameter to about 700-800.
true
0
3446
("hybrid electric powerplant" "hybrid electric powerplants" "Electric"
"Electrical" "Electricity"
Hi Erick,
Qtime comes down with rows set as 1. Also it was noted that qtime comes down
when debug parameter is not added with the query. It comes to about 900.
Thanks,
Arun
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
On Tue, 2017-09-26 at 07:43 -0700, sasarun wrote:
> Allocated heap size for young generation is about 8 gb and old
> generation is about 24 gb. And gc analysis showed peak
> size utlisation is really low compared to these values.
That does not come as a surprise. Your collections would normally b
Hi Arun,
This is not the most simple query either - a dozen of phrase queries on several
fields + the same query as bq. Can you provide debugQuery info.
I did not look much into debug times and what includes what, but one thing that
is strange to me is that QTime is 4s while query in debug is 1.3
Well, 15 second responses are not what I'd expect either. But two
things (just looked again)
1> note that the time to assemble the debug information is a large
majority of your total time (14 of 15.3 seconds).
2> you're specifying 600 rows which is quite a lot as each one
requires that a 16K block
Hi Erick,
Thank you for the quick response. Query time was relatively faster once it
is read from memory. But personally I always felt response time could be far
better. As suggested, We will try and set up in a non HDFS environment and
update on the results.
Thanks,
Arun
--
Sent from: http
Does the query time _stay_ low? Once the data is read from HDFS it
should pretty much stay in memory. So my question is whether, once
Solr warms up you see this kind of query response time.
Have you tried this on a non HDFS system? That would be useful to help
figure out where to look.
And given
Hi All,
I have been using Solr for some time now but mostly in standalone mode. Now
my current project is using Solr 6.5.1 hosted on hadoop. My solrconfig.xml
has the following configuration. In the prod environment the performance on
querying seems to really slow. Can anyone help me with few poin
> Also we will try to decouple tika to solr.
+1
-Original Message-
From: tstusr [mailto:ulfrhe...@gmail.com]
Sent: Friday, March 31, 2017 4:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr performance issue on indexing
Hi, thanks for the feedback.
Yes, it is about OOM, ind
decouple tika to
> solr.
>
> By the way, make it available with solr cloud will improve performance? Or
> there will be no perceptible improvement?
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html
> Sent from the Solr - User mailing list archive at Nabble.com.
ble improvement?
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html
Sent from the Solr - User mailing list archive at Nabble.com.
with 4gb of JVM
> Memory and ~50gb of physical memory (reported through dashboard) we are
> using a single instance.
>
> I don't think is a normal behaviour that handler crashes. So, what are some
> general tips about improving performance for this scenario?
>
>
>
> --
andler crashes. So, what are some
general tips about improving performance for this scenario?
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886.html
Sent from the Solr - User mailing list archive at Nabble.com.
1 million document isn't considered big for Solr. How much RAM does your
machine have?
Regards,
Edwin
On 8 February 2016 at 23:45, Susheel Kumar wrote:
> 1 million document shouldn't have any issues at all. Something else is
> wrong with your hw/system configuration.
>
> Thanks,
> Susheel
>
>
1 million document shouldn't have any issues at all. Something else is
wrong with your hw/system configuration.
Thanks,
Susheel
On Mon, Feb 8, 2016 at 6:45 AM, sara hajili wrote:
> On Mon, Feb 8, 2016 at 3:04 AM, sara hajili wrote:
>
> > sorry i made a mistake i have a bout 1000 K doc.
> > i
On Mon, Feb 8, 2016 at 3:04 AM, sara hajili wrote:
> sorry i made a mistake i have a bout 1000 K doc.
> i mean about 100 doc.
>
> On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic <
> emir.arnauto...@sematext.com> wrote:
>
>> Hi Sara,
>> Not sure if I am reading this right, but I read it as you
Hi Sara,
It is still considered to be small index. Can you give us bit details
about your setup?
Thanks,
Emir
On 08.02.2016 12:04, sara hajili wrote:
sorry i made a mistake i have a bout 1000 K doc.
i mean about 100 doc.
On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic <
emir.arnauto...@s
sorry i made a mistake i have a bout 1000 K doc.
i mean about 100 doc.
On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic <
emir.arnauto...@sematext.com> wrote:
> Hi Sara,
> Not sure if I am reading this right, but I read it as you have 1000 doc
> index and issues? Can you tell us bit more about
Hi Sara,
Not sure if I am reading this right, but I read it as you have 1000 doc
index and issues? Can you tell us bit more about your setup: number of
servers, hw, index size, number of shards, queries that you run, do you
index at the same time...
It seems to me that you are running Solr on
hi all.
i have a problem with my solr performance and usage hardware like a
ram,cup...
i have a lot of document and so indexed file about 1000 doc in solr that
every doc has about 8 field in average.
and each field has about 60 char.
i set my field as a storedfield = "false" except of 1 field. //
Thanks Furkan. Looking forward to seeing your test results.
Sent from Yahoo Mail on Android
Hi Hien;
Actually high index rate is a relative concept. I could index such kind of
data within a few hours. I aim to index much much more data within same
time soon. I can share my test results when I do.
Thanks;
Furkan KAMACI
6 Aralık 2013 Cuma tarihinde Hien Luu adlı kullanıcı şöyle
yazdı:
>
On 12/5/2013 4:08 PM, Hien Luu wrote:
Just curious what was the index rate that you were able to achieve?
What I've usually seen based on my experience and what people have said
here and on IRC is that the data source is usually the bottleneck - Solr
typically indexes VERY fast, as long as yo
Hi Furkan,
Just curious what was the index rate that you were able to achieve?
Regards,
Hien
On Thursday, December 5, 2013 3:06 PM, Furkan KAMACI
wrote:
Hi;
Erick and Shawn have explained that we need more information about your
infrastructure. I should add that: I had test data at my
Hi;
Erick and Shawn have explained that we need more information about your
infrastructure. I should add that: I had test data at my SolrCloud nearly
as much as yours and I did not have any problems except for when indexing
at a huge index rate and it can be solved with turning. You should optimiz
On 12/4/2013 6:31 AM, kumar wrote:
> I am having almost 5 to 6 crores of indexed documents in solr. And when i am
> going to change anything in the configuration file solr server is going
> down.
If you mean crore and not core, then you are talking about 50 to 60
million documents. That's a lot.
Exception: Cannot call sendError() after
> the response has been committed] with root cause
>
>
>
> Can anybody help me how can i solve this problem.
>
> Kumar.
>
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-Performance-Issue-tp4104907.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
threw
exception [java.lang.IllegalStateException: Cannot call sendError() after
the response has been committed] with root cause
Can anybody help me how can i solve this problem.
Kumar.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-Performance-Issue-tp4104907.html
Sent from
Hello,
The problem turned out to be some sort of sharding/searching weirdness. We
modified some code in sharding but I don't think it is related. In any case,
we just added a new server that just shards (but doesn't do any searching /
doesn't contain any index) and performance is very very good.
> Btw, I am monitoring output via jconsole with 8gb of ram and it still goes
> to 8gb every 20 seconds or so,
> gc runs, falls down to 1gb.
Hmm, jvm is eating 8Gb for 20 seconds - sounds a lot.
Do you return all results (ids) for your queries? Any tricky
faceting/sorting/function queries?
The host is dual quad-core, each Xen VM has been given two CPUs. Not
counting dom0, two of the hosts have 10/8 CPUs allocated, two of them
have 8/8. The dom0 VM is also allocated two CPUs.
I'm not really sure how that works out when it comes to Java running on
the VM, but if at all possible,
CMS is very good for multicore CPU's. Use incremental mode only when you have
a single CPU with only one or two cores.
On Tuesday 15 March 2011 16:03:38 Shawn Heisey wrote:
> My solr+jetty+java6 install seems to work well with these GC options.
> It's a dual processor environment:
>
> -XX:+UseCo
My solr+jetty+java6 install seems to work well with these GC options.
It's a dual processor environment:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
I've never had a real problem with memory, so I've not done any kind of
auditing. I probably should, but time is a limited resource.
Shaw
2011/3/14 Markus Jelsma
> Mmm. SearchHander.handleRequestBody takes care of sharding. Could your
> system
> suffer from
> http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock
> ?
>
>
We increased thread limit (which was 1 before) but it did not help.
Anyway, we will try to disa
Mmm. SearchHander.handleRequestBody takes care of sharding. Could your system
suffer from http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock
?
I'm not sure, i haven't seen a similar issue in a sharded environment,
probably because it was a controlled environment.
> Hello,
>
>
Hello,
2011/3/14 Markus Jelsma
> That depends on your GC settings and generation sizes. And, instead of
> UseParallelGC you'd better use UseParNewGC in combination with CMS.
>
>
JConsole now shows a different profile output but load is still high and
performance is still bad.
Btw, here is the t
That depends on your GC settings and generation sizes. And, instead of
UseParallelGC you'd better use UseParNewGC in combination with CMS.
See 22: http://java.sun.com/docs/hotspot/gc1.4.2/faq.html
> It's actually, as I understand it, expected JVM behavior to see the heap
> rise to close to it's
You might also want to add the following switches for your GC log.
> JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps
> -XX:+PrintGCDetails - Xloggc:/var/log/tomcat6/gc.log"
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
>
> Also, what JVM version are you using
It's actually, as I understand it, expected JVM behavior to see the heap
rise to close to it's limit before it gets GC'd, that's how Java GC
works. Whether that should happen every 20 seconds or what, I don't nkow.
Another option is setting better JVM garbage collection arguments, so GC
doesn
> Nope, no OOM errors.
That's a good start!
> Insanity count is 0 and fieldCAche has 12 entries. We do use some boosting
> functions.
>
> Btw, I am monitoring output via jconsole with 8gb of ram and it still goes
> to 8gb every 20 seconds or so,
> gc runs, falls down to 1gb.
Hmm, maybe the garb
Hello again,
2011/3/14 Markus Jelsma
> > Hello,
> >
> > 2011/3/14 Markus Jelsma
> >
> > > Hi Doğacan,
> > >
> > > Are you, at some point, running out of heap space? In my experience,
> > > that's the common cause of increased load and excessivly high response
> > > times (or time
> > > outs).
>
I've definitely had cases in 1.4.1 where even though I didn't have an
OOM error, Solr was being weirdly slow, and increasing the JVM heap size
fixed it. I can't explain why it happened, or exactly how you'd know
this was going on, I didn't see anything odd in the logs to indicate, I
just tried
> Hello,
>
> 2011/3/14 Markus Jelsma
>
> > Hi Doğacan,
> >
> > Are you, at some point, running out of heap space? In my experience,
> > that's the common cause of increased load and excessivly high response
> > times (or time
> > outs).
>
> How much of a heap size would be enough? Our index si
Hello,
2011/3/14 Markus Jelsma
> Hi Doğacan,
>
> Are you, at some point, running out of heap space? In my experience, that's
> the common cause of increased load and excessivly high response times (or
> time
> outs).
>
>
How much of a heap size would be enough? Our index size is growing slowly
b
Hi Doğacan,
Are you, at some point, running out of heap space? In my experience, that's
the common cause of increased load and excessivly high response times (or time
outs).
Cheers,
> Hello everyone,
>
> First of all here is our Solr setup:
>
> - Solr nightly build 986158
> - Running solr in
Hello everyone,
First of all here is our Solr setup:
- Solr nightly build 986158
- Running solr inside the default jetty comes with solr build
- 1 write only Master , 4 read only Slaves (quad core 5640 with 24gb of RAM)
- Index replicated (on optimize) to slaves via Solr Replication
- Size of ind
51 matches
Mail list logo
