Hello Guys
Im using Apache solr 3.6.1 on tomcat 7 for indexing csv files using curl on
windows machine
** My question is that what would be the max csv file size limit when doing
a HTTP POST or while using the following curl command
curl http://localhost:8080/solr/update/csv -F "stream.file=D:\e
Hi - instead of trying to make the system ingest such large files perhaps you
can split the files in many small pieces.
-Original message-
> From:mitra
> Sent: Tue 13-Nov-2012 09:05
> To: solr-user@lucene.apache.org
> Subject: Solr Indexing MAX FILE LIMIT
>
> Hello Guys
>
> Im using
Thankyou
*** I understand that the default size for HTTP POST in tomcat is 2mb can we
change that somehow
so that i dont need to split the 10gb csv into 2mb chunks
curl http://localhost:8080/solr/update/csv -F "stream.file=D:\eighth.csv" -F
"commit=true" -F "optimize=true" -F "encapsulate
Hi all,
We've just updated to SOLR 4.0 production and Zookeeper 3.3.6 from SOLR 4.0
development version circa November 2011. We keep 6 months of data online in
our primary cluster, and archive off old stuff to a slower disk archive
cluster. We used to remove SOLR cores with the following cod
Gerald,
I wonder if you tried to approach BlockJoin for your problem? Can you
afford less frequent updates?
On Wed, Nov 7, 2012 at 5:40 PM, Gerald Blanck wrote:
> Thank you Erick for your reply. I understand that search is not an RDBMS.
> Yes, we do have a huge combinatorial explosion if we
L'ubov',
Yes it does. There were only two long request with huge bodies contains
roughly about 125K docs. You can also check Solr side LogUpdateProcessor
log messages regarding number of requests and docs passed in each.
On Wed, Nov 7, 2012 at 5:26 PM, Lyuba Romanchuk
wrote:
> Hi,
> If I run my
Thanks Yonik.
Should I consider sharding in this case ( actually I have one big index
with replication) ? Or create 2 index (one for search and other for facet
on a different machine) ?
Thanks folks
With love from Paris (it's raining today :(
Le mardi 13 novembre 2012, Yonik Seeley a écrit :
Could you just sort the suggestions at the app level? That is, read them
all into a list and sort before presenting them to the user?
Best
Erick
On Sun, Nov 11, 2012 at 10:52 PM, 徐郑 wrote:
> following is my config , it suggests words well .
> i want to get a sorted result when it suggest, so i
I'd say you are at a point where sharding may well help. But, as others
have suggested, you have other issues to consider first - less memory
for Solr and upgrade to a more modern Solr.
Also, if as Yonik asks only the first query is slow, you can set up a
NewSearcher query in your solrconfig.xml
At a guess you have leftover jars from your earlier installation in your
classpath that are being picked up. I've always found that figuring out how
_that_ happened is...er... "interesting"...
Best
Erick
On Mon, Nov 12, 2012 at 7:44 AM, Adam Neal wrote:
> Hi,
>
> I have been running two multic
Slap them firmly on the wrist if they do?
The Solr admin is really designed with trusted users in mind. There are no
provisions that I know of for securing some of the functions.
Your developers have access to the Solr server through the browser, right?
They can do all of that via URL, see: http:
Hi Erick,
Thanks for the info, I figured out that it was a jar problem earlier today but
I don't think it is an old jar. Both of the instances I ran included the
extraction libraries and it appears that the problem is due to the
xercesImpl-2.9.1.jar. If I remove the extraction tool jars from on
Have you tried the really simple solution of giving your JVM more memory
(-Xmx option)?
Best
Erick
On Tue, Nov 13, 2012 at 2:38 AM, uwe72 wrote:
> Version is 3.6.1 of solr
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/java-io-IOException-Map-failed-OutOfMemory
Have you considered writing a small SolrJ (or other client) program that
processed the rows in your huge file and sent them to solr in sensible
chunks? That would give you much finer control over how the file was
processed, how many docs were sent to Solr at a time, what to do with
errors. You coul
Thanks Eric. We are using:
export JAVA_OPTS="-XX:MaxPermSize=400m -Xmx2000m -Xms200M
-Dsolr.solr.home=/home/connect/ConnectPORTAL/preview/solr-home"
We have arround 5 Millions documents. The index size is arround 50GB.
Before we add a document we delete the same id in the cache, doesn't matter
i
Just a quick comment from our experience: since we have quite a lot of data
indexed in our Solr, we take some extra measures to ensure, no bogus
wild-card queries are accepted by the system (for instance *, **, *** etc).
And that is done in the QueryParser. Wanted to mention this approach as one
wa
Odd...the unload command should be enough...
On Tue, Nov 13, 2012 at 5:26 AM, Gilles Comeau wrote:
> Hi all,
>
> We've just updated to SOLR 4.0 production and Zookeeper 3.3.6 from SOLR 4.0
> development version circa November 2011. We keep 6 months of data online in
> our primary cluster, and
The Overseer isn't mentioned much because it's an implementation
detail that the user doesn't have to really consider.
The Overseer first came about to handle writing the clusterstate.json
file, as a suggestion by Ted Dunning.
Originally, each node would try and update the custerstate.json file
t
Please find reference materials
http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
http://blog.griddynamics.com/2012/08/block-join-query-performs.html
On Tue, Nov 13, 2012 at 3:25 PM, Gerald Blanck <
gerald.bla...@barometerit.com> wrote:
> Thank you. I've not heard
Kernel: 2.6.32.29-0.3-default #1 SMP 2011-02-25 13:36:59 +0100 x86_64
x86_64 x86_64 GNU/Linux
SUSE Linux Enterprise Server 11 SP1 (x86_64)
physical Memory: 4 GB
portadm@smtcax0033:/srv/connect/tomcat/instances/SYSTEST_Portal_01/bin>
java -version
java version "1.6.0_33"
Java(TM) SE Runtime Envi
On Tue, Nov 13, 2012 at 12:22 AM, deniz wrote:
> so do we need to add one of the servers from the -DzkHost string to -DzkRun?
> should it look like
>
> -DzkRun=host1:port -DzkHost=host:port, host1:port, host2:port in the
> start up command?
Yeah, something to that affect.
>
>
> and will wiki
On Tue, Nov 13, 2012 at 12:22 AM, deniz wrote:
> so do we need to add one of the servers from the -DzkHost string to -DzkRun?
By the way - not just any of the servers has to be added to zkRun -
but the address for the current server - that is, the server you are
running the command on. This is so
When I do the unload through the UI, I see the below messages in the solr log.
Nothing in the zookeeper log.
Then right after I try:
http://217.147.83.124:9090/solr/experiment_master/select?q=*%3A*&wt=xml&distrib=true
and get no servers hosting shard:. Also, I still see
the shard bein
Sorry forgot.. pictures are no good.. From cluster.json, the same information,
the core I unloaded shard sticks around:
“"solrexperiment:8080_solr_experiment_02_10_2012":{"replicas":{”
Do I need a special command to delete the shard or something? I’ve never seen
a command that does that?
https://issues.apache.org/jira/browse/SOLR-3993 has been resolved.
Just few question, is it in trunk, I mean in main distrib downloadable on
main solr site.
Because I have downloaded it and get still same behaviour while running
first instance..or second shards.
--
View this message in context
today the same exception:
INFO: [] webapp=/solr path=/update
params={waitSearcher=true&commit=true&wt=javabin&waitFlush=true&version=2}
status=0 QTime=1009
Nov 13, 2012 2:02:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/net/smtcax
Hi,
I'm using solr 4 (4.0.0.2012.03.17.15.05.35) with cloud architecture and
I would use SolrCloudServer from solrJ, but I received a
SolrServerException.
org.apache.solr.client.solrj.SolrServerException: No live SolrServers
available to handle this request
at
org.apache.solr.client.sol
I just saw that you are running on SUSE 11 - unlike RHEL for example, it does
not have virtual memory set to "unlimited" by default.
Please check is the virtual memory limit (ulimit -v, check this for the
operating system user that runs Tomcat /Solr).
Since 3.1, Solr maps the index files to vi
Thanks Andrew!
Parallel i also found this thread:
http://grokbase.com/t/lucene/solr-user/117m8e9n8t/solr-3-3-exception-in-thread-lucene-merge-thread-1
they are talking about the same
We just started the importer again, with the unlimited-flag (/ulimit -v
unlimited /), then we will see.
We're getting close to deploying our Solr search solution, and we're doing
performance testing, and we've run into some questions and concerns.
Our number one problem: Doing a commit from loading records, which can happen
throughout the day, makes all queries stop for 5-7 seconds. This is a
sh
We're getting close to deploying our Solr search solution, and we're doing
performance testing, and we've run into some questions and concerns.
Our number one problem: Doing a commit from loading records, which can happen
throughout the day, makes all queries stop for 5-7 seconds. This is a
sh
Hello,
I would like to send a request to the FieldAnalysisRequestHandler. The
javadoc lists the parameter names such as analysis.field, but sending those
as URL parameters does not seem to work:
mysolr.umich.edu/analysis/field?analysis.name=title&q=fire-fly
leaving out the "analysis" doesn't w
I think the UI uses this behind the scenes, as in no more
"analysis.jsp" like before?
So maybe try using something like burpsuite and just using the
analysis UI in your browser to see what requests its sending.
On Tue, Nov 13, 2012 at 11:00 AM, Tom Burton-West wrote:
> Hello,
>
> I would like t
Andy,
Solr is supposed to serve requests by old searcher for a while. If the
pause lasts few seconds you can take a thread dump and see clear what it
waits for.
Just a guess: if you have many threads configured in servlet container pool
and push high load then warming can significantly impact your
Hi Yonik
I will give the latest 4.0 release a try.
Thanks anyway.
Cheers
Ben
From: ysee...@gmail.com [ysee...@gmail.com] on behalf of Yonik Seeley
[yo...@lucidworks.com]
Sent: Tuesday, November 13, 2012 2:04 PM
To: solr-user@lucene.apache.org
Subject: R
Thanks Robert,
Somehow I read the doc but still entered the params wrong. Should have
been "analysis.fieldname" instead of "analysis.name" Works fine now.
Tom
On Tue, Nov 13, 2012 at 2:11 PM, Robert Muir wrote:
> I think the UI uses this behind the scenes, as in no more
> "analysis.jsp" like
Don't use query time synonyms. Explanation here:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory
wunder
On Nov 13, 2012, at 1:25 PM, dm_tim wrote:
> I'm looking at the sample docs for Solr v4 and I noted something in the
> schema.xml file: The field type us
We do have a custom query parser that is responsible for expanding the user
input query into a bunch of prefix, phrase and regular boolean queries in a
manner similar to that done by DisMax.
Analyzing heap with jhat/YourKit is on my list of things to do but I
haven't gotten around to doing it yet.
Suppose I have a special data search type (something different than a string or
numeric value) that I want to integrate into the Solr server. For example,
suppose I wanted to implement a KD-tree as a filter that would integrate with
standard Solr filters and queries. I might want to say "find
Looks like after timeout has finished, first solr instance respond
I was not waiting enough. Is it possible to reduce this *timeout* value ?
Thanks
--
View this message in context:
http://lucene.472066.n3.nabble.com/Testing-Solr-Cloud-with-ZooKeeper-tp4018900p4020190.html
Sent from the Solr
If so, would you be willing to share the .dic and .aff files with me?
When I try to load a dictionary file, Solr is complaining that:
java.lang.RuntimeException: java.io.IOException: Unable to load hunspell
data! [dictionary=en_GB.dic,affix=en_GB.aff]
at org.apache.solr.schema.IndexSchema.(Ind
Heck,
I originally started using the default query parser but gave up on it
because all of my search results are equally important and idf was messing
up my results pretty badly. So I discovered the DisMax query parser which
doesn't use idf. I was elated until I started testing. My initial results
Good to know. Thanks.
T
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-v4-Synonyms-better-at-index-time-or-query-time-tp4020179p4020198.html
Sent from the Solr - User mailing list archive at Nabble.com.
Thanks Mark. I meant ConcurrentMergeScheduler and ramBufferSizeMB (not
maxBuffer). These are my settings for Merge.
/
960
40
/
--Shreejay
Mark Miller-3 wrote
> On Nov 9, 2012, at 1:20 PM, shreejay <
> shreejayn@
> > wrote:
>
>> Instead of doing an optimize, I have now change
Thank you Mikhail. Unfortunately BlockJoinQuery is not an option we can
leverage.
- We have modeled our document types as different indexes/cores.
- Our relationships which we are attempting to join across are not
single-parent to many-children relationships. They are in fact many to
many.
- Add
Yes, basically I want to at least avoid leader election and the other
dynamic behaviors. I don't have any experience with ZK, and a lot of
"magic" behavior seems baked in now that's I'm concerned I'd need to
dig into SK to debug or monitor what's really happening as we scale
out.
We also have a s
Hi,
Yeah, large heap can be problematic like that. :)
But if there is some sort of a leak, and if I had to bet I'd put my money
on your custom QP knowing what I know about this situation, you could also
start Solr with a much smaller heap and grab the heap snapshot as soon as
you see some number o
Hi,
If you have high query rate, running multiple instances of Solr on the same
server doesn't typically make sense. I'd stop and rethink :)
Otis
--
Solr Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Nov 13, 2012 at 5:46 PM, Rohit Harchandani wrote:
> Hi All,
> I am
Hello Andy,
On Tue, Nov 13, 2012 at 1:26 PM, Andy Lester wrote:
> We're getting close to deploying our Solr search solution, and we're doing
> performance testing, and we've run into some questions and concerns.
>
> Our number one problem: Doing a commit from loading records, which can
> happen
Hi,
Using Solr 1.2.0, the following works (and I get hits searching on Chinese
text):
and it won't work.
I run it through the analyzer and I see this (I hope the table will show up
fine on the mailing list):
Gerald,
Nice to hear the the your problem is solved. Can you contribute a test case
to reproduce this issue?
FWIW, my team successfully deals with Many-to-Many in BlockJoin. It works,
but solution is a little bit immature yet.
On Wed, Nov 14, 2012 at 5:59 AM, Gerald Blanck <
gerald.bla...@barome
51 matches
Mail list logo
