Solr 7.7 Indexing issue

Hello all We are using Apache Solr 7.7 on Windows platform. The data is synced to Solr using Solr.Net commit. The data is being synced to SOLR in batches. The document size is very huge (~0.5GB average) and solr indexing is taking long time. Total document size is ~200GB. As the solr commit is

Re: Solr 8.5.2 indexing issue

It seems that the issue is not with reference_url field itself. There is one copy field which has the reference_url field as source and another field called url_path as destination. This destination field url_path has the following field type definition.

Re: Solr 8.5.2 indexing issue

How are you sending this to Solr? I just tried 8.5, submitting that doc through the admin UI and it works fine. I defined “asset_id” with as the same type as your reference_url field. And does the log on the Solr node that tries to index this give any more info? Best, Erick > On Jun 27, 2020,

Solr 8.5.2 indexing issue

Hi, I have the following document which fails to get indexed. { "asset_id":"add-ons:576deefef7453a9189aa039b66500eb2", "reference_url":"modeling-a-high-speed-backplane-part-3-4-port-s-parameters-to-differential-tdr-and-tdt.html"} I am not sure what is so special about the content in the

Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

Thanks Shawn for checking. As advised we will execute the indexing with the new settings as mentioned and will update the results. Here are the links to missing attachments: Attachment 1: OpenJDK 11 vs OpenJDK 8 key metrics

Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

On 10/24/2019 11:50 AM, Junagade, Anup wrote: * Attachment 1: OpenJDK 8 vs OpenJDK 8 key metrics * Attachment 2: OpenJDK 8 vs OpenJDK 8 waiting QTP Threads * Attachment 3: OpenJDK 11 Thread dump There are no attachments. Apache mailing lists swallow almost all attachments. Yo

Solr 8.1.1 Indexing issue while migrating Java8 -> Java11

We are trying to migrate our SOLR 8.1.1 cluster from OpenJDK Java 8 to OpenJDK Java 11 and are facing issues with Indexing. While our indexing is happening flawlessly on Java 8, it crawls or maybe I should say it stalls with Java 11. Any pointers/help is appreciated. *Symptoms* With OpenJDK 1

Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

We are trying to migrate our SOLR 8.1.1 cluster from OpenJDK Java 8 to OpenJDK Java 11 and are facing issues with Indexing. While our indexing is happening flawlessly on Java 8, it crawls or maybe I should say it stalls with Java 11. Any pointers/help is appreciated. Symptoms * With OpenJD

Re: Regarding pdf indexing issue

Walter, Well said.  (And I love the hamburger conversion analogy - very apt.) The only thing I will add is that when you have a collection of similar rich text documents, you might be able to construct queries to respect internal structures within the documents.  If all/most of your documents hav

Re: Regarding pdf indexing issue

You may try to use tesseract tool to check data extraction from pdf or images and then go forward accordingly. As far as I understand the PDF is an image and not data. The searchable PDF actually overlays the selectable text as hidden text over the PDF image. These PDFs can be indexed and extracted

Re: Regarding pdf indexing issue

PDF is not a structured document format. It is a printer control format. PDF does not have a paragraph marker. Instead, it says to move to this spot on the page, choose this font, and print this letter. For a paragraph, it moves farther. For the next letter in a word, it moves a little bit. Extra

Re: Regarding pdf indexing issue

Solr will not do this automatically, the Extracting Request Handler simply indexes the entire contents of the doc without regard to things like paragraphs etc. Ditto with HTML. This is actually a task that requires getting into Tika and using all the bells and whistles there. I'd recommend two thi

Regarding pdf indexing issue

Hello Team, I am using the Solr for indexing and searching for pdf document I have go through with your website document and installed solr but unable to index and search the document. For example: Suppose we have a PDF file which have no of paragraph with separate heading. So If I search for t

Re: Indexing issue - index get deleted

Hi Chris, Amazing Analysis ! I did actually not investigated the log, because I was first trying to get more information from the user. "We are running full import and delta import crons . Fulll index once a day delta index : every 10 mins last night my index automatically deleted(numdocs=0).

Re: Indexing issue - index get deleted

Thanks . for replying .. please find the data-config On Thu, Jun 11, 2015 at 6:06 AM, Chris Hostetter wrote: > > : The guys was using delta import anyway, so maybe the problem is > : different and not related to the clean. > > that's not what the logs say. > > Here's what i see... > > Log beg

Re: Indexing issue - index get deleted

: The guys was using delta import anyway, so maybe the problem is : different and not related to the clean. that's not what the logs say. Here's what i see... Log begins with server startup @ "Jun 10, 2015 11:14:56 AM" The DeletionPolicy for the "shopclue_prod" core is initialized at "Jun 10,

Re: Indexing issue - index get deleted

Just taking a look to the code : " if (requestParams.containsKey("clean")) { clean = StrUtils.parseBool( (String) requestParams.get("clean"), true); } else if (DataImporter.DELTA_IMPORT_CMD.equals(command) || DataImporter.IMPORT_CMD.equals(command)) { clean = false; } else { clean = debug ?

Re: Indexing issue - index get deleted

I was only speaking about full import regarding the default of clean=true. However, looking at the source code, it doesn't seem to differentiate especially between a full and a delta in relation to the default of clean=true, which would be pretty crappy. However, I'd need to try it. Upayavira On

Re: Indexing issue - index get deleted

Wow, Upaya, I didn't know that clean was default=true in the delta import as well! I did know it was default in the full import, but I agree with you that having a default to true for delta import is very dangerous ! But assuming the user was using the delta import so far, if cleaning every time,

Re: Indexing issue - index get deleted

Let me answer in line, to get more info : 2015-06-10 10:59 GMT+01:00 Midas A : > Hi Alessandro, > > Please find the answers inline and help me out to figure out this problem. > > 1) Solr version : *4.2.1* > 2) Solr architecture :* Master -slave/ Replication with requestHandler* > > Where happene

Re: Indexing issue - index get deleted

Note the clean= parameter to the DIH. It defaults to true. It will wipe your index before it runs. Perhaps it succeeded at wiping, but failed to connect to your database. Hence an empty DB? clean=true is, IMO, a very dangerous default option. Upayavira On Wed, Jun 10, 2015, at 10:59 AM, Midas A

Re: Indexing issue - index get deleted

Hi Alessandro, Please find the answers inline and help me out to figure out this problem. 1) Solr version : *4.2.1* 2) Solr architecture :* Master -slave/ Replication with requestHandler* 3) Kind of data source indexed : *Mysql * 4) What happened to the datasource ? any change in there ? : *No c

Re: Indexing issue - index get deleted

Let me try to help you, first of all I would like to encourage people to post more information about their scenario than "This is my log, index deleted, help me" :) This kind of Info can be really useful : 1) Solr version 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual Shard

Re: indexing issue

we are indexing around 5 docs par 10 min . On Thu, Jun 4, 2015 at 11:02 PM, Midas A wrote: > Shwan, > > Please find the log . give me some sense what is happening > > On Thu, Jun 4, 2015 at 10:56 PM, Shawn Heisey wrote: > >> On 6/4/2015 11:12 AM, Midas A wrote: >> > sorry Shawn , >> > >> >

Re: indexing issue

Shwan, Please find the log . give me some sense what is happening On Thu, Jun 4, 2015 at 10:56 PM, Shawn Heisey wrote: > On 6/4/2015 11:12 AM, Midas A wrote: > > sorry Shawn , > > > > a) Total docs solr is handling is 3 million . > > b) index size is only 5 GB > > If your total index size is on

Re: indexing issue

On 6/4/2015 11:12 AM, Midas A wrote: > sorry Shawn , > > a) Total docs solr is handling is 3 million . > b) index size is only 5 GB If your total index size is only 5GB, then there should be no need for a 30GB heap. For that much index, I'd start with 4GB, and implement GC tuning. A high iowait

Re: indexing issue

AM, Midas A wrote: > >>> I have some indexing issue . While indexing IOwait is high in solr > server > >>> and load also. > >> My first suspect here is that you don't have enough RAM for your index > >> size. > >> > >> * How many tot

Re: indexing issue

On 6/4/2015 7:38 AM, Midas A wrote: > On Thu, Jun 4, 2015 at 6:48 PM, Shawn Heisey wrote: > >> On 6/4/2015 5:15 AM, Midas A wrote: >>> I have some indexing issue . While indexing IOwait is high in solr server >>> and load also. >> My first suspect here is tha

Re: indexing issue

Hi shawn, Please find comment in line. On Thu, Jun 4, 2015 at 6:48 PM, Shawn Heisey wrote: > On 6/4/2015 5:15 AM, Midas A wrote: > > I have some indexing issue . While indexing IOwait is high in solr server > > and load also. > > My first suspect here is that you don'

Re: indexing issue

On 6/4/2015 5:15 AM, Midas A wrote: > I have some indexing issue . While indexing IOwait is high in solr server > and load also. My first suspect here is that you don't have enough RAM for your index size. * How many total docs is Solr handling (all cores)? * What is the total size

Re: indexing issue

Hi Alessandro, On Thu, Jun 4, 2015 at 5:19 PM, Alessandro Benedetti < benedetti.ale...@gmail.com> wrote: > Honestly your auto-commit configuration seems not alarming at all! > Can you give me more details regarding : > > Load expected : currently it is 7- 15 should be below 1 > *[Abhishek] : s

Re: indexing issue

Honestly your auto-commit configuration seems not alarming at all! Can you give me more details regarding : Load expected : currently it is 7- 15 should be below 1 What does this mean ? Without a unit of measure i find hard to understand plain numbers :) was expecting the number of documents per

Re: indexing issue

ch ? > When does your problem happens ? > > More detail we give, easier will be to provide help. > > Cheers > > 2015-06-04 12:19 GMT+01:00 Toke Eskildsen : > > > On Thu, 2015-06-04 at 16:45 +0530, Midas A wrote: > > > I have some indexing issue . While

Re: indexing issue

Thanks for replying below is commit frequency 6 false 60 On Thu, Jun 4, 2015 at 4:49 PM, Toke Eskildsen wrote: > On Thu, 2015-06-04 at 16:45 +0530, Midas A wrote: > > I have some indexing issue . While indexing IOwait is high in solr server > > and load als

Re: indexing issue

015-06-04 at 16:45 +0530, Midas A wrote: > > I have some indexing issue . While indexing IOwait is high in solr server > > and load also. > > Might be because you commit too frequently. How often do you do that? > > - Toke Eskildsen, State and

Re: indexing issue

On Thu, 2015-06-04 at 16:45 +0530, Midas A wrote: > I have some indexing issue . While indexing IOwait is high in solr server > and load also. Might be because you commit too frequently. How often do you do that? - Toke Eskildsen, State and University Library, Denmark

indexing issue

I have some indexing issue . While indexing IOwait is high in solr server and load also.

solr parallel update and total indexing Issue

's happening is that in between commit is taken place... i.e untill total indexer runs i got limited records(1000). How to solve this ? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-parallel-update-and-total-indexing-Issue-tp4132652.html Sent from the Solr - User mailing list archive at Nabble.com.

Solr Full Indexing issue (solr-user@lucene.apache.org)

I've shared an item with you: Solr Full Indexing issue https://drive.google.com/folderview?id=0B7UpFqsS5lSjWEhxRE1NN2tMNTQ&usp=sharing&invite=CJXE8q4O It's not an attachment -- it's stored online. To open this item, just click the link above.

Re: solr parallel update and total indexing Issue

indexer) what's > happening is that in between commit is taken place... i.e untill total > indexer runs i got limited records(1000). > > How to solve this ? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/solr-parallel-update-and-total-indexing-Issue-tp4131935.html > Sent from the Solr - User mailing list archive at Nabble.com.

solr parallel update and total indexing Issue

happening is that in between commit is taken place... i.e untill total indexer runs i got limited records(1000). How to solve this ? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-parallel-update-and-total-indexing-Issue-tp4131935.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: indexing issue

very possible segment of > '8E0061123-8E1' but just some of them. > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/indexing-issue-tp4009122p4009372.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: indexing issue

user@lucene.apache.org Subject: indexing issue Dear fellows, I have a field in solr with value '8E0061123-8E1'. Now when i seach '8E*', it does return me all values starting with'8E' which is totally right but it returns nothing when i search '8E0*'. I gues

Re: indexing issue

#x27;, '8E006' > etc. But currently it return result only when i type 8E or comeplete > ''8E0061123-8E1'...any idea?? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/indexing-issue-tp4009122.html > Sent from the Solr - User mailing list archive at Nabble.com.

indexing issue

h with all combinations likes '8E', '8E0', '8E00', '8E006' etc. But currently it return result only when i type 8E or comeplete ''8E0061123-8E1'...any idea?? -- View this message in context: http://lucene.472066.n3.nabble.com/indexing-issue-tp4009122.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4 Alpha SolrJ Indexing Issue

Thanks Mark! On Thu, Jul 19, 2012 at 4:07 PM, Mark Miller wrote: > https://issues.apache.org/jira/browse/SOLR-3649 > > On Thu, Jul 19, 2012 at 3:34 PM, Briggs Thompson < > w.briggs.thomp...@gmail.com> wrote: > > > This is unrelated for the most part, but the javabin update request > handler > >

Re: Solr 4 Alpha SolrJ Indexing Issue

https://issues.apache.org/jira/browse/SOLR-3649 On Thu, Jul 19, 2012 at 3:34 PM, Briggs Thompson < w.briggs.thomp...@gmail.com> wrote: > This is unrelated for the most part, but the javabin update request handler > does not seem to be working properly when calling solrj > method*HttpSolrServer.de

Re: Solr 4 Alpha SolrJ Indexing Issue

This is unrelated for the most part, but the javabin update request handler does not seem to be working properly when calling solrj method*HttpSolrServer.deleteById(List ids) *. A single Id gets deleted from the index as opposed to the full list. It appears properly in the logs - shows delete of al

Re: Solr 4 Alpha SolrJ Indexing Issue

we really need to resolve that issue soon... On Jul 19, 2012, at 12:08 AM, Briggs Thompson wrote: > Yury, > > Thank you so much! That was it. Man, I spent a good long while trouble > shooting this. Probably would have spent quite a bit more time. I > appreciate your help!! > > -Briggs > > On W

Re: Solr 4 Alpha SolrJ Indexing Issue

Yury, Thank you so much! That was it. Man, I spent a good long while trouble shooting this. Probably would have spent quite a bit more time. I appreciate your help!! -Briggs On Wed, Jul 18, 2012 at 9:35 PM, Yury Kats wrote: > On 7/18/2012 7:11 PM, Briggs Thompson wrote: > > I have realized thi

Re: Solr 4 Alpha SolrJ Indexing Issue

On 7/18/2012 7:11 PM, Briggs Thompson wrote: > I have realized this is not specific to SolrJ but to my instance of Solr. > Using curl to delete by query is not working either. Can be this: https://issues.apache.org/jira/browse/SOLR-3432

Re: Solr 4 Alpha SolrJ Indexing Issue

Hi Briggs, I'm not sure about Solr 4.0, but do you need to commit? > curl http://localhost:8983/solr/coupon/update?commit=true -H "Content-Type: > text/xml" --data-binary '*:*' Brendan www.kuripai.com On Jul 18, 2012, at 7:11 PM, Briggs Thompson wrote: > I have realized this is not specifi

Re: Solr 4 Alpha SolrJ Indexing Issue

I have realized this is not specific to SolrJ but to my instance of Solr. Using curl to delete by query is not working either. Running curl http://localhost:8983/solr/coupon/update -H "Content-Type: text/xml" --data-binary '*:*' Yields this in the logs: INFO: [coupon] webapp=/solr path=/update pa

Indexing Issue between Mac OS X 10.5 and 10.6

Greetings Everyone - I am hoping someone can help me with this unusual issue I have here. Issue Indexing information in a database (i.e. /dataimport [full-import]) succeeds when I perform this function on a Mac OS X 10.6 with Java 1.6, but fails when I attempt the same indexing task on a 10.5

Re: indexing: issue with default values

thanx Eric, that was very helpfull 2010/2/12 Erik Hatcher > That would be the problem then, I believe. Simply don't post a value to > get the default value to work. > >Erik > > > On Feb 12, 2010, at 10:18 AM, nabil rabhi wrote: > > yes, sometimes the document has postal_code with no va

Re: indexing: issue with default values

That would be the problem then, I believe. Simply don't post a value to get the default value to work. Erik On Feb 12, 2010, at 10:18 AM, nabil rabhi wrote: yes, sometimes the document has postal_code with no values , i still post it to solr 2010/2/12 Erik Hatcher When a documen

Re: indexing: issue with default values

yes, sometimes the document has postal_code with no values , i still post it to solr 2010/2/12 Erik Hatcher > When a document has no value, are you still sending a postal_code field in > your post to Solr? Seems like you are. > >Erik > > > On Feb 12, 2010, at 8:12 AM, nabil rabhi wrote:

Re: indexing: issue with default values

When a document has no value, are you still sending a postal_code field in your post to Solr? Seems like you are. Erik On Feb 12, 2010, at 8:12 AM, nabil rabhi wrote: in the schema.xml I have fileds with int type and default value exp: stored="true" default="0"/> but when a docume

Fwd: indexing: issue with default values

in the schema.xml I have fileds with int type and default value exp: but when a document has no value for the field "postal_code" at indexing, I get the following error: Posting file Immo.xml to http://localhost:8983/solr/update/ Error 500 HTTP ERROR: 500For input string: "" java.lang.Numb

Indexing issue with XML control characters

During indexing I will often get this error: SEVERE: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR, code 3)) at [row,col {unknown-source}]: [2,1] at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675) By looking at this list and elsewhe

Re: Indexing issue in DIH - not all records are Indexed

I changed the UniqueKey and it worked fine.thank you very much Nobel 2009/5/18 Noble Paul നോബിള്‍ नोब्ळ् > the problem is that your uniquekey may not be unique > > just remove the entry altogether > > On Mon, May 18, 2009 at 10:53 PM, jayakeerthi s > wrote: > > Hi Noble, > > Many thanks for

Re: Indexing issue in DIH - not all records are Indexed

Hi Noble, Many thanks for the reply Yes there is a UniqueKey in the Schema which is the ProductID. I also tried PROD_ID. But no luck same only one document seen after querying *:* I have attached the Schema.xml used for your reference,please advise. Thanks and regards, Jay 2009/5/16 Noble Paul

Re: Indexing issue in DIH - not all records are Indexed

check out if you have a uniqueKey in your schema. I there are duplicates they are overwritten On Sat, May 16, 2009 at 1:38 AM, jayakeerthi s wrote: > I am using Solr for our application with JBoss Integration. > > I have managed to configure the indexing from Oralce db for 22 fields.Here > is the

Re: Indexing issue

: I have two cores in different machines which are referring to the same data directory. this isn't really considered a supported configuration ... both solr instances are going to try and "own" the directory for updating, and unless you do somethign special to ensure only one has control you

Indexing issue

Hi,   I have two cores in different machines which are referring to the same data directory. I have implemented this mechanism to have fault tolerance in place, if any of the machines are down the fault tolerance take care to index data from other machine.   Since two cores are referring to the