Defining SOLR nested fields

[santosh sidnal]

Hi All,

I want to define nested fileds in SOLR using schema.xml. we are using Apache
Solr 4.7.0.

i see some links which says how to do, but not sure how can i do it in
schema.xml
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-BlockJoinQueryParsers


any help over here is appreciable.

-- 
Regards,
Santosh Sidnal

Data form cataloggroup and catalogentry cores

[santosh sidnal]

Hi All,

Is there a way to get a combined data from 2 different cores together in a
single call?


like a data form both CatalogEntry and CatalogGroup cores in a single call
to solr.



-- 
Regards,
Santosh Sidnal

Index corruption with replication

[Santosh Sidnal]

Hi all,

I am facing issues of index corruption at regular intervals of the time on live 
server where i pull index data from one master server.

Can anyone please give us some ppinters why we are facing issue on regular 
interval of time? 
I am aware of how can we correct corrupted index but i am looking some pointers 
how can i stop or reduce this occurrence. 

Thanks in advance.


Sent from my iPhone

Re: Index corruption with replication

[santosh sidnal]

Hi Erik/David,

Schema is same on both live and stage servers. We are using the same schema
files on the stage and live files.


   - Schema files are included in replication but these are not being
   changed whenever we are observing schema corruption issue.
   - My guess is that because of replication the core is getting corrupted.
   - SOLR version used is 4.7.0
   -



Exception which I see in log is

org.apache.solr.common.SolrException log
org.apache.lucene.index.CorruptIndexException: Corrupted: docID=8195,
docBase=7, chunkDocs=249, numDocs=10596
(resource=MMapIndexInput(path="/app/IBM/WebSphere/CommerceServer70/instances/RBUATLV/search/solr/home/MC_10001/fr_FR/CatalogEntry/data/index/_5a.fdt"))
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:236)
at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276)
at
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110)
at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:661)
at
org.apache.solr.util.SolrPluginUtils.optimizePreFetchDocs(SolrPluginUtils.java:213)
at
org.apache.solr.handler.component.QueryComponent.doPrefetch(QueryComponent.java:568)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:475)
at

On 15 March 2017 at 22:42, Erick Erickson  wrote:

> You can specify your replication to include config files, but if the
> schema has changed you'll have to restart your Solr afterwards.
>
> How is it corrupt? what is the symptom? Any error messages in the solr
> log on the slave? What version of Solr? Details matter.
>
> Best,
> Erick
>
> On Wed, Mar 15, 2017 at 9:12 AM, David Hastings
>  wrote:
> > are you certain the schema is the same on both master and slave?  I find
> > that the schema file doesnt always go with the replication and if a field
> > is different on the slave it will cause problems
> >
> > On Wed, Mar 15, 2017 at 12:08 PM, Santosh Sidnal <
> sidnal.sant...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> I am facing issues of index corruption at regular intervals of the time
> on
> >> live server where i pull index data from one master server.
> >>
> >> Can anyone please give us some ppinters why we are facing issue on
> regular
> >> interval of time?
> >> I am aware of how can we correct corrupted index but i am looking some
> >> pointers how can i stop or reduce this occurrence.
> >>
> >> Thanks in advance.
> >>
> >>
> >> Sent from my iPhone
>



-- 
Regards,
Santosh Sidnal

Closed connection issue while doing dataimport

[santosh sidnal]

d Connection information (shared partitions 200)
  No shared connections

Free Connection information (free distribution table/partitions 27/1)
  (0)(0)MCWrapper id 7cc87cc8  Managed connection
WSRdbManagedConnectionImpl@e870e87  State:STATE_ACTIVE_FREE
  (0)(0)MCWrapper id 68b068b  Managed connection
WSRdbManagedConnectionImpl@fdd0fdd  State:STATE_ACTIVE_FREE

  Total number of connection in free pool: 2
UnShared Connection information
MCWrapper id 2d232d23  Managed connection
WSRdbManagedConnectionImpl@3a4c3a4c  State:STATE_ACTIVE_INUSE Thread Id:
00b4 Thread Name: Thread-92 Handle count 1 Start time inuse Mon Mar 27
08:48:49 CDT 2017 Time inuse 351 (seconds)
  Total number of connection in unshared pool: 1

Connection Leak Logic Information:
  MCWrapper id 2d232d23  Managed connection
WSRdbManagedConnectionImpl@3a4c3a4c  State:STATE_ACTIVE_INUSE Thread Id:
00b4 Thread Name: Thread-92 Handle count 1
 Start time inuse Mon Mar 27 08:48:49 CDT 2017 Time inuse 351 (seconds)
 Last allocation time Mon Mar 27 08:48:49 CDT 2017
   getConnection stack trace information:

com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:900)

com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)

com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)

org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:213)

org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:139)

org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:129)

com.ibm.commerce.solr.handler.SchemaJdbcDataSource$1.call(SchemaJdbcDataSource.java:189)

com.ibm.commerce.solr.handler.SchemaJdbcDataSource$1.call(SchemaJdbcDataSource.java:1)

org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:392)

org.apache.solr.handler.dataimport.JdbcDataSource.access$200(JdbcDataSource.java:40)

org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:266)

org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:236)

org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:40)

org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)

org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)

org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)

org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)

org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)

org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:237)

org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)

org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)

org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)

, Details: MCWrapper id 2d232d23  Managed connection
WSRdbManagedConnectionImpl@3a4c3a4c  State:STATE_ACTIVE_INUSE Thread Id:
00b4 Thread Name: Thread-92 Handle count 1

[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit >  prepare entry Entry
[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit <  prepare entry Exit
[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit >  writeDataAndHeader Entry
[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit 3   writeDataAndHeader -
writing 2686 bytes to  location 1781784
[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit 3   writeDataAndHeader -
write complete, filePointer = 1784470
[3/27/17 8:54:41:406 CDT] 00b4 SharedLogWrit 3   writeDataAndHeader -
freeSpace ptr = 1784470
[3/27/17 8:54:41:406 CDT] 000000b4 SharedLogWrit <  writeDataAndHeader Exit

-- 
Regards,
Santosh Sidnal

SOLR scalability porblem

[santosh sidnal]

Hi All,

I have a problem with scalability on my project. we are running almost
close of 100 cores which are having documents of ~25000 each and the total
size of the index files being 7.5 GB.


Also, we have the staging server where we build index files using data
importer and using replication we are pushing data to data LIVE servers
which are used for serving the live application. because we are initiating
index pulling from live servers currently 5 servers against one stage
server. Stage server is overburden and will not be able to respond properly
either to indexing job or to other services (very minimal).


So now the problem statement is I understand my current SOLR architecture
is not able to handle all my needs so we are thinking to upgrade to next
levels but I am confused with below questions,


   1. Can I use index Sharding for my problem? it is recommended to use if
   my have more than ~1million docs in a core but I have in only 25000 in one
   core and having 100 cores.
   2. Can I consider SOLR cloud? if yes please let me know why?
   3. How about using apache zookeeper and maintain only LIVE servers by
   dividing every 20 cores to each server? so that I can make only 25 snap
   pull request to stage server instead of 100 snap-puller per server as
   current.

Any new suggestions or reply to this email is greatly appreciated. Thanks
in advance.



-- 
Regards,
Santosh Sidnal

Re: Closed connection issue while doing dataimport

[santosh sidnal]

Thanks Shwan, that was good point to consider but we had porblem with
holdability param in data-config.xml and Oracle 12c version of DB and
client.

BY remove Holdability parameter we are able to solve the issue.

On 28 March 2017 at 18:19, Shawn Heisey  wrote:

> On 3/27/2017 7:13 PM, santosh sidnal wrote:
> > i am facing closed connection issue while doing dataimporter, any
> solution
> > to this> stack trace is as below
> >
> >
> > [3/27/17 8:54:41:399 CDT] 00b4 OracleDataSto >  findMappingClass for
> :
> > Entry
> >  java.sql.SQLRecoverableException:
> Closed
> > Connection
>
> Does the import appear to work correctly at first, then stop before it's
> done with this exception after a few hours?  If so, then I think you may
> be running into a situation where the merge scheduler has multiple
> merges scheduled and stops the incoming indexing thread until the
> largest merge is done.  If the indexing thread is stopped for long
> enough, JDBC will disconnect the database, and when the indexing thread
> finally starts back up, the dataimporter finds that it cannot read from
> the database any more.
>
> If this is what's happening, then the solution is to allow the merge
> scheduler to schedule more merges simultaneously.  Here's the
> indexConfig that I use in solrconfig.xml:
>
> 
>   
> 35
> 35
> 105
>   
>   
> 1
> 6
>   
> 
>
> You don't need the mergePolicy part of this config to solve this issue.
> That is the settings that I use, but this greatly increases the number
> of files that can be in an index, which may require OS tuning for max
> open files.
>
> The mergeScheduler is the important part, and specifically
> maxMergeCount.  These settings are for standard spinning-magnetic
> disks.  I increase the maxMergeCount to 6, so more merges can be
> scheduled without halting the indexing thread.  The maxThreadCount value
> should be set to 1 if you're using standard disks.  If you're using SSD,
> then you can bump it a little bit, because SSD can easily handle
> multiple threads randomly writing to the disk.  I don't know what values
> are appropriate for SSD, but I would probably start with 2-4.
>
> Thanks,
> Shawn
>
>


-- 
Regards,
Santosh Sidnal

sort by an attribute values sequence

[santosh sidnal]

Hi All,

We have a specific requirement of sorting the products as per a specific
attribute value sequence. Any pointer or source of info would help us.

Example of the scenario;

Let's say for search result i want to sort results based on a attribute
producttype. Where producttype has following values, A, B, C, D.

so while in solr query i can give either producttype asc, producttype desc.

But I want get result in a specific way by saying first give me All results
of values 'C' then B, A, D.


-- 
Regards,
Santosh Sidnal

Re: sort by an attribute values sequence

[santosh sidnal]

Re-sending my e-mail. any pointers/ links for the issue will help me lot.

Thanks in advance.


On Tue, Apr 1, 2014 at 4:25 PM, santosh sidnal wrote:

> Hi All,
>
> We have a specific requirement of sorting the products as per a specific
> attribute value sequence. Any pointer or source of info would help us.
>
> Example of the scenario;
>
> Let's say for search result i want to sort results based on a attribute
> producttype. Where producttype has following values, A, B, C, D.
>
> so while in solr query i can give either producttype asc, producttype desc.
>
> But I want get result in a specific way by saying first give me All
> results of values 'C' then B, A, D.
>
>
> --
> Regards,
> Santosh Sidnal
>
>


-- 
Regards,
Santosh Sidnal

Re: sort by an attribute values sequence

[santosh sidnal]

Hi Ahmet/All,

Thanks for the reply.

The Solution of boosting those product type values will work fine if i
don't apply any 'sort' .

But my requirement is i want sorting to be applied and boost a
particular/some attribute values (C,B etc) in the sorted result, which is
not working. Looks like sorting will take precedence over boosting. Correct
me if i am wrong.

Also i am trying  functional query, looks like even i will face same
problem over there.

we have more only 4 values for producttype attribute, but for different
keywords we have to use different attributes in sorting and boosting the
result, that we can manage in our application.


Regards,
Santosh


On Wed, Apr 2, 2014 at 8:49 PM, Ahmet Arslan  wrote:

> Hi,
>
> How many distinct producttype do you have?
>
> May be
>
> q=C^5000 OR B^4000 OR A^3000 OR D&df=producttype
>
> could work.
>
> If you can came up with a function that takes maximum value when
> producttype=C ... etc you can sort by function queries too.
> http://wiki.apache.org/solr/FunctionQuery
>
>
> Ahmet
>
>
> On Wednesday, April 2, 2014 1:52 PM, santosh sidnal <
> sidnal.sant...@gmail.com> wrote:
> Re-sending my e-mail. any pointers/ links for the issue will help me lot.
>
> Thanks in advance.
>
>
> On Tue, Apr 1, 2014 at 4:25 PM, santosh sidnal  >wrote:
>
> > Hi All,
> >
> > We have a specific requirement of sorting the products as per a specific
> > attribute value sequence. Any pointer or source of info would help us.
> >
> > Example of the scenario;
> >
> > Let's say for search result i want to sort results based on a attribute
> > producttype. Where producttype has following values, A, B, C, D.
> >
> > so while in solr query i can give either producttype asc, producttype
> desc.
> >
> > But I want get result in a specific way by saying first give me All
> > results of values 'C' then B, A, D.
> >
> >
> > --
> > Regards,
> > Santosh Sidnal
>
> >
> >
>
>
> --
> Regards,
> Santosh Sidnal
>
>


-- 
Regards,
Santosh Sidnal

Re: java.net.SocketException: Connection reset

[santosh sidnal]

Even i am facing same issue.  AFTER doing a server restart again indexing
can run fine once,  but for second time same issue.
On 3 Jul 2014 23:37, "heaven"  wrote:

> Hi, trying DigitalOcean for Solr, everything seems well, except sometimes I
> see these errors:
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
>
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
>
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
>
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
> at
>
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
> at
>
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
> at
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
> at
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
> at
>
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
> at
>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> 
>
> Solr version is 4.8.1, on Ubuntu Linux. We have 2 nodes, one run 2 shards
> and another 2 replicas.
>
> Errors happen during indexing process. Does it require some
> tweaks/optimizations? I have no idea where to look to fix this. Any
> suggestions are welcome.
>
> Thank you,
> Alex
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Query using doc Id

[santosh sidnal]

i guess you can use these two params in your query,

rows=100&start=100

which will give you 100 documents after 100th document.


On Wed, Jul 23, 2014 at 10:19 AM, Mukundaraman Valakumaresan <
muk...@8kmiles.com> wrote:

> Hi,
>
> Is it possible to execute queries using doc Id as a query parameter
>
> For eg, query docs whose doc Id is between 100 and 200
>
> Thanks & Regards
> Mukund
>



-- 
Regards,
Santosh Sidnal