RE: Escape characters for core.properties User-Defined Properties

[Jens Hittenkofer]

Found a solution, if anyone else has this issue the solution was:

1. (If its not already there) Move the  part from 
db-data-config.xml into the requesthandler for data import ()  in  
solrconfig.xml.
2. Modify the strings for the connection/user/password so the request handler 
looks like this:



  db-data-config.xml

com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbc:sqlserver://${sqlserver:unset}\${sqlinstance:unset};databaseName=${sqldatabase:unset}
${sqluser:unset}
${qlpassword:unset}




This makes it possible to have those ': \ = ;' characters in the jdbcurl and 
just set the:

sqlserver
sqlinstance
sqldatabase

parts as core.properties.


-Original Message-
From: Jens Hittenkofer  
Sent: den 13 november 2020 15:25
To: solr-user@lucene.apache.org
Subject: Escape characters for core.properties User-Defined Properties

Hi,

Im using SolrCloud (Solr 6.6.6 together with Zookeeper) . I want to use the 
same template/configuration for multiple Solr cores. Each core should use its 
own connectionstring for the data-import-handler (MSSQL). What im trying to do 
is to add the connectionstring as a User-Defined Property in the 
core.properties file while creating the core to reference in the 
data-import-handler, as such:
http://X:8983/solr/admin/collections?action=CREATE&name=TESTCORE&numShards=1&maxShardsPerNode=2&replicationFactor=3&collection.configName=TESTCORECONFIG&property.connectionstring=sqlserver://server.domain\instance;databaseName=SOMEDBNAME

But when I add the connectionstring like above I get escape an character ('\') 
before ':', '=' and '\'. So the connectionstring in core.properties looks like 
this: 'sqlserver\://server.domain\\instance;databaseName\=SOMEDBNAME'
I've tried to urlencode and escape the characters in different ways (also read 
documentation and googled ofc).
Is there anyway to do this or should I go for a totally different approach 
somehow?

Thanks

What do you usually look for in Solr logs?

[Radu Gheorghe]

Hello Solr users,

I've recently added a Solr logs integration
 to our logging SaaS
 and I wanted to ask what would be useful
that I may have missed.

First, there are some regexes to parse Solr Logs here:
https://github.com/sematext/logagent-js/blob/master/patterns.yml#L140-L379
BTW, Logagent is open-source, so you can use it not only with our SaaS, but
with your own Elasticsearch (at least until there's an output plugin for
Solr :D) or you can simply reuse the regexes.

I've tested parsing on a number of Solr versions and they should work on
7.x and later. But what about the extracted fields? Do you see something
interesting that's not already there?

On the frontend side, what do you find useful? Because we have some default
dashboards and we can always add/update them. Let me quickly tell you
what's already included.

The first one you'd see, called Overview, is quite self-explanatory:


The one I use most is called Queries. All widgets have a filter by (+path:*
-path:admin -path:update) and most of them are built around QTime and Hits
metrics:


I won't spam you with more screenshots, but there are other dashboards, too:
- *Errors*: quite similar to Overview, but with a (severity:error) filter
- *Zookeeper*: I wanted to catch ZK communication here, so the filter is
(thread:zk* OR class:zk* OR class.raw:o.a.z.*) and the "usual" breakdown
widgets e.g. by host or severity
- *Overseer*: similar to Zookeeper, but filtering on (thread:overseer* OR
class:ShardLeaderElection* OR class:SyncStrategy)
- *Audit*: this is for Solr Audit Logging
, filtering on
(class.raw:"o.a.s.s.SolrLogAuditLoggerPlugin") Besides the usual
breakdowns, there are audit-specific widgets, like request types over time
- *Start & stop*: I'm trying to catch startup- and shutdown-specific logs
here. The best filters I found are (thread:main) for startup and
(thread:ShutdownMonitor OR thread:closeThreadPool* OR
thread:coreCloseExecutor*) for shutdown. Do you see better criteria?

Last but not least, what do you usually look for in Solr logs? Anything
that we don't cover in the above? Any feedback will be very much
appreciated!

Thanks and best regards,
Radu
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/

Re: Solr 8.4.1, NOT NULL query not working on plong & pint type fields (fieldname:* )

[Shawn Heisey]

On 11/25/2020 10:42 AM, Deepu wrote:

We are in the process of migrating from Solr 5 to Solr 8, during testing
identified that "Not null" queries on plong & pint field types are not
giving any results, it is working fine with solr 5.4 version.

could you please let me know if you have suggestions on this issue?


Here's a couple of facts:

1) Points-based fields have certain limitations that make explicit value 
lookups very slow, and make them unsuitable for use on uniqueKey fields. 
 Something about the field not having a "term" available.


2) A query of the type "fieldname:*" is a wildcard query.  These tend to 
be slow and inefficient, when they work.


It might be that the limitations of point-based fields make it so that 
wildcard queries don't work.  I have no idea here.  Points-based fields 
did not exist in Solr 5.4, chances are that you were using a Trie-based 
field at that time.  A wildcard query would have worked, but it would 
have been slow.


I may have a solution even though I am pretty clueless about what's 
going on.  When you are looking to do a NOT NULL sort of query, you 
should do it as a range query rather than a wildcard query.  This means 
the following syntax.   Note that it is case sensitive -- the "TO" must 
be uppercase:


fieldname:[* TO *]

This is how all NOT NULL queries should be constructed, regardless of 
the type of field.  Range queries tend to very efficient.


Thanks,
Shawn

Re: Solr 8.4.1, NOT NULL query not working on plong & pint type fields (fieldname:* )

[Deepu]

Hi Shawn,
Thanks for taking time and replay.

Thanks,
Deepu

On Thu, Nov 26, 2020 at 10:53 PM Shawn Heisey  wrote:

> On 11/25/2020 10:42 AM, Deepu wrote:
> > We are in the process of migrating from Solr 5 to Solr 8, during testing
> > identified that "Not null" queries on plong & pint field types are not
> > giving any results, it is working fine with solr 5.4 version.
> >
> > could you please let me know if you have suggestions on this issue?
>
> Here's a couple of facts:
>
> 1) Points-based fields have certain limitations that make explicit value
> lookups very slow, and make them unsuitable for use on uniqueKey fields.
>   Something about the field not having a "term" available.
>
> 2) A query of the type "fieldname:*" is a wildcard query.  These tend to
> be slow and inefficient, when they work.
>
> It might be that the limitations of point-based fields make it so that
> wildcard queries don't work.  I have no idea here.  Points-based fields
> did not exist in Solr 5.4, chances are that you were using a Trie-based
> field at that time.  A wildcard query would have worked, but it would
> have been slow.
>
> I may have a solution even though I am pretty clueless about what's
> going on.  When you are looking to do a NOT NULL sort of query, you
> should do it as a range query rather than a wildcard query.  This means
> the following syntax.   Note that it is case sensitive -- the "TO" must
> be uppercase:
>
> fieldname:[* TO *]
>
> This is how all NOT NULL queries should be constructed, regardless of
> the type of field.  Range queries tend to very efficient.
>
> Thanks,
> Shawn
>

Uploading Features in Solr 6.6

[vishal patel]

Hi

I have read the concept of "Learning To Rank".  I see the Example: 
/path/myFeatures.json

{
"name" : "documentRecency",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : {
  "q" : "{!func}recip( ms(NOW,last_modified), 3.16e-11, 1, 1)"
}
  },
  {
"name" : "isBook",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : {
  "fq": ["{!terms f=cat}book"]
}
  }

I want to make feature like q=title:"Test with some GB18030 encoded 
characters". Can I make?
Can I search sentence "test book" in terms like "fq": ["{!terms f=cat}test 
book"]?

I want to use q and fq same like normal in feature.

Regards,
Vishal Patel


Sent from Outlook

Solr Highlighting not working

[Ajay Sharma]

Hi Community,

This is the first time, I am implementing a solr *highlighting *feature.
I have read the concept via solr documentation
Link- https://lucene.apache.org/solr/guide/8_2/highlighting.html

To enable highlighting I just have to add *&hl=true&hl.fl=* *in our solr
query and got the snippet in the solr response and it is working fine in
most of the cases.

*But highlighting does not work when synonyms came into action*

*Issue:*
I am searching leopard (q=leopard) in field title (qf=title)

In our synonym file, we have an entry like below
*leopard,tenduaa,panther*

and in one document id:123456, field title contains below text:
title:"Jindal Panther TMT Bars

For the query (q=leopard) , i am getting this document (id:123456) in solr
response
I could check that due to synonym document is matched  and I confirmed it
via Solr UI analysis screen where I put Analyse FieldName= title,  Field
Value (Index) ="Jindal Panther TMT rebars" and Field Value (Query) =
leopard and I could see in index chain, token panther getting saved as
leopard also but in highlighting I don't get any matched token and getting
below response


   - highlighting:
   {
  - 123456: { }
  }



I just need the matched synonym token like panther in the above case to be
returned in solr highlighting response
I have read and re-read the solr documentation, searched on google gone
through many articles even checked StackOverflow but could not find a
solution.
Any help from community members will be highly appreciated.

Thanks in advance.


-- 
Regards,
Ajay Sharma
Software Engineer, Product-Search,
IndiaMART InterMESH Ltd

-- 

uploading model in Solr 6.6

[vishal patel]

Hi

what is meaning of weight of feature at the time Uploading a Model for 
Re-Ranking?
How can we calculate the weight? Ranking is depended on weight?

Please give me more details about weight.
https://lucene.apache.org/solr/guide/8_1/learning-to-rank.html#uploading-a-model

Regards,
Vishal

Sent from Outlook