just testing if my emails are reaching the mailing list

[uyilmaz]

Hello all,

I have never got an answer to my questions in this mailing list yet, and my 
mail client shows INVALID next to my mail address, so I thought I should check 
if my emails are reaching to you.

Can anyone reply?

Regards

-- 
uyilmaz 

Re: solr1.3からsolr8.4へのデータ移行について

[Kayak28]

Hi,

Replacing fils under the data directory won't work as you expected.
Solr has changed its index format,
so whenever you consider upgrading, you are strongly recommended to
re-index all of your documents for your safety.

FYI: not only index format, but also other things have been changed from
Solr1.3 to Solr 8.4.
You have to pay attention to managed-schema, which is the replacement of
the schema.xlm.
Additionally, scoring and ranking never be able to the same between Solr
1.3 and 8.4

I strongly recommend taking a look at changes.txt.

Good Luck with your upgrading.


Sincerely,
Kaya Ota






2020年10月12日(月) 19:57 阿部真也 :

> 私はsolrを使用した古いシステムから新しいシステムに再構築し
> データ移行を行う必要があります。
>
> 現在システムはsolr1.3で動作していて、新規に構築するシステムでは
> solrのバージョンを8.4 にアップデートしようと考えています。
> そこで、/var/solr/{system_name}/data のデータを
> 旧システムから新システムに移し替えることでうまくいくかどうか、知りたいです。
>
> 既にsolrconfig.xmlのほとんどの設定が移行できないことが分かっていますが、
> こちらは使用している設定名の代替手段がエラーログに出てくるため
> 何とかなるかもしれないと思っています。
>
> よろしくお願いします。
>


-- 

Sincerely,
Kaya
github: https://github.com/28kayak

Re: just testing if my emails are reaching the mailing list

[Szűcs Roland]

Hi,
I got it from the solr user list.


Roland

uyilmaz  ezt írta (időpont: 2020. okt. 14.,
Sze, 9:39):

> Hello all,
>
> I have never got an answer to my questions in this mailing list yet, and
> my mail client shows INVALID next to my mail address, so I thought I should
> check if my emails are reaching to you.
>
> Can anyone reply?
>
> Regards
>
> --
> uyilmaz 
>

Re: Solr Document Update issues

[Radu Gheorghe]

Hi,

I wouldn’t commit on every update. The general practice is to use autoCommit 
and autoSoftCommit, so this work is done in background depending on how quickly 
you want data persisted and available for search: 
https://lucene.apache.org/solr/guide/6_6/updatehandlers-in-solrconfig.html#UpdateHandlersinSolrConfig-Commits

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 13 Oct 2020, at 07:18, aparana bhatt  wrote:
> 
> Hi ,
> 
> I have been facing lot of issues in using solr update functionality .
> Multitude of requests respond with
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> * org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://192.169.33.86/solr/cms
> : Expected mime type
> application/octet-stream but got text/html.  "-//IETF//DTD HTML 2.0//EN">502 Proxy
> ErrorProxy ErrorThe proxy server received
> an invalid^Mresponse from an upstream server.^MThe proxy server could
> not handle the request  href="/solr/cms/update">POST /solr/cms/update.Reason:
> Error reading from remote server*
> 
> Used solr version -> 6.5.0  Type -> master/Slave config
> Error in solr.log ->
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *2020-10-07 05:43:50.639 WARN  (qtp142261320-27831) [   x:cms]
> o.a.s.c.SolrCore slow: [cms]  webapp=/solr path=/update
> params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
> status=0 QTime=443272020-10-07 05:43:50.640 WARN  (qtp142261320-27837) [
> x:cms] o.a.s.u.DefaultSolrCoreState WARNING - Dangerous
> interruptjava.lang.InterruptedExceptionat
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
>  at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
>  at
> org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:167)
>  at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:112)
>  at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:618)
>  at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
>  at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
>  at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1895)
>  at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1872)
>  at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
>  at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:72)
>  at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
>  at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
>  at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>  at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>  at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>  at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>  at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>  at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>  at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)*
> 
> 
> 
> The rate of update query on master solr is 6 request per min only .
> Solr also slows down and search becomes really slow .
> I don't understand where to look for an issue .
> I have tried to check various parameters in update request , if I do
> softcommit=true and commit =false then updates do not reflect , so i have
> set below options ->
> 
> UpdateRequest updateRequest = new UpdateRequest();
>  updateRequest.setAction( UpdateRequest.ACTION.COMMIT, true,
> true);
> waitsearcher=true ,
> waitflush=true .
> 
> I do not get what is causing the issue . Kindly suggest .
> Also I could not find much help from internet about given issues as well .
> 
> 
> -- 
> Regards
> 
> Aparana Bhatt

Re: how to config split authentication methods -- BasicAuth for WebUI, & none (or SSL client) for client connections?

[Radu Gheorghe]

Hello,

If you enable authentication, this will work on your HTTP port. Solr won’t make 
a difference on whether the request comes from the Web UI or Dovecot.

I guess the workaround could be to put the web UI behind a proxy like NGINX and 
have authentication there?

But if anyone can have direct HTTP access to Solr, then it’s not really secure.

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 12 Oct 2020, at 05:11, PGNet Dev  wrote:
> 
>  I'm running,
> 
>   solr -version
>   8.6.3
> 
> on
> 
>   uname -rm
>   5.8.13-200.fc32.x86_64 x86_64
> 
>   grep _NAME /etc/os-release
>   PRETTY_NAME="Fedora 32 (Server Edition)"
>   CPE_NAME="cpe:/o:fedoraproject:fedora:32"
> 
> with
> 
>   java -version
>   openjdk version "15" 2020-09-15
>   OpenJDK Runtime Environment 20.9 (build 15+36)
>   OpenJDK 64-Bit Server VM 20.9 (build 15+36, mixed mode, sharing)
> 
> solr's configured for SSL usage.  both client search connections and WebUI 
> access work OK, with EC certs in use
> 
>   SOLR_SSL_KEY_STORE="/srv/ssl/solr.server.EC.pfx"
>   SOLR_SSL_TRUST_STORE="/srv/ssl/solr.server.EC.pfx"
> 
> If I enable BasicAuth, adding
> 
>   /security.json
>   {
>   "authentication":{
>   "blockUnknown": true,
>   "class":"solr.BasicAuthPlugin",
>   "credentials":{
>   "myuser":"jO... Fe..."
> 
>   },
>   "realm":"Solr REALM",
>   "forwardCredentials": false
>   },
>   "authorization":{
>   "class":"solr.RuleBasedAuthorizationPlugin",
>   "permissions":[{
>   "name":"security-edit",
>   "role":"admin"
>   }],
>   "user-role":{
>   "solr":"admin"
>   }
>   }
>   }
> 
> as expected, WebUI requires/accepts valid credentials for access.
> 
> BUT ... client connections, e.g. from a mail MUA using dovecot's fts solr 
> plugin, immediately fail, returning "401 Unauthorized".
> 
> How can solr authentication be configured to split method -- using BasicAuth 
> for WebUI access ONLY, and still allowing the client connections?
> 
> Eventually, I want those client connections to require solr-side SSL client 
> auth.
> Atm, I'd just like to get it working -- _with_ the BasicAuth WebUI protection 
> in place.
> 

Re: Is metrics api enabled by default in solr 8.2

[Radu Gheorghe]

Hi,

Yes, the API works by default on 8.2: 
https://lucene.apache.org/solr/guide/8_2/metrics-reporting.html

I don’t know of a way to disable it, but he configuration is described in the 
page above (i.e. on how to configure different reporters).

Best regards,
Radu
--
Sematext Cloud - Full Stack Observability - https://sematext.com
Solr and Elasticsearch Consulting, Training and Production Support

> On 14 Oct 2020, at 06:05, yaswanth kumar  wrote:
> 
> Can I get some info on where to disable or enable metrics api on solr 8.2 ?
> 
> I believe its enabled by default on solr 8.2 , where can I check the
> configurations? and also how can I disable if I want to disable it
> 
> -- 
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanth...@gmail.com

Re: Analytics for Solr logs

[Zisis T.]

Thanks Alexandre, silly me. I though 8.4.1 was recent enough...



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

How to specify time zone in SOLR GC log?

[Amy Bai]

Hi community,

When I set solr time zone to Asia/Tokyo to specify solr log time.
However, the solr gc log is still using my machine's default timezone.
How to keep time zone in gc log consistent with solr log?

Thanks.

Re: just testing if my emails are reaching the mailing list

[uyilmaz]

Thank you!

On Wed, 14 Oct 2020 09:41:16 +0200
Szűcs Roland  wrote:

> Hi,
> I got it from the solr user list.
> 
> 
> Roland
> 
> uyilmaz  ezt írta (időpont: 2020. okt. 14.,
> Sze, 9:39):
> 
> > Hello all,
> >
> > I have never got an answer to my questions in this mailing list yet, and
> > my mail client shows INVALID next to my mail address, so I thought I should
> > check if my emails are reaching to you.
> >
> > Can anyone reply?
> >
> > Regards
> >
> > --
> > uyilmaz 
> >


-- 
uyilmaz 

Re: Need urgent help -- High cpu on solr

[Zisis T.]

The values you have for the caches and the maxwarmingsearchers do not look
like the default. Cache sizes are 512 for the most part and
maxwarmingsearchers are 2 (if not limit them to 2)

Sudden CPU spikes probably indicate GC issues. The #  of documents you have
is small, are they huge documents? The # of collections is OK in general but
since they are crammed in 5 Solr nodes the memory requirements might be
bigger. Especially if filter and the other caches get populated with 50K
entries. 

I'd first go through the GC activity to make sure that this is not causing
the issue. The fact that you lose some Solr servers is also an indicator of
large GC pauses that might create a problem when Solr communicates with
Zookeeper. 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Is metrics api enabled by default in solr 8.2

[Andrzej Białecki]

SOLR-14914 (scheduled for 8.7) adds a boolean property (either by modifying 
solr.xml:/metrics element or via “metricsEnabled” system property) to almost 
completely turn off the metrics collection and processing. The “almost” part 
means that the instrumentation still remains in place, but the cost is reduced 
to empty method calls.

> On 14 Oct 2020, at 10:03, Radu Gheorghe  wrote:
> 
> Hi,
> 
> Yes, the API works by default on 8.2: 
> https://lucene.apache.org/solr/guide/8_2/metrics-reporting.html
> 
> I don’t know of a way to disable it, but he configuration is described in the 
> page above (i.e. on how to configure different reporters).
> 
> Best regards,
> Radu
> --
> Sematext Cloud - Full Stack Observability - https://sematext.com
> Solr and Elasticsearch Consulting, Training and Production Support
> 
>> On 14 Oct 2020, at 06:05, yaswanth kumar  wrote:
>> 
>> Can I get some info on where to disable or enable metrics api on solr 8.2 ?
>> 
>> I believe its enabled by default on solr 8.2 , where can I check the
>> configurations? and also how can I disable if I want to disable it
>> 
>> -- 
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanth...@gmail.com
> 

Re: Need urgent help -- High cpu on solr

[Erick Erickson]

Zisis makes good points. One other thing is I’d look to 
see if the CPU spikes coincide with commits. But GC
is where I’d look first.

Continuing on with the theme of caches, yours are far too large
at first glance. The default is, indeed, size=512. Every time
you open a new searcher, you’ll be executing 128 queries
for autowarming the filterCache and another 128 for the queryResultCache.
autowarming alone might be accounting for it. I’d reduce
the size back to 512 and an autowarm count nearer 16
and monitor the cache hit ratio. There’s little or no benefit
in squeezing the last few percent from the hit ratio. If your
hit ratio is small even with the settings you have, then your caches
don’t do you much good anyway so I’d make them much smaller.

You haven’t told us how often your indexes are
updated, which will be significant CPU hit due to
your autowarming.

Once you’re done with that, I’d then try reducing the heap. Most
of the actual searching is done in Lucene via MMapDirectory,
which resides in the OS memory space. See:

https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Finally, if it is GC, consider G1GC if you’re not using that
already.

Best,
Erick


> On Oct 14, 2020, at 7:37 AM, Zisis T.  wrote:
> 
> The values you have for the caches and the maxwarmingsearchers do not look
> like the default. Cache sizes are 512 for the most part and
> maxwarmingsearchers are 2 (if not limit them to 2)
> 
> Sudden CPU spikes probably indicate GC issues. The #  of documents you have
> is small, are they huge documents? The # of collections is OK in general but
> since they are crammed in 5 Solr nodes the memory requirements might be
> bigger. Especially if filter and the other caches get populated with 50K
> entries. 
> 
> I'd first go through the GC activity to make sure that this is not causing
> the issue. The fact that you lose some Solr servers is also an indicator of
> large GC pauses that might create a problem when Solr communicates with
> Zookeeper. 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: solr1.3からsolr8.4へのデータ移行について

[Erick Erickson]

Kaya is certainly correct. I’d add that Solr has changed so much
in the last 12 years that you should treat it as a new Solr installation.
Do not, for instance, just use the configs from Solr 1.3, start
with the ones from the version of Solr you install.

Best,
Erick

> On Oct 14, 2020, at 3:41 AM, Kayak28  wrote:
> 
> Hi,
> 
> Replacing fils under the data directory won't work as you expected.
> Solr has changed its index format,
> so whenever you consider upgrading, you are strongly recommended to
> re-index all of your documents for your safety.
> 
> FYI: not only index format, but also other things have been changed from
> Solr1.3 to Solr 8.4.
> You have to pay attention to managed-schema, which is the replacement of
> the schema.xlm.
> Additionally, scoring and ranking never be able to the same between Solr
> 1.3 and 8.4
> 
> I strongly recommend taking a look at changes.txt.
> 
> Good Luck with your upgrading.
> 
> 
> Sincerely,
> Kaya Ota
> 
> 
> 
> 
> 
> 
> 2020年10月12日(月) 19:57 阿部真也 :
> 
>> 私はsolrを使用した古いシステムから新しいシステムに再構築し
>> データ移行を行う必要があります。
>> 
>> 現在システムはsolr1.3で動作していて、新規に構築するシステムでは
>> solrのバージョンを8.4 にアップデートしようと考えています。
>> そこで、/var/solr/{system_name}/data のデータを
>> 旧システムから新システムに移し替えることでうまくいくかどうか、知りたいです。
>> 
>> 既にsolrconfig.xmlのほとんどの設定が移行できないことが分かっていますが、
>> こちらは使用している設定名の代替手段がエラーログに出てくるため
>> 何とかなるかもしれないと思っています。
>> 
>> よろしくお願いします。
>> 
> 
> 
> -- 
> 
> Sincerely,
> Kaya
> github: https://github.com/28kayak

Re: Analytics for Solr logs

[Cassandra Targett]

While the tool is only included in 8.5 and higher, it will index logs from any 
version of Solr 7.x or 8.x (and possibly even 6.x). So if you want to use it, 
you could download Solr 8.5 or higher to your local machine and index your 
8.4.1 logs there, or use an 8.5 or higher Docker image. You probably need to 
copy them locally; AFAIK, it can't take a URL to go get files from a non-local 
filestore like S3, etc.

I use it at least weekly in my work…it’s an immense help to troubleshooting 
when you aren’t sure what’s going on with the system.
On Oct 14, 2020, 3:50 AM -0500, Zisis T. , wrote:
> Thanks Alexandre, silly me. I though 8.4.1 was recent enough...
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Strange fetch streaming expression doesn't fetch fields sometimes?

[Joel Bernstein]

Yes, the docs mention one-to-one and many-to-one fetches, but one-to-many
is not supported currently. I've never really been happy with fetch. It
really needs to be replaced with a standard nested loop join that handles
all scenarios.


Joel Bernstein
http://joelsolr.blogspot.com/


On Tue, Oct 13, 2020 at 6:30 PM uyilmaz  wrote:

> I think I found the reason right after asking (facepalm), but it took me
> days to realize this.
>
> I think fetch performs a naive "in" query, something like:
>
> q="userid:(123123 123123123 12432423321323)&rows={batchSize}"
>
> When userid to document relation is one-to-many, it is possible that above
> query will result in documents consisting entirely of last two userid's
> documents, so the first one is left out, resulting in empty username. Docs
> state that one to many is not supported with fetch, but I didn't stumble
> onto this issue until recently so I just assumed it would work.
>
> Sorry to take your time, I hope this helps somebody later.
>
> Have a nice day.
>
> On Wed, 14 Oct 2020 00:38:05 +0300
> uyilmaz  wrote:
>
> >
> > Hi all,
> >
> > I have a streaming expression looking like:
> >
> > fetch(
> >   myAlias,
> >   top(
> >   n=3,
> >   various expressions here
> > sort="count(*) desc"
> >   ),
> >   fl="username", on="userid=userid", batchSize=3
> > )
> >
> > which fails to fetch username field for the 1st result:
> >
> > {
> >  "result-set":{
> >   "docs":[{
> > "userid":"123123",
> > "count(*)":58}
> >,{
> > "userid":"123123123",
> > "count(*)":32,
> > "username":"Ayha"}
> >,{
> > "userid":"12432423321323",
> > "count(*)":30,
> > "username":"MEHM"}
> >,{
> > "EOF":true,
> > "RESPONSE_TIME":34889}]}}
> >
> > But strangely, when I change n and batchSize both to 2 and touch nothing
> else, fetch fetches the first username correctly:
> >
> > fetch(
> >   myAlias,
> >   top(
> >   n=2,
> >   various expressions here
> > sort="count(*) desc"
> >   ),
> >   fl="username", on="userid=userid", batchSize=2
> > )
> >
> > Result is:
> >
> > {
> >  "result-set":{
> >   "docs":[{
> > "userid":"123123",
> > "count(*)":58,
> > "username":"mura"}
> >,{
> > "userid":"123123123",
> > "count(*)":32,
> > "username":"Ayha"}
> >,{
> > "EOF":true,
> > "RESPONSE_TIME":34889}]}}
> >
> > What can be the problem?
> >
> > Regards
> >
> > ~~ufuk
> >
> > --
> > uyilmaz 
>
>
> --
> uyilmaz 
>