transaction log size
Hi, As Transaction log(Tlog) play important role while restarting the SolrCloud cluster, we are trying to decrease the size. Many of the posts on net which we find describing that - "decreasing the AutoCommit and increasing autoSoftCommit would generate the small size of transaction log". To test the aforesaid statement we executed some Run: Document Size: ~2KB. 1st Run: AutoCommit: 30 Sec autoSoftCommit: 20 Sec openSearcher: false Index size: 4.7 GB Transaction log: Master: 740KB Slave: 86 MB 2nd Run: AutoCommit: 20 Sec autoSoftCommit: 30 Sec openSearcher: false Index size: 4.7 GB Transaction log: Master: 740KB Slave: 202 MB PFA zip containing cluster(master, slave) transaction logs disk usage at the interval of 1 Min. transactionLog.zip <http://lucene.472066.n3.nabble.com/file/n4128354/transactionLog.zip> > Schema.xml > Solr-config.xml > transactionLog/master/* > transactionLog/slave/* May you please give us some pointer so that we can control on Transaction log(Tlog) generation. Thanks, --Gurfan -- View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: transaction log size
Thanks Shawn for the quick reply. We are using Solr Cloud version 4.6.1 Usually we see higher transaction log on replica. Leader`s tlog size is in KB`s. We also tried keeping the hard commit(autoCommit) as 20 Sec and autoSoftCommit as 30 Sec. We written a script to monitor the disk usage of tlog directory in 1 Min interval, also noticed that the logs are purge at a particular time. For instance: Tlog starts with ~4MB and it increases at some point i.e. 20MB,50MB,220MB,600MB again it reduces with ~10MB. http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ How can we reduce Tlog size at its lowest, so that our system restart up time will less. Thanks, --Gurfan -- View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354p4128547.html Sent from the Solr - User mailing list archive at Nabble.com.
Delete field in Solr
Hi,
We have a setup of SolrCloud 4.6. The fields Stored value is true.
Now I want to delete a field from indexed document. Is there any way from
which we can delete the field??
Field which we are trying to delete(extracted from schema.xml):
We comment out this field(SField2) entry from schema.xml and reload/optimize
index from solr admin UI.
Commit the solr index.
curl http://<>:8983/solr/update/json?commit=true
Again fired query for the same but the removed field(SField2) is back
showing in query result.
We followed below link to target the requirement:
https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
https://wiki.apache.org/solr/UpdateJSON
we tried another commands to delete the document ID:
1> For Deletion:
curl http://localhost:8983/solr/update -H 'Content-type:application/json' -d
'
[
{
"delete":{
"id":"c7d30e6850c54429b888794f7433e3c5"
}
}
]'
Output: {"responseHeader":{"status":400,"QTime":0},"error":{"msg":"Document
is missing mandatory uniqueKey field: id","code":400}}
2> to set null in existing indexed field:
curl http://localhost:8983/solr/update -H 'Content-type:application/json' -d
'
[
{"id" : "c7d30e6850c54429b888794f7433e3c5","SField2":{"set":null} }
]'
output:
{"responseHeader":{"status":500,"QTime":0},"error":{"msg":"For input string:
\"88\"","trace":"java.lang.NumberFormatException: For input string:
\"88\"\n\tat
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
Please Help !
--
View this message in context:
http://lucene.472066.n3.nabble.com/Delete-field-in-Solr-tp4131003.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delete field in Solr
Thanks for the quick response. Delete works fine as suggested.
Still we are facing an issue in updating document rest command:
Field description in schema.xml:
/curl http://192.168.213.52:8983/solr/update -H
'Content-type:application/json' -d '
[
{"id": "969dcba7c3ec49cf9d61017afe8a2768","SField2":{"set":"Bahama"}
}
]'/
{"responseHeader":{"status":500,"QTime":2},"error":{"msg":"For input string:
\"88\"","trace":"java.lang.NumberFormatException: For input string:
\"88\"\n\tat
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
*Server Log:*
2069175 [qtp17275133-18] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [collection1]
webapp=/solr path=/update params={} {} 0 2
2069176 [qtp17275133-18] ERROR org.apache.solr.core.SolrCore –
java.lang.NumberFormatException: For input string: "88"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:495)
at java.lang.Integer.valueOf(Integer.java:582)
On the other hand we are able to update from
UI(http://<>:8983/solr/#/collection1/documents). Please find attached
snap.
<http://lucene.472066.n3.nabble.com/file/n4131751/updateDocument.png>
While updating a field vaule(From Solr UI Screen) in a document for example:
As shown in the attached screenshot we are able to update the value of
SField2 to Bahama but when we fire query with the same id we only see the
latest value of SField2 in the response, however we have other fields
defined in schema as stored=true.
Though we were expecting that our query with same id should give all stored
attribute including Sfield2 with latest value.
Please let us know if we are missing some thing here.
Thanks,
--Gurfan
--
View this message in context:
http://lucene.472066.n3.nabble.com/Delete-field-in-Solr-tp4131003p4131751.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delete field in Solr
*Sorry, missed to add Filed entry from schema.xml * Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/Delete-field-in-Solr-tp4131003p4131769.html Sent from the Solr - User mailing list archive at Nabble.com.
search using Ngram.
Hi All, We are using EdgeNGramFilterFactory for searching with minGramSize="3", as per Business logic, auto fill suggestions should appear on entering 3 characters in search filter. While searching for contact with name "Bill Moor", the value will does not get listed when we type 'Bill M' but when we type 'Bill Moo' or 'Bill' it suggests 'Bill Moor'. Clearly, The tokens are not generated when there is space in between, we cannot set set minGramSize="1" as that will generate many tokens and slow the performance. Do we have a solution without using Ngram to generate tokens on entering 3 characters? Please suggest. Thanks, --Gurfan -- View this message in context: http://lucene.472066.n3.nabble.com/search-using-Ngram-tp4138596.html Sent from the Solr - User mailing list archive at Nabble.com.
Leader Selection Error
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
java.lang.Thread.run(Thread.java:744)","2014-06-18T01:26:45.818+",1,18,26,june,45,wednesday,2014,local"nix-all-logs
nix_errors""renew-sdb-3.int.ssi-cloud.com",database,21,"_-_--_::.;_;_:_t_..(.:)t_..(.:",,"/var/log/tomcat7/solr.log",sdb,"splunkindexer-3.int.ssi-cloud.com"error,error,31,,8,,,
,,"INFO - 2014-06-18 01:26:45.818;
org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for
coreNodeName: core_node1, state: recovering, checkLive: true, onlyIfLeader:
true","2014-06-18T01:26:45.818+",1,18,26,june,45,wednesday,2014,local"nix-all-logs""renew-sdb-3.int.ssi-cloud.com",database,1,"__-_--_::.;_.;_:_,_:_,_:_,_:_",,"/var/log/tomcat7/solr.log",sdb,"splunkindexer-3.int.ssi-cloud.com",,31,,8,,,
,,"INFO - 2014-06-18 01:26:45.814;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2
message={
Appreciate any pointer on it.
Thanks,
--Gurfan
--
View this message in context:
http://lucene.472066.n3.nabble.com/Leader-Selection-Error-tp4142529.html
Sent from the Solr - User mailing list archive at Nabble.com.
