On 15/12/2015 14:13, Vikram Parmar wrote:
Hi Mikhail,
Hi,
In case you're interested, several years ago we prototyped a Lucene
codec using Redis for just this sort of application:
http://www.flax.co.uk/blog/2012/06/22/updating-individual-fields-in-lucene-with-a-redis-backed-codec/
It's a slightly crazy idea but appeared to work :)
Charlie
Thanks for chiming in. Looking forward to your post regarding updatable
numeric DocValues.
What would be the 2nd most promising approach for now, would you say EFF
should be ok to go with?
Updating and reloading the EFF external file (containing a millions lines)
at very short intervals is fine? Say every 10 seconds?
Thanks!
On Tue, Dec 15, 2015 at 5:46 PM, Mikhail Khludnev <
mkhlud...@griddynamics.com> wrote:
I believe https://issues.apache.org/jira/browse/SOLR-5944 is the most
promising approach for such scenarios.
Despite it's not delivered in distro.
We are going to publish a post about it at blog.griddynamics.com.
FWIW, I suppose EFF can be returned in result list.
On Fri, Dec 11, 2015 at 1:48 PM, Vikram Parmar <parmar.vik...@gmail.com>
wrote:
We are creating a web application which would contain posts (something
like
FB or say Youtube). For the stable part of the data (i.e.the facets,
search
results & its content), we plan to use SOLR.
What should we use for the unstable part of the data (i.e. dynamic and
volatile content such as Like counts, Comments counts, Viewcounts)?
Option 1) Redis
What about storing the "dynamic" data in a different data store (like
Redis)? Thus, everytime the counts get refreshed, I do not have to
reindex
the data into SOLR at all. Thus SOLR indexing is only triggered when new
posts are added to the site, and never on any activity on the posts by
the
users.
Side-note :-
I also looked at the SOLR-Redis plugin at
https://github.com/sematext/solr-redis
The plugin looks good, but not sure if the plugin can be used to fetch
the
data stored in Redis as part of the solr result set, i.e. in docs. The
description looks more like the Redis data can be used in the function
queries for boosting, sorting, etc. Anyone has experience with this?
Option 2) SOLR NRT with Soft Commits
We would depend on the in-built NRT features. Let's say we do
soft-commits
every second and hard-commits every 10 seconds. Suppose huge amount of
dynamic data is created on the site across hundreds of posts, e.g. 100000
likes across 10000 posts. Thus, this would mean soft-commiting on 10000
rows every second. And then hard-commiting those many rows every 10
seconds. Isn't this overkill?
Which option is preferred? How would you compare both options in terms of
scalibility, maintenance, feasibility, best-practices, etc? Any real-life
experiences or links to articles?
Many thanks!
p.s. EFF (external file fields) is not an option, as I read that the data
in that file can only be used in function queries and cannot be returned
as
part of a document.
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mkhlud...@griddynamics.com>
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
