Hello Russ,
It's an interesting case! Can you get a brief context?
- is it possible to keep both type of data at the same core? Why not?
- can you manually shard both indices by those longValues?
- It seems like you query a plenty of data, don't you have another
query/filter to intersect that join result with?
Such a long time for "universe of 5 docs" seems really strange. Can you
open the index with Solr 5.3 and run the same query with number of result
in universe:universeValue, but adding local param {!join ... score=none}?
that triggers alternative algorithm.
Also, profiler snapshots always help, you know. I've given a brief intro in
join algorithms, and problems in Solr at recent Berlin Buzzwords, feel free
to have a look if you are interested.
On Tue, Sep 8, 2015 at 3:09 PM, Russell Taylor <
russell.tay...@interactivedata.com> wrote:
> Hi,
> I hope somebody can help.
>
> We have two indexes, one which holds the descriptive data and the other
> one which holds lists of docs which are
> of a certain type (called universes in our world). They need to be joined
> together to show a list of data from indexA
> where a filtered indexB (by universe:value) has matching longs (The join
> field).
>
> At the moment the query is taking 55 seconds we need to get it under a
> second, any help most appreciated.
>
> INDEXES:
>
> Index a (primary index)
> 31 million docs with a converted alphanumeric to a long value with a
> possible 10 million unique values.
>
> Index B (the joined index)
> 250 million documents with a converted alphanumeric to a long value with a
> possible 10 million unique values.
> IndexB is filtered by universe which could be between 1 and 500,000 docs.
>
> QUERY:
>
> http://127.0.0.1:8080/solr/indexA/select?q={!join+from=longValue+to=longValue+fromIndex=IndexB}universe
> :<
> http://127.0.0.1:8080/solr/indexA/select?q=%7b!join+from=longValue+to=longValue+fromIndex=IndexB%7duniverse
> :>universeValue
>
> Qtime is 55 seconds for either a universe of 5 docs or 500,000 docs.
>
>
>
> Thanks
>
>
> Russ.
>
>
> *******************************************************
> This message (including any files transmitted with it) may contain
> confidential and/or proprietary information, is the property of Interactive
> Data Corporation and/or its subsidiaries, and is directed only to the
> addressee(s). If you are not the designated recipient or have reason to
> believe you received this message in error, please delete this message from
> your system and notify the sender immediately. An unintended recipient's
> disclosure, copying, distribution, or use of this message or any
> attachments is prohibited and may be unlawful.
> *******************************************************
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mkhlud...@griddynamics.com>
