I may have used wrong terminology, by complex types I meant non-primitive
types. Mutlivalued can be conceptualized as a list of values for instance
in your example myint = [ 32, 77] etc which you can possibly analyze and
query upon. What I was trying to ask is if a complex type can be
multi-valued or something along those lines that can be supported by range
queries.
For instance: Below rows will have to be individual docs in Solr (in my
knowledge) - If I want to range query from ts=Jan 12 to ts=Jan 15 give me
sum of 'unique' where 'contentId=1,product=mobile'
contentId=1,product=mobile ts=Jan15 total=12,unique=5
contentId=1,product=mobile ts=Jan14 total=10,unique=3
contentId=1,product=mobile ts=Jan13 total=15,unique=2
contentId=1,product=mobile ts=Jan12 total=17,unique=4
......
This increases number of documents in Solr by a lot. Only if there was a
way to do something like:
{
contentId=1
product=mobile
ts = [
{
time = Jan15
total = 12
unique = 15
},
{
time = Jan16
total = 10
unique = 3
},
..
..
]}
Of course above isn't allowed, but some way to squeeze timestamps in single
document so that it doesn't increase the number of document by a lot and I
am still able to range query on 'ts'.
For some (combination of fields) rows the timestamps may go upto last 3-6
months!
Let me know if I am still being unclear.
On Sun, Jan 15, 2017 at 8:04 PM, Erick Erickson <erickerick...@gmail.com>
wrote:
> bq: I know multivalued fields don't support complex data types
>
> Not sure what you're talking about here. mulitValued actually has
> nothing to do with data types. You can have text fields which
> are analyzed and produce multiple tokens and are multiValued.
> You can have primitive types (string, int/long/float/double,
> boolean etc) that are multivalued. or they can be single valued.
>
> All "multiValued" means is that the _input_ can have the same field
> repeated, i.e.
> <doc>
> <field name="mytext">some stuff</field>
> <field name="mytext">more stuff</field>
> <field name='myint"">32</field>
> <field name='myint"">77</field>
> </doc>
>
> This doc would fail of mytext or myint were multiValued=false but
> succeed if multiValued=true at index time.
>
> There are some subtleties with text (analyzed) multivalued fields having
> to do with token offsets, but that's not germane.
>
> Does that change your problem? Your document could have a dozen
> timestamps....
>
> However, there isn't a good way to query across multiple multivalued fields
> in parallel. That is, a doc like
>
> myint=1
> myint=2
> myint=3
> mylong=4
> mylong=5
> mylong=6
>
> there's no good way to say "only match this document if mhyint=1 AND
> mylong=4 AND they_are_both_in_the_same_position.
>
> That is, asking for myint=1 AND mylong=6 would match the above. Is
> that what you're
> wondering about?
>
> ------------------
> I expect you're really asking to do the second above, in which case you
> might
> want to look at StreamingExpressions and/or ParallelSQL in Solr 6.x
>
> Best,
> Erick
>
> On Sun, Jan 15, 2017 at 7:31 PM, map reduced <k3t.gi...@gmail.com> wrote:
> > Hi,
> >
> > I am trying to fit the following data in Solr to support flexible queries
> > and would like to get your input on the same. I have data about users
> say:
> >
> > contentID (assume uuid),
> > platform (eg. website, mobile etc),
> > softwareVersion (eg. sw1.1, sw2.5, ..etc),
> > regionId (eg. us144, uk123, etc..)
> > ....
> >
> > and few more other such fields. This data is partially pre aggregated
> (read
> > Hadoop jobs): so let’s assume for "contentID = uuid123 and platform =
> > mobile and softwareVersion = sw1.2 and regionId = ANY" I have data in
> > format:
> >
> > timestamp pre-aggregated data [ uniques, total]
> > Jan 15 [ 12, 4]
> > Jan 14 [ 4, 3]
> > Jan 13 [ 8, 7]
> > ... ...
> >
> > And then I also have less granular data say "contentID = uuid123 and
> > platform = mobile and softwareVersion = ANY and regionId = ANY (These
> > values will be more than above table since granularity is reduced)
> >
> > timestamp : pre-aggregated data [uniques, total]
> > Jan 15 [ 100, 40]
> > Jan 14 [ 45, 30]
> > ... ...
> >
> > I'll get queries like "contentID = uuid123 and platform = mobile" , give
> > sum of 'uniques' for Jan15 - Jan13 or for "contentID=uuid123 and
> > platform=mobile and softwareVersion=sw1.2", give sum of 'total' for
> Jan15 -
> > Jan01.
> >
> > I was thinking of simple schema where documents will be like (first
> example
> > above):
> >
> > {
> > "contentID": "uuid12349789",
> > "platform" : "mobile",
> > "softwareVersion": "sw1.2",
> > "regionId": "ANY",
> > "ts" : "2017-01-15T01:01:21Z",
> > "unique": 12,
> > "total": 4
> > }
> >
> > second example from above:
> >
> > {
> > "contentID": "uuid12349789",
> > "platform" : "mobile",
> > "softwareVersion": "ANY",
> > "regionId": "ANY",
> > "ts" : "2017-01-15T01:01:21Z",
> > "unique": 100,
> > "total": 40
> > }
> >
> > Possible optimization:
> >
> > {
> > "contentID": "uuid12349789",
> > "platform.mobile.softwareVersion.sw1.2.region.us12" : {
> > "unique": 12,
> > "total": 4
> > },
> > "platform.mobile.softwareVersion.sw1.2.region.ANY" : {
> > "unique": 100,
> > "total": 40
> > },
> > "ts" : "2017-01-15T01:01:21Z"
> > }
> >
> > Challenges: Number of such rows is very large and it'll grow
> exponentially
> > with every new field - For instance if I go with above suggested schema,
> > I'll end up storing a new document for each combination of
> > contentID,platform,softwareVersion,regionId. Now if we throw in another
> > field to this document, number of combinations increase exponentially.I
> > have more than a billion such combination rows already.
> >
> > I am hoping to find advice by experts if
> >
> > 1. Multiple such fields can be fit in same document for different 'ts'
> > such that range queries are possible on it.
> > 2. time range (ts) can be fit in same document as a list(?) (to reduce
> > number of rows). I know multivalued fields don't support complex data
> > types, but if anything else can be done with the data/schema to reduce
> > query time and number of rows.
> >
> > The number of these rows are very large, for sure more than 1billion (if
> we
> > go with the schema I was suggesting). What schema would you suggest for
> > this that'll fit query requirements?
> >
> > FYI: All queries will be exact match on fields (no partial or tokenized),
> > so no analysis on fields is necessary. And almost all queries are range
> > queries.
> >
> > Thanks,
> >
> > KP
>
