GitHub user okram opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/285
TINKERPOP-1225: Do a "rolling reduce" for GroupXXXStep in OLAP.
https://issues.apache.org/jira/browse/TINKERPOP-1225
We now can do mid-barrier reductions in with `group()` on OLAP. This is
huge as this means that if you have an reducer in your `by()`-valueTraversal,
the stream is constantly reduced to limit memory consumption. Its more
expensive in terms of time (not by much) for small data, but for large data, no
worries about OME with group() (both both OLTP and OLAP).
CHANGELOG
```
* `GroupStep` and `GroupSideEffectStep` make use of mid-traversal reducers
to limit memory consumption in OLAP.
```
VOTE +1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-tinkerpop TINKERPOP-1225
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #285
----
commit 8eb53815acfcec05316dcf39bd91d9e3a43d971a
Author: Marko A. Rodriguez <[email protected]>
Date: 2016-03-30T20:24:38Z
GroupStep and GroupSideEffectStep now make use mid-traversal barriers to do
data reduction on the fly in order to limite the memory footprint and reduce
the chances of OME. OLTP always did this, but now OLAP (which needs it more)
does it. This is epic. Also fixed a minor bug in ReducingBarrierStep. Added
some more GroupTest test cases -- one that does a groupCount() instead of a
group() just to make sure things are working as expected.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
