> On Jan. 23, 2017, 8:36 p.m., Suma Shivaprasad wrote: > > repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java, > > line 99 > > <https://reviews.apache.org/r/55813/diff/3/?file=1613196#file1613196line99> > > > > This shoudnt have superTypeNames check directly in the query since > > there is no index on guid + typeName. I had removed it in ATLAS-1404
reverted changes in DataSetLineageService class since its used only in unit tests. - Sarath ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55813/#review162765 ----------------------------------------------------------- On Jan. 24, 2017, 12:08 a.m., Sarath Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55813/ > ----------------------------------------------------------- > > (Updated Jan. 24, 2017, 12:08 a.m.) > > > Review request for atlas, Madhan Neethiraj and Suma Shivaprasad. > > > Bugs: ATLAS-1403 > https://issues.apache.org/jira/browse/ATLAS-1403 > > > Repository: atlas > > > Description > ------- > > Currently DSL uses a fill function during Gremlin Translation to merge > results by typeName and superTypeName and fill function loads the resulting > vertices in memory. This causes significant memory usage and ATLAS server > spends lot of time doing GC instead of useful work resulting in OOO sometimes > ( when GC is not able to recover and search queries are run in parallel) > The proposal is to replace this with typeName checks along by finding all the > subtypes for a given type and using an IN clause in the filter. > For eg: > Query = Person where (birthday < "1950-01-01T02:35:58.440Z") limit 40 offset 0 > Optimized query > Gremlin Query = L: > {g.V.has("__typeName", T.in, > ['Person','Manager']).and(_().has("Person.birthday", T.lt, -631142641560)) > [0..<40].toList()} > > > Diffs > ----- > > > repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java > fd5dba7 > > repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java > 266f27c > > repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java > b637f90 > > repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java > 41dc65f > > repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java > 3677544 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 889236c > repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala daef582 > > repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala > a9dcdff > repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala > ade4176 > > repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java > a0ee26c > repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 33513c5 > > Diff: https://reviews.apache.org/r/55813/diff/ > > > Testing > ------- > > Ran Unit Tests and was successful. > > > Thanks, > > Sarath Subramanian > >
