This is an automated email from the ASF dual-hosted git repository. yaqian pushed a change to branch master-hadoop3 in repository https://gitbox.apache.org/repos/asf/kylin.git.
omit e8b357f Resolve shaded-guava new 13df006 [maven-release-plugin] prepare for next development iteration new fe990b1 Reset version for shaded-guava new f586d9a KYLIN-4787 Auto add USE $DATABASE new fec4e91 KYLIN-4636 Make /api/admin/public_config callable for profile saml new 25b08d4 KYLIN-4771 Clear the recordCachePool when the deadline has reached;add timeout for the recordCachePool offer method. new 52dafa1 Fix code to pass the UT new 5c37105 KYLIN-4626 add set kylin home shell new ae8d5f2 KYLIN-4711 Change default value to 3 for kylin.metadata.hbase-client-retries-number new 6dd6168 [KYLIN-4702] Fix missing cube-level lookup table snapshot when doing cube migration new 630b5b9 KYLIN-4819 build cube failed when `kylin.metadata.hbase-client-retries-number` great than 1 new 3113d0b KYLIN-4807 Add liveness and readiness probe to k8s deployment new c6742c9 FIX KYLIN-4810 new 83817b9 FIX KYLIN-4810, Add some tips and test case new 023dd85 KYLIN-4826 fix kylin.source.hive.warehouse-dir can't found new c8ed788 KYLIN-4826 fix kylin.source.hive.warehouse-dir can't found new 347d264 KYLIN-4838 fix KYLIN-4679 bug new b9f484e KYLIN-4841 Spark RDD cache is invalid when building with spark engine new b7d8733 Remove redundant code new 5bb5bd2 KYLIN-4736 Upgrade flink version to 1.11.1 (#1462) new 25e47b5 Proofread README new 3fbf5f3 [KYLIN-4702] Correct cube-level snapshot path when the related snapshot can be reused when doing cube migration. new 6aa7fc8 update kylin.metric.subject-suffix to kylin.metrics.subject-suffix new 5bae643 KYLIN-4613 add build cubev2 and JobRestClient new a958c72 minor, add cyclonedx-maven-plugin dependency new 590b24d [KYLIN-4827] Fix SparkMergingDictionary parallelize not work. new 44e7008 revert hadoop version change to recover UT new 3f4c3d8 KYLIN-4658 Fix union all issue with regarding to windows function & aggregation on new f8385fd KYLIN-4833 create hfile on hadoop hdfs and distcp to hbase hdfs (#1494) new 9341310 KYLIN-4682 Fix java.lang.IndexOutOfBoundsException due to not setting havingFilter correctly (#1529) new 7b2d06d KYLIN-4847 Cuboid to HFile step failed on multiple job server env new 94fe023 Minor, unify thread capacity for scannerThreadPool new 779ec21 KYLIN-4896 Optimize the process of writing big resource files into HDFS. new ce7b9f6 KYLIN-4855 kylin.metrics.prefix bug fix (#1536) new 233e202 real-time stream support diff database hive table in lambda mode new 6070f8f fix bug: real-time stream don't support GMT-N timezone and other non-derived time columns new 2754c05 check derived column for stream table new e1a6b2b KYLIN-4879 Fix the function of remove sql comments new 4b85051 KYLIN-4929 skip update metrics for queries that hit no realization (#1605) new d88dbe1 KYLIN-4921: New rowkey for streamv2 source config and fix bug that can't create same name table in diff project (#1604) new 1b89f4c KYLIN-4930 reset current page when searching for users/groups (#1607) new d1a9cda KYLIN-4667 Automatically set kylin.query.cache-signature-enabled to be true when memcached is enabled (#1540) new 85b0a3d [KYLIN-4940] Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine new e8c1f16 [KYLIN-4938] remove segment by UUID new 70111c4 KYLIN-4826 JobRunner should not do onExecuteError or onExecuteFinished if it has lost the job lock new 25b90d5 [KYLIN-4794] Make it possible to force hit a cube set for sqls with cube join new 0e41e6e [KYLIN-4939] Transform lookup table snapshot from segment level to cube level new d569134 KYLIN-4933 Support set cache strength for dict cache new ddb6002 [KYLIN-4955] fix typo in KYLIN UI when not set dictionary for count_distinct measure. new 7c12746 Revert "[maven-release-plugin] prepare for next development iteration" new a084ea9 KYLIN-4962 Fix NPE in ShrunkDict step new a168573 prepare release kylin-3.1.2 new 1e7f2af Update .travis.yml new 2bbd025 [maven-release-plugin] prepare for next development iteration new 663f6c8 Reset external module new a7224f1 [KYLIN-4971] Add new measure bitmap_map for count distinct at frontend. (#222) new fd94970 [KYLIN-4554] validate filter condition on model saving new 85bcbde [KYLIN-4554] validate filter condition on model saving, fix bug added all the columns of all tables to the validate collection new fd1ecf3 [KYLIN-4948] Provide an API to allow users to adjust cuboids manually new b77b53b NPE: refrech segment when there is no segments new 141a69f [KYLIN-4970] fix spark.executor.extraJavaOptions args in SparkExecutable. new e95b49d [KYLIN-4964] Receiver consumer thread should be stoped while encounting unrecoverable error (#1622) new cbecb13 KYLIN-4879 add UT new 7647f26 fix bug new 6d9c383 KYLIN-4944 kylin3 Docker image is upgraded from centos6.9 to centos7.9.2009 (#1629) new 6317d65 [KYLIN-4992] Source row count statistics calculated in a wrong way in MergeDictionaryMapper new 170d1d6 add more check rules for migration streaming cube new dba1dca KYLIN-4995 fix query exception when the query statement contains a single left parenthesis new 905c444 KYLIN-5003 Remove legacy front end dependencies new 00acd44 Minor fix, remove unused import new b3e5714 [KYLIN-4947] Implement spark engine for cube optimization jobs. new 92543e4 Add rule for submit build job: forbid submitting appending segment in the lambda stream cube (#228) new 1904b11 KYLIN-5016 avoid npe issue in rdbms pushdown new bd3d7de KYLIN-5007 queries with limit clause may fail when string dimension i… (#1664) new 5b382d4 add checking rule: before create new replicaset, we should check whether the node are in the other sets new 42b913a [KYLIN-5032] Not allow change project for a cubeInstance. new 8d5081e When coordinator leader change, clear the cached submitted job info new 85e70fb [KYLIN-5035] Fix Merge Cuboid Statistics EOFException. new 47955e3 streaming cube just supports the MR engine new 590798c fix bug: retry the same error url when coordinator leader changes new 374eb82 [KYLIN-5033] RealizationRegistry provider MUST always containing CubeManager new 6af6f3e fix new d207950 fix bug new 14404f8 KYLIN-5054 kylin_system's project and cubes create time is wrong 1970 new db44b99 KYLIN-4974 add time serializer to handle CURRENT_TIME calcite function. (#1637) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e8b357f) \ N -- N -- N refs/heads/master-hadoop3 (db44b99) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 84 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 3 +- README.md | 12 +- assembly/pom.xml | 2 +- build/bin/build-incremental-cube.sh | 7 +- build/bin/find-hive-dependency.sh | 7 +- build/bin/header.sh | 1 + build/bin/kylin-port-replace-util.sh | 1 + build/bin/kylin.sh | 36 ++- build/bin/sample.sh | 3 + build/bin/set-kylin-home.sh | 4 + build/bin/system-cube.sh | 212 +++++++----- build/script/build.sh | 1 + build/smoke-test/testBuildCube.py | 2 +- cache/pom.xml | 2 +- core-common/pom.xml | 2 +- .../apache/kylin/common/JDBCConnectionUtils.java | 142 ++++++++ .../java/org/apache/kylin/common/KylinConfig.java | 16 +- .../org/apache/kylin/common/KylinConfigBase.java | 66 +++- .../apache/kylin/common/restclient/RestClient.java | 20 +- .../org/apache/kylin/common/util/ClassUtil.java | 6 +- .../kylin/common/util/CliCommandExecutor.java | 10 +- .../org/apache/kylin/common/util/ServerMode.java | 3 +- .../kylin/common/JDBCConnectionUtilsTest.java | 142 ++++++++ core-cube/pom.xml | 2 +- .../org/apache/kylin/cube/CubeDescManager.java | 12 + .../java/org/apache/kylin/cube/CubeInstance.java | 9 + .../java/org/apache/kylin/cube/CubeManager.java | 21 ++ .../java/org/apache/kylin/cube/CubeSegment.java | 4 + .../apache/kylin/cube/cuboid/CuboidScheduler.java | 3 +- .../org/apache/kylin/cube/cuboid/CuboidUtil.java | 1 + .../kylin/cube/cuboid/TreeCuboidScheduler.java | 7 +- .../org/apache/kylin/cube/kv/RowKeyDecoder.java | 3 +- .../java/org/apache/kylin/cube/model/CubeDesc.java | 33 ++ core-dictionary/pom.xml | 2 +- .../org/apache/kylin/dict/DictionaryManager.java | 19 +- .../apache/kylin/dict/TrieDictionaryBuilder.java | 26 +- .../org/apache/kylin/dict/TrieDictionaryTest.java | 20 ++ core-job/pom.xml | 2 +- .../java/org/apache/kylin/job/JoinedFormatter.java | 10 +- .../apache/kylin/job/common/ShellExecutable.java | 3 +- .../kylin/job/constant/ExecutableConstants.java | 7 + .../kylin/job/execution/AbstractExecutable.java | 13 +- .../kylin/job/execution/BrokenExecutable.java | 3 +- .../job/execution/DefaultChainedExecutable.java | 5 +- .../org/apache/kylin/job/execution/Executable.java | 3 +- .../job/impl/threadpool/DefaultScheduler.java | 9 +- .../job/impl/threadpool/DistributedScheduler.java | 11 +- .../{JobExecutor.java => IJobRunner.java} | 16 +- .../org/apache/kylin/job/util/JobRestClient.java | 133 ++++++++ .../org/apache/kylin/job/ErrorTestExecutable.java | 3 +- .../org/apache/kylin/job/FailedTestExecutable.java | 3 +- .../kylin/job/FiveSecondSucceedTestExecutable.java | 3 +- .../kylin/job/PersistExceptionExecutable.java | 3 +- .../apache/kylin/job/RunningTestExecutable.java | 3 +- .../org/apache/kylin/job/SelfStopExecutable.java | 3 +- .../apache/kylin/job/SucceedTestExecutable.java | 3 +- core-metadata/pom.xml | 2 +- .../metadata/datatype/DataTypeSerializer.java | 1 + .../apache/kylin/metadata/model/ColumnDesc.java | 3 + .../org/apache/kylin/metadata/model/Segments.java | 5 +- .../org/apache/kylin/metadata/model/TableDesc.java | 4 + .../kylin/metadata/model/tool/CalciteParser.java | 7 + .../metadata/realization/RealizationRegistry.java | 4 + .../org/apache/kylin/metadata/util/ModelUtil.java | 124 +++++++ .../source/adhocquery/HivePushDownConverter.java | 12 +- .../apache/kylin/model/tool/CalciteParserTest.java | 126 ++++++++ .../adhocquery/HivePushDownConverterTest.java | 7 + core-metrics/pom.xml | 2 +- core-storage/pom.xml | 2 +- .../kylin/storage/gtrecord/CubeTupleConverter.java | 12 +- .../storage/gtrecord/GTCubeStorageQueryBase.java | 26 ++ cube-migration/pom.xml | 2 +- .../kylin/rest/controller/MigrationController.java | 42 +++ .../kylin/rest/service/MigrationRuleSet.java | 60 +++- .../kylin/rest/service/MigrationService.java | 2 +- datasource-sdk/pom.xml | 2 +- .../sdk/datasource/adaptor/AdaptorConfig.java | 6 +- .../framework/conv/DefaultConfigurer.java | 8 +- .../datasource/framework/conv/SqlConverter.java | 4 +- .../sdk/datasource/adaptor/AdaptorConfigTest.java | 16 +- docker/Dockerfile_hadoop | 11 +- docker/entrypoint.sh | 9 +- engine-flink/pom.xml | 2 +- .../apache/kylin/engine/flink/FlinkExecutable.java | 5 +- engine-mr/pom.xml | 2 +- .../kylin/engine/mr/BatchOptimizeJobBuilder2.java | 1 + .../kylin/engine/mr/common/BatchConstants.java | 5 +- .../kylin/engine/mr/common/CubeStatsWriter.java | 6 + .../engine/mr/common/HadoopShellExecutable.java | 3 +- .../engine/mr/common/MapReduceExecutable.java | 3 +- .../mr/steps/CalculateStatsFromBaseCuboidJob.java | 3 + .../kylin/engine/mr/steps/CopyDictionaryStep.java | 3 +- .../engine/mr/steps/MergeDictionaryMapper.java | 3 +- .../kylin/engine/mr/steps/MergeDictionaryStep.java | 3 +- .../kylin/engine/mr/steps/MergeStatisticsStep.java | 3 +- .../mr/steps/MergeStatisticsWithOldStep.java | 3 +- .../kylin/engine/mr/steps/SaveStatisticsStep.java | 3 +- .../mr/steps/UpdateCubeInfoAfterBuildStep.java | 3 +- .../steps/UpdateCubeInfoAfterCheckpointStep.java | 3 +- .../mr/steps/UpdateCubeInfoAfterMergeStep.java | 3 +- .../mr/steps/UpdateCubeInfoAfterOptimizeStep.java | 3 +- .../engine/mr/steps/UpdateDictionaryStep.java | 3 +- .../lookup/LookupSnapshotToMetaStoreStep.java | 3 +- .../steps/lookup/UpdateCubeAfterSnapshotStep.java | 3 +- .../kylin/engine/mr/streaming/SaveDictStep.java | 3 +- .../engine/mr/common/JobInfoConverterTest.java | 5 +- .../FactDistinctColumnsReducerMappingTest.java | 3 +- engine-spark/pom.xml | 2 +- .../kylin/engine/spark/KylinKryoRegistrator.java | 5 + .../engine/spark/SparkBatchCubingEngine2.java | 7 +- .../engine/spark/SparkBatchCubingJobBuilder2.java | 4 + .../spark/SparkBatchOptimizeJobBuilder2.java | 211 ++++++++++++ .../kylin/engine/spark/SparkBuildDictionary.java | 4 +- .../SparkCalculateStatsFromBaseCuboidJob.java | 356 +++++++++++++++++++++ .../kylin/engine/spark/SparkCubingByLayer.java | 186 ++++++++++- .../engine/spark/SparkCubingByLayerForOpt.java | 269 ++++++++++++++++ .../kylin/engine/spark/SparkCubingMerge.java | 4 +- .../apache/kylin/engine/spark/SparkExecutable.java | 26 +- .../kylin/engine/spark/SparkExecutableLivy.java | 3 +- .../kylin/engine/spark/SparkFactDistinct.java | 4 +- .../spark/SparkFilterRecommendCuboidDataJob.java | 168 ++++++++++ .../apache/kylin/engine/spark/SparkFunction.java | 104 ++++++ .../kylin/engine/spark/SparkMergingDictionary.java | 6 +- .../kylin/engine/spark/SparkUHCDictionary.java | 4 +- .../SparkUpdateShardForOldCuboidDataStep.java | 223 +++++++++++++ .../org/apache/kylin/engine/spark/SparkUtil.java | 139 +++++++- .../localmeta/cube_desc/ci_left_join_cube.json | 3 +- external/pom.xml | 9 +- external/shaded-guava/pom.xml | 2 +- jdbc/pom.xml | 2 +- .../deployment/kylin/kylin-job-statefulset.yaml | 15 +- .../deployment/kylin/kylin-query-statefulset.yaml | 15 +- .../streaming/kylin-receiver-statefulset.yaml | 2 +- .../kylin-job/kylin-job-statefulset.yaml | 15 +- .../kylin-query/kylin-query-statefulset.yaml | 15 +- .../deployment/kylin/kylin-all-statefulset.yaml | 15 +- kylin-it/pom.xml | 2 +- .../kylin/realtime/BuildCubeWithStreamV2.java | 3 +- .../{sql_streaming => sql_expression}/query10.sql | 9 +- metrics-reporter-hive/pom.xml | 2 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml | 54 ++-- query/pom.xml | 29 +- .../javacc/org/apache/kylin/query/util/.gitignore | 10 + .../org/apache/kylin/query/util/CommentParser.jj | 161 ++++++++++ .../query/enumerator/LookupTableEnumerator.java | 23 +- .../kylin/query/relnode/OLAPAggregateRel.java | 33 +- .../apache/kylin/query/relnode/OLAPFilterRel.java | 60 +++- .../apache/kylin/query/relnode/OLAPWindowRel.java | 33 +- .../relnode/visitor/TupleExpressionVisitor.java | 9 +- .../apache/kylin/query/routing/QueryRouter.java | 31 +- .../org/apache/kylin/query/util/QueryUtil.java | 17 +- .../org/apache/kylin/query/util/QueryUtilTest.java | 85 ++++- server-base/pom.xml | 2 +- .../kylin/rest/controller/CubeController.java | 129 ++++++++ .../rest/controller/StreamingV2Controller.java | 55 ++-- .../kylin/rest/controller/TableController.java | 2 +- .../kylin/rest/metrics/QueryMetrics2Facade.java | 3 +- .../kylin/rest/metrics/QueryMetricsFacade.java | 4 + .../java/org/apache/kylin/rest/msg/Message.java | 12 + .../kylin/rest/request/JobOptimizeRequest2.java | 24 +- .../apache/kylin/rest/service/BasicService.java | 5 + .../apache/kylin/rest/service/CacheService.java | 9 +- .../org/apache/kylin/rest/service/CubeService.java | 73 +++++ .../org/apache/kylin/rest/service/JobService.java | 19 ++ .../apache/kylin/rest/service/ModelService.java | 17 +- .../apache/kylin/rest/service/ProjectService.java | 2 +- .../apache/kylin/rest/service/QueryService.java | 26 +- .../kylin/rest/service/StreamingV2Service.java | 38 ++- .../rest/service/TableSchemaUpdateChecker.java | 38 ++- .../apache/kylin/rest/service/TableService.java | 19 +- server/pom.xml | 2 +- server/src/main/resources/kylinSecurity.xml | 3 + .../apache/kylin/rest/service/JobServiceTest.java | 3 +- .../kylin/rest/service/ModelServiceTest.java | 25 +- .../kylin/rest/service/QueryServiceTest.java | 6 + source-hive/pom.xml | 2 +- .../kylin/source/hive/BeelineHiveClient.java | 2 +- .../source/hive/CreateFlatHiveTableByLivyStep.java | 3 +- .../kylin/source/hive/CreateFlatHiveTableStep.java | 3 +- .../kylin/source/hive/CreateMrHiveDictStep.java | 3 +- .../org/apache/kylin/source/hive/DBConnConf.java | 3 + .../kylin/source/hive/GarbageCollectionStep.java | 3 +- .../hive/RedistributeFlatHiveTableByLivyStep.java | 3 +- .../source/hive/RedistributeFlatHiveTableStep.java | 3 +- .../java/org/apache/kylin/source/jdbc/CmdStep.java | 3 +- source-jdbc/pom.xml | 2 +- .../org/apache/kylin/source/jdbc/HiveCmdStep.java | 3 +- .../kylin/source/jdbc/sqoop/SqoopCmdStep.java | 3 +- .../jdbc/extensible/JdbcHiveMRInputTest.java | 2 +- source-kafka/pom.xml | 2 +- .../kylin/source/kafka/job/MergeOffsetStep.java | 3 +- storage-hbase/pom.xml | 12 +- .../kylin/storage/hbase/HBaseResourceStore.java | 15 +- .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 2 +- .../UpdateSnapshotCacheForQueryServersStep.java | 3 +- .../kylin/storage/hbase/steps/BulkLoadJob.java | 2 +- .../kylin/storage/hbase/steps/CreateHTableJob.java | 2 + .../hbase/steps/HBaseFlinkOutputTransition.java | 6 + .../kylin/storage/hbase/steps/HBaseFlinkSteps.java | 14 +- .../kylin/storage/hbase/steps/HBaseJobSteps.java | 28 ++ .../hbase/steps/HBaseMROutput2Transition.java | 9 + .../kylin/storage/hbase/steps/HBaseMRSteps.java | 12 +- .../hbase/steps/HBaseSparkOutputTransition.java | 32 +- .../hbase/steps/HDFSPathGarbageCollectionStep.java | 6 +- .../kylin/storage/hbase/steps/HFileDistcpJob.java | 97 +++--- .../kylin/storage/hbase/steps/MergeGCStep.java | 3 +- .../kylin/storage/hbase/steps/SparkCubeHFile.java | 4 +- ...UpdateSnapshotCacheForQueryServersStepTest.java | 11 +- storage-stream/pom.xml | 2 +- stream-coordinator/pom.xml | 2 +- .../kylin/stream/coordinator/Coordinator.java | 12 +- .../client/CoordinatorClientFactory.java | 17 +- .../coordinator/client/HttpCoordinatorClient.java | 8 +- .../coordinator/coordinate/BuildJobSubmitter.java | 4 + .../coordinate/StreamingCoordinator.java | 14 +- stream-core/pom.xml | 2 +- .../core/consumer/StreamingConsumerChannel.java | 6 + .../org/apache/kylin/stream/core/model/Node.java | 2 + .../stream/core/model/stats/ConsumerStats.java | 11 + .../core/query/MultiThreadsResultCollector.java | 14 +- .../stream/core/query/StreamingTupleConverter.java | 12 +- .../stream/core/source/MessageParserInfo.java | 17 + .../stream/core/source/StreamingSourceConfig.java | 32 +- .../core/source/StreamingSourceConfigManager.java | 88 +++-- .../columnar/ColumnarMemoryStorePersister.java | 5 +- .../storage/columnar/ColumnarSegmentStore.java | 3 + stream-receiver/pom.xml | 2 +- .../kylin/stream/server/StreamingServer.java | 2 +- stream-source-kafka/pom.xml | 2 +- .../source/kafka/KafkaBatchSourceAdaptor.java | 2 +- .../kylin/stream/source/kafka/KafkaSource.java | 8 +- .../stream/source/kafka/TimedJsonStreamParser.java | 9 +- tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 2 +- tool/pom.xml | 2 +- .../org/apache/kylin/tool/CubeMigrationCLI.java | 50 ++- .../kylin/tool/extractor/CubeMetaExtractor.java | 17 +- .../apache/kylin/tool/job/CubeBuildingCLIV2.java | 161 ++++++++++ .../metrics/systemcube/CubeInstanceCreator.java | 2 +- .../tool/metrics/systemcube/ProjectCreator.java | 2 +- .../StreamTableCompatibilityCheckRequest.java | 34 +- webapp/app/js/controllers/cubeAdvanceSetting.js | 4 + webapp/app/js/controllers/cubeEdit.js | 14 +- webapp/app/js/controllers/cubeSchema.js | 2 +- webapp/app/js/controllers/sourceMeta.js | 7 +- webapp/app/js/controllers/userGroup.js | 4 + webapp/app/js/model/cubeConfig.js | 8 +- webapp/app/js/model/cubeDescModel.js | 3 +- .../partials/cubeDesigner/advanced_settings.html | 12 +- webapp/app/partials/cubeDesigner/measures.html | 4 +- webapp/bower.json | 2 +- 252 files changed, 4855 insertions(+), 628 deletions(-) create mode 100644 build/bin/set-kylin-home.sh create mode 100644 core-common/src/main/java/org/apache/kylin/common/JDBCConnectionUtils.java create mode 100644 core-common/src/test/java/org/apache/kylin/common/JDBCConnectionUtilsTest.java copy core-job/src/main/java/org/apache/kylin/job/impl/threadpool/{JobExecutor.java => IJobRunner.java} (74%) create mode 100644 core-job/src/main/java/org/apache/kylin/job/util/JobRestClient.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/util/ModelUtil.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/model/tool/CalciteParserTest.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchOptimizeJobBuilder2.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCalculateStatsFromBaseCuboidJob.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingByLayerForOpt.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFilterRecommendCuboidDataJob.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFunction.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkUpdateShardForOldCuboidDataStep.java copy kylin-it/src/test/resources/query/{sql_streaming => sql_expression}/query10.sql (79%) create mode 100644 query/src/main/codegen/javacc/org/apache/kylin/query/util/.gitignore create mode 100644 query/src/main/codegen/javacc/org/apache/kylin/query/util/CommentParser.jj copy query/src/main/java/org/apache/kylin/query/udf/ConcatUDF.java => server-base/src/main/java/org/apache/kylin/rest/request/JobOptimizeRequest2.java (61%) copy engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateOldCuboidShardJob.java => storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileDistcpJob.java (50%) create mode 100644 tool/src/main/java/org/apache/kylin/tool/job/CubeBuildingCLIV2.java copy cube-migration/src/main/java/org/apache/kylin/rest/request/MigrationRequest.java => tool/src/main/java/org/apache/kylin/tool/migration/StreamTableCompatibilityCheckRequest.java (68%)