This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch kylin-on-parquet-v2
in repository https://gitbox.apache.org/repos/asf/kylin.git.
discard 97db36c KYLIN-4660 Remove some module from maven project
add a8de6d4 KYLIN-4660 Create two module
add db1eba8 KYLIN-4660 remove some useless modules
add db866b7 KYLIN-4660 remove some unsupported features
add 82045c2 KYLIN-4660 Fix UT and compile error
add 4bfaf99 KYLIN-4646 Fix merge error
add 5764e47 KYLIN-4660 Normalize variable and file naming
add dc1a80f KYLIN-4660 fix package.sh
add 3d14e97 KYLIN-4680 Avoid annoying log messages of unit test and
integration test
add a2e9bfa KYLIN-4516 support System cube
add 15305ea KYLIN-4662 Migrate from third-party Spark to offical Apache
Spark
add 8e6c90a KYLIN-4516 Fix 'next on empty iterator' error
add 47dd4a2 KYLIN-4680 Avoid annoying log messages of query module and
unit test
add 1b4e4c2 KYLIN-4694 Fix 'NoClassDefFoundError:
Lcom/esotericsoftware/kryo/io/Output' when query with sparder on yarn
add c9370ea KYLIN-4704 Upgrade sample metadata
add 14b5750 KYLIN-4705 add rest api for sparder-context url monitor
add 35308f7 KYLIN-4701 Front end change for Parquet Storage
add 656d9f3 KYLIN-4660 Cleanup for IT and logger
add ccdff32 KYLIN-4660 Add comparison with prepared data set
add 3f0b2a8 KYLIN-4713 Support use diff spark schedule pool for diff query
add 65ad984 KYLIN-4722 Add more statistics to the query results
add d43aca4 KYLIN-4721 The default source source type should be CSV not
Hive with the local debug mode
add 7a336fa KYLIN-4660 Remove duplicated/misleading code or comment
add 56cf4e6 KYLIN-4698 Delete segment storage path after merging segment,
deleting segment and droping cube
add cfdcb9f KYLIN-4699 Delete job_tmp path after build/merge successfully
add ff338de KYLIN-2971 Fix the wrong "Realization Names" and missing
"Cuboid Ids" in logQuery when hit cache
add 621868c KYLIN-4660 remove deprecated config
'spark.yarn.executor.memoryOverhead'
add 2d998f1 KYLIN-4695 Automatically start sparder (for query)
application when start kylin instance
add ea267e2 KYLIN-4704 Add 'PERCENTILE' measure for sample metadata
add 2d6d8b5 KYLIN-4695 Rewrite application id to file when restart
sparder application
add 698af2c KYLIN-4723 Set the configurations about shard by to cube level
add 07fd1b4 KYLIN-4724 Support to download Apache Spark 2.4.6 binary
package in download-spark.sh
add 1b63cec KYLIN-4730 Add scan bytes metric to the query results
add e573ed3 KYLIN-4705 add rest api for sparder-context url monitor
add f48a032 KYLIN-4733 The cube size is inconsistent with the size of all
segments
add 5eb9b37 KYLIN-4732 The cube size is wrong after disabling/enabling
the cube
add 24fabdc HOTFIX Fix test cases for merging segments
add e62ca6e KYLIN-4725 NSparkCubingStep returns error state when pause
build job
add be0aedd KYLIN-4449 The spark process is not actually killed after the
job be paused
add 762e7c8 KYLIN-4742 NullPointerException when auto merge segments if
exist discard jobs
add 582969d KYLIN-4744 Add tracking URL for build spark job on yarn
add 1147553 KYLIN-4660 Cleanup
add 0da41e2 KYLIN-4746 Improve build performance by reducing the count of
calling 'count()' function
add 276f906 KYLIN-4747 Use the first dimension column as sort column
within a partiton
add 63f8c84 KYLIN-4705 Monitor Spark's AppMaster for QueryServer - Part II
add 3405ab2 KYLIN-4719 Refine kylin-defaults.properties for parquet
Storage
new a31a2b1 KYLIN-4660 Fix QueryMetricsTest
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (97db36c)
\
N -- N -- N refs/heads/kylin-on-parquet-v2 (a31a2b1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore | 3 +-
.travis.yml | 4 +-
KEYS | 58 +
build/bin/check-migration-acl.sh | 2 +-
build/bin/download-spark.sh | 16 +-
.../prepare.sh => bin/kill-process-tree.sh} | 54 +-
build/bin/kylin.sh | 58 +-
build/conf/kylin-parquet-log4j.properties | 2 +
build/conf/kylin-server-log4j.properties | 18 +-
build/conf/kylin-tools-log4j.properties | 3 +-
build/deploy/context.xml | 2 +-
build/script/download-spark.sh | 75 -
build/script/prepare-libs.sh | 8 +-
.../org/apache/kylin/common/JobProcessContext.java | 10 +
.../java/org/apache/kylin/common/KylinConfig.java | 21 +-
.../org/apache/kylin/common/KylinConfigBase.java | 35 +-
.../java/org/apache/kylin/common/KylinVersion.java | 2 +-
.../java/org/apache/kylin/common/QueryContext.java | 48 +-
.../apache/kylin/common/debug/BackdoorToggles.java | 6 +
.../kylin/common/persistence/ResourceTool.java | 2 +-
.../src/main/resources/kylin-defaults.properties | 204 +-
.../kylin/common/util/AbstractKylinTestCase.java | 5 -
.../kylin/common/util/TempMetadataBuilder.java | 19 +-
.../java/org/apache/kylin/cube/CubeManager.java | 2 +-
.../java/org/apache/kylin/cube/CubeSegment.java | 2 +-
.../java/org/apache/kylin/cube/model/CubeDesc.java | 23 +-
.../model/validation/rule/FunctionRuleTest.java | 17 -
core-dictionary/pom.xml | 14 +-
.../kylin/job/execution/AbstractExecutable.java | 13 +-
.../kylin/job/execution/ExecutableManager.java | 38 +-
.../org/apache/kylin/job/execution/Idempotent.java | 2 +-
.../apache/kylin/job/execution/JobTypeEnum.java | 25 -
.../apache/kylin/measure/MeasureTypeFactory.java | 8 +-
.../apache/kylin/metadata/MetadataConstants.java | 11 +-
.../metadata/realization/RealizationType.java | 2 +-
.../kylin/engine/spark/SparkUHCDictionary.java | 2 +-
examples/sample_cube/create_sample_tables.sql | 1 +
examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv | 20000 +++++++++----------
.../template/cube_desc/kylin_sales_cube.json | 36 +-
.../template/model_desc/kylin_sales_model.json | 3 +-
.../table/DEFAULT.KYLIN_SALES--learn_kylin.json | 114 +-
.../test_kylin_cube_with_slr_left_join_desc.json | 4 +-
.../parquet_test/cube_desc/ci_left_join_cube.json | 21 +-
.../sample_local/table/DEFAULT.KYLIN_ACCOUNT.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_CAL_DT.json | 2 +-
.../table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_COUNTRY.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_SALES.json | 2 +-
examples/test_case_data/sandbox/kylin.properties | 4 +-
.../resources/query/sql_h2_uncapable/query03.sql | 5 +-
.../resources/query/sql_h2_uncapable/query04.sql | 6 +-
.../build}/conf/kylin-tools-log4j.properties | 11 +-
.../spark/classloader/DebugTomcatClassLoader.java | 147 -
.../spark/classloader/KylinItClassLoader.java | 174 -
.../spark/classloader/KylinItSparkClassLoader.java | 189 -
.../kylin/spark/classloader/SparkClassLoader.java | 22 +-
.../kylin/spark/classloader/TomcatClassLoader.java | 84 +-
...ateTimeUtils.scala => KylinDateTimeUtils.scala} | 2 +-
.../engine/spark/metadata/cube/PathManager.java | 52 +-
.../org/apache/spark/dict/NBucketDictionary.java | 2 +-
.../spark/dict/NGlobalDictBuilderAssist.scala | 4 +-
...balDictionaryV2.java => NGlobalDictionary.java} | 8 +-
.../apache/spark/sql/KylinDataFrameManager.scala | 3 +-
.../org/apache/spark/sql/KylinFunctions.scala | 34 +-
.../sql/catalyst/expressions/DictEncodeImpl.scala | 4 +-
...pExpresssions.scala => KylinExpresssions.scala} | 32 +-
.../catalyst/expressions/TimestampAddImpl.scala | 2 +-
.../catalyst/expressions/TimestampDiffImpl.scala | 10 +-
.../sql/execution/KylinFileSourceScanExec.scala | 5 +-
.../sql/execution/datasource/FilePruner.scala | 6 +-
.../datasource/ResetShufflePartition.scala | 9 +-
.../spark/sql/hive/utils/QueryMetricUtils.scala | 26 +-
.../main/scala/org/apache/spark/utils/LogEx.scala | 2 +-
kylin-spark-project/kylin-spark-engine/pom.xml | 20 +-
.../engine/spark/application/SparkApplication.java | 5 +-
.../engine/spark/builder/NBuildSourceInfo.java | 8 -
.../engine/spark/job/CubingExecutableUtil.java | 159 -
.../apache/kylin/engine/spark/job/CubingJob.java | 365 -
.../kylin/engine/spark/job/JobStepFactory.java | 12 -
.../apache/kylin/engine/spark/job/JobStepType.java | 4 +-
.../kylin/engine/spark/job/NSparkCubingJob.java | 11 +-
.../kylin/engine/spark/job/NSparkCubingStep.java | 25 +
.../kylin/engine/spark/job/NSparkCubingUtil.java | 18 +-
.../kylin/engine/spark/job/NSparkExecutable.java | 36 +-
.../kylin/engine/spark/job/NSparkMergingJob.java | 12 +-
.../NSparkUpdateMetaAndCleanupAfterMergeStep.java | 29 +-
.../merger/AfterMergeOrRefreshResourceMerger.java | 8 +-
.../kylin/engine/spark/merger/MetadataMerger.java | 2 +-
.../kylin/engine/spark/source/CsvSource.java | 6 +-
.../engine/spark/builder/CreateFlatTable.scala | 8 +-
...BuilderHelper.scala => CubeBuilderHelper.scala} | 4 +-
...ryBuilder.scala => CubeDictionaryBuilder.scala} | 14 +-
...LayoutMergeAssist.java => CubeMergeAssist.java} | 4 +-
...shotBuilder.scala => CubeSnapshotBuilder.scala} | 2 +-
...DFTableEncoder.scala => CubeTableEncoder.scala} | 10 +-
.../kylin/engine/spark/builder/DictHelper.scala | 4 +-
.../spark/builder/DictionaryBuilderHelper.java | 4 +-
.../kylin/engine/spark/job/CubeBuildJob.java | 46 +-
.../kylin/engine/spark/job/CubeMergeJob.java | 44 +-
.../kylin/engine/spark/job/CuboidAggregator.scala | 2 +-
.../engine/spark/job/ParentSourceChooser.scala | 4 +-
.../spark/job/ResourceDetectBeforeMergingJob.java | 8 +-
.../kylin/engine/spark/utils/BuildUtils.scala | 5 +-
.../kylin/engine/spark/utils/JobMetricsUtils.scala | 11 +-
.../kylin/engine/spark/utils/Repartitioner.java | 26 +-
.../kylin/query/runtime/ExpressionConverter.scala | 4 +-
.../apache/spark/sql/udf/TimestampAddImpl.scala | 2 +-
.../apache/spark/sql/udf/TimestampDiffImpl.scala | 10 +-
.../engine/spark/LocalWithSparkSessionTest.java | 62 +-
...onaryV2Test.java => NGlobalDictionaryTest.java} | 16 +-
.../kylin/engine/spark/job/JobStepFactoryTest.java | 11 +-
.../kylin/engine/spark/job/SparkCubingJobTest.java | 3 +-
.../kylin/engine/spark/source/CsvSourceTest.java | 2 +-
.../src/test/resources/log4j.properties | 26 +-
.../engine/spark/builder/TestCreateFlatTable.scala | 2 +-
.../engine/spark/builder/TestGlobalDictBuild.scala | 6 +-
.../engine/spark/builder/TestSnapshotBuilder.scala | 6 +-
.../kylin/engine/spark/job/TestCubeBuildJob.scala | 6 +-
.../kylin/engine/spark/job/TestUdfManager.scala | 2 +-
kylin-spark-project/kylin-spark-query/pom.xml | 7 +-
.../kylin/query/pushdown/SparkSqlClient.scala | 12 +-
.../kylin/query/runtime/DerivedProcess.scala | 1 -
.../kylin/query/runtime/ExpressionConverter.scala | 4 +-
.../apache/kylin/query/runtime/RuntimeHelper.scala | 1 -
.../kylin/query/runtime/SparderRexVisitor.scala | 4 +-
.../apache/kylin/query/runtime/SparkEngine.java | 12 +-
.../kylin/query/runtime/plans/ProjectPlan.scala | 4 +-
.../kylin/query/runtime/plans/ResultPlan.scala | 37 +-
.../org/apache/spark/sql/SparderContext.scala | 36 +-
.../pushdown/PushDownRunnerSparkImplTest.java | 10 +-
kylin-spark-project/kylin-spark-test/pom.xml | 119 +-
...ldAndQueryTest.java => NBuildAndQueryTest.java} | 108 +-
.../apache/kylin/engine/spark2/NExecAndComp.java | 73 +-
.../spark2/NManualBuildAndQueryCuboidTest.java | 4 +-
.../spark2/file_pruning/NFilePruningTest.java | 4 +-
.../kylin/engine/spark2/utils/QueryUtil.java | 26 +-
metrics-reporter-hive/pom.xml | 5 +
.../kylin/metrics/lib/impl/hive/HiveProducer.java | 1 +
pom.xml | 11 +-
.../query/relnode/OLAPToEnumerableConverter.java | 21 +-
.../org/apache/kylin/query/routing/Candidate.java | 1 -
.../apache/kylin/query/routing/RoutingRule.java | 2 +-
query/src/test/resources/log4j.properties | 2 +-
server-base/pom.xml | 9 +-
.../kylin/rest/controller/AdminController.java | 9 +
.../kylin/rest/controller/CubeController.java | 45 +-
.../kylin/rest/init/InitialSparderContext.java | 31 +-
.../apache/kylin/rest/job/KylinHealthCheckJob.java | 4 +-
.../apache/kylin/rest/job/MetadataCleanupJob.java | 2 +-
.../apache/kylin/rest/response/SQLResponse.java | 40 +
.../{HBaseResponse.java => StorageResponse.java} | 4 +-
.../kylin/rest/service/BadQueryDetector.java | 2 +-
.../org/apache/kylin/rest/service/CubeService.java | 59 +-
.../apache/kylin/rest/service/HBaseInfoUtil.java | 6 +-
.../org/apache/kylin/rest/service/JobService.java | 4 +-
.../apache/kylin/rest/service/QueryService.java | 46 +-
.../kylin/rest/job/MetadataCleanupJobTest.java | 2 +-
.../kylin/rest/response/SQLResponseTest.java | 3 +-
server/src/main/resources/applicationContext.xml | 1 +
.../kylin/rest/service/AdminServiceTest.java | 4 +-
source-hive/pom.xml | 10 +-
.../apache/kylin/source/hive/HiveFlinkInput.java | 92 -
.../apache/kylin/source/hive/HiveInputBase.java | 57 -
.../source}/hive/HiveMetaStoreClientFactory.java | 2 +-
.../org/apache/kylin/source/hive/HiveSource.java | 13 +-
.../apache/kylin/source/hive/HiveSparkInput.java | 54 -
.../tool/metrics/systemcube/CubeDescCreator.java | 2 +-
webapp/app/js/controllers/admin.js | 16 +-
webapp/app/js/controllers/cube.js | 4 +-
webapp/app/js/controllers/cubes.js | 2 +
webapp/app/js/filters/filter.js | 4 +-
webapp/app/js/model/cubeConfig.js | 2 +-
webapp/app/js/services/admin.js | 3 +-
webapp/app/js/services/cubes.js | 2 +-
webapp/app/partials/admin/admin.html | 6 +-
webapp/app/partials/admin/config.html | 3 +
.../partials/cubeDesigner/advanced_settings.html | 722 +-
webapp/app/partials/cubes/cube_detail.html | 2 +-
webapp/app/partials/cubes/cubes.html | 2 +-
webapp/app/partials/tables/source_table_tree.html | 4 +-
180 files changed, 11745 insertions(+), 12953 deletions(-)
copy build/{script/prepare.sh => bin/kill-process-tree.sh} (51%)
mode change 100755 => 100644
delete mode 100755 build/script/download-spark.sh
delete mode 100644
core-job/src/main/java/org/apache/kylin/job/execution/JobTypeEnum.java
copy {build => kylin-spark-project/build}/conf/kylin-tools-log4j.properties
(73%)
delete mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/DebugTomcatClassLoader.java
delete mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/KylinItClassLoader.java
delete mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/KylinItSparkClassLoader.java
rename
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/common/util/{KapDateTimeUtils.scala
=> KylinDateTimeUtils.scala} (99%)
rename
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/{NGlobalDictionaryV2.java
=> NGlobalDictionary.java} (94%)
rename
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/{KapExpresssions.scala
=> KylinExpresssions.scala} (93%)
delete mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/CubingExecutableUtil.java
delete mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/CubingJob.java
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{DFBuilderHelper.scala
=> CubeBuilderHelper.scala} (97%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{DFDictionaryBuilder.scala
=> CubeDictionaryBuilder.scala} (88%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{DFLayoutMergeAssist.java
=> CubeMergeAssist.java} (97%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{DFSnapshotBuilder.scala
=> CubeSnapshotBuilder.scala} (99%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{DFTableEncoder.scala
=> CubeTableEncoder.scala} (87%)
rename
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/dict/{NGlobalDictionaryV2Test.java
=> NGlobalDictionaryTest.java} (89%)
copy build/conf/kylin-tools-log4j.properties =>
kylin-spark-project/kylin-spark-engine/src/test/resources/log4j.properties (59%)
rename
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/{NManualBuildAndQueryTest.java
=> NBuildAndQueryTest.java} (77%)
copy
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/CubeStorageQuery.java
=>
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparderContext.java
(56%)
rename
server-base/src/main/java/org/apache/kylin/rest/response/{HBaseResponse.java =>
StorageResponse.java} (97%)
delete mode 100644
source-hive/src/main/java/org/apache/kylin/source/hive/HiveFlinkInput.java
rename {metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl
=>
source-hive/src/main/java/org/apache/kylin/source}/hive/HiveMetaStoreClientFactory.java
(98%)
delete mode 100644
source-hive/src/main/java/org/apache/kylin/source/hive/HiveSparkInput.java
