This is an automated email from the ASF dual-hosted git repository. englefly pushed a commit to branch improvement/statistics-linear-no-hot-value in repository https://gitbox.apache.org/repos/asf/doris.git
commit a4872e0bdbd9f725f7ae4bb1842fb13dc4ffb7ff Author: englefly <[email protected]> AuthorDate: Sat Oct 17 08:37:07 2026 +0800 [improvement](statistics) Stop collecting hot value in the linear sample estimator ### What problem does this PR solve? Issue Number: None Related PR: #52048 Problem Summary: `LINEAR_ANALYZE_TEMPLATE` collected hot values in a third CTE: it grouped the sampled column by value, counted each group, and kept the top `hot_value_collect_count` groups. That CTE re-reads the whole sample, and it is the dominant cost of the template - the one sample analyze picks for a single distribution column / single unique key column. The same template also serves the full-scan fallback of a sample analyze (the requested sample rows exceed the table, or the picked tablets hold fewer rows than requested), where the extra aggregation is not bounded by the sample size at all. This PR removes that CTE. `hot_value` keeps its position in the result set and is reported as NULL, exactly like `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE` and `MERGE_PARTITION_TEMPLATE` already do, and every reader already treats NULL as "not collected" (`ColumnStatistic.hotValues == null`). Hot values are still collected by `DUJ1_ANALYZE_TEMPLATE`, which derives them from the grouped sample it already builds for the NDV estimate, and by a full analyze that asks for them explicitly (`ANALYZE TABLE ... WITH HOT VALUE`). The params that only served the removed CTE (`subStringColName`, `rowCount2` on the linear path, `hotValueCollectCount`) are now filled for the DUJ1 algorithm only. As a side effect, the linear branch of `PluginDrivenSampleAnalysisTask` no longer renders an unsubstituted `${subStringColName}` into its SQL: it never set that param (the legacy `HMSAnalysisTask` it was ported from did not either), so a bucketed plain-hive table that took that branch got invalid SQL before this change. Behaviour changes: - A sample analyze no longer collects hot values for columns estimated with the linear estimator: a single distribution column / single unique key column, and any sample analyze that falls back to a full table scan. - `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was already rejected before this change; its message changes from "Sample analyze always collects hot value" to "Sample analyze does not support WITH HOT VALUE", because the old wording is no longer true. ### Release note Sample analyze no longer collects hot values for columns that use the linear NDV estimator (single distribution / single key column, and the full-scan fallback of a sample analyze), which removes a second aggregation over the sampled data. Hot values are still collected by sample analyze for columns estimated with DUJ1, and by `ANALYZE TABLE ... WITH HOT VALUE` in a full analyze. ### Check List (For Author) - Test: Unit Test - FE UT: `OlapAnalysisTaskTest` (25 tests), `AnalyzeTableCommandTest` (5 tests) - Regression: `test_hot_value` and `test_full_analyze_hot_value` updated for the new behaviour; not run locally (needs a built FE + cluster), covered by CI - Behavior changed: Yes. Sample analyze no longer collects hot values for columns analyzed by the linear estimator, and the `WITH HOT VALUE` rejection message for sample analyze changed. - Does this need documentation: Yes. Any statement that sample analyze collects hot values needs to be narrowed; no docs PR yet. --- .../trees/plans/commands/AnalyzeCommand.java | 2 +- .../statistics/analysis/BaseAnalysisTask.java | 25 +++++----- .../statistics/analysis/OlapAnalysisTask.java | 18 +++---- .../analysis/PluginDrivenSampleAnalysisTask.java | 3 +- .../doris/statistics/model/ColumnStatistic.java | 3 +- .../plans/commands/AnalyzeTableCommandTest.java | 2 +- .../statistics/analysis/OlapAnalysisTaskTest.java | 56 +++++++++++++--------- .../statistics/test_full_analyze_hot_value.groovy | 54 +++++++++++++-------- .../suites/statistics/test_hot_value.groovy | 14 ++---- 9 files changed, 98 insertions(+), 79 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java index 917511885e6..2b612683458 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java @@ -146,7 +146,7 @@ public abstract class AnalyzeCommand extends Command implements ForwardWithSync analyzeProperties.check(); } if (analyzeProperties.hasCollectHotValue() && getAnalysisMethod() == AnalysisInfo.AnalysisMethod.SAMPLE) { - throw new AnalysisException("Sample analyze always collects hot value"); + throw new AnalysisException("Sample analyze does not support WITH HOT VALUE"); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java index 5bbb96e1134..f18e2af8f2d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java @@ -142,6 +142,14 @@ public abstract class BaseAnalysisTask { + "FROM (SELECT ${colName}${lengthAssert} " + "FROM ${catalogName}.${dbName}.${tblName} ${index}) __lc_t"; + // Hot values are intentionally NOT collected by the linear estimator. The extra + // GROUP BY over the sampled column re-reads the whole sample and is the dominant cost + // of this template, which is used for every single distribution / single key column + // sample analyze. `hot_value` keeps its position in the result set and is reported as + // NULL, which every reader treats as "not collected" (see ColumnStatistic#hotValues), + // so the row layout stays identical to that of the other templates. Hot values are + // still collected by DUJ1_ANALYZE_TEMPLATE, which derives them from the grouped sample + // it already builds for the NDV estimate. protected static final String LINEAR_ANALYZE_TEMPLATE = "WITH cte1 AS (" + "SELECT ${colName}${lengthAssert} " + "FROM ${catalogName}.${dbName}.${tblName} ${index} ${sampleHints} ${limit} ${preAggHint}), " @@ -159,19 +167,10 @@ public abstract class BaseAnalysisTask { + "SUBSTRING(CAST(${min} AS STRING), 1, 1024) AS `min`, " + "SUBSTRING(CAST(${max} AS STRING), 1, 1024) AS `max`, " + "${dataSizeFunction} * ${scaleFactor} AS `data_size`, " - + "NOW() FROM cte1), " - + "cte3 AS (" - + "SELECT IFNULL(GROUP_CONCAT(CONCAT(" - + "REPLACE(REPLACE(t.`column_key`, \":\", \"\\\\:\"), \";\", \"\\\\;\"), " - + "\" :\", ROUND(t.`count` / ${rowCount2}, 2)), \" ;\"), '') " - + "as `hot_value` " - + "FROM (" - + "SELECT ${subStringColName} as `hash_value`, " - + "MAX(${colName}) as `column_key`, " - + "COUNT(1) AS `count` " - + "FROM cte1 WHERE ${colName} IS NOT NULL " - + "GROUP BY `hash_value` ORDER BY `count` DESC LIMIT ${hotValueCollectCount}) t) " - + "SELECT * FROM cte2 CROSS JOIN cte3"; + + "NOW() AS `update_time`, " + + "null as `hot_value` " + + "FROM cte1) " + + "SELECT * FROM cte2"; protected static final String DUJ1_ANALYZE_TEMPLATE = "WITH cte1 AS (" + "SELECT MAX(t0.`col_value`) as `col_value`, COUNT(1) as `count`, SUM(`len`) as `column_length` " diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java index 606a2424d10..b1c3c63b675 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java @@ -124,19 +124,20 @@ public class OlapAnalysisTask extends BaseAnalysisTask { Map<String, String> params = buildSqlParams(); params.put("min", StatisticsUtil.quote(min)); params.put("max", StatisticsUtil.quote(max)); - params.put("hotValueCollectCount", String.valueOf(SessionVariable.getHotValueCollectCount())); long tableRowCount = info.indexId == -1 ? tbl.getRowCount() : ((OlapTable) tbl).getRowCountForIndex(info.indexId, false); SampleCollectInfo collectInfo = getSampleCollectInfo(tableRowCount); getSampleParams(params, tableRowCount, collectInfo); - StringSubstitutor stringSubstitutor = new StringSubstitutor(params); - String sql; + String template = LINEAR_ANALYZE_TEMPLATE; if (collectInfo.algorithm == AnalyzeSampleAlgorithm.DUJ1) { - sql = stringSubstitutor.replace(DUJ1_ANALYZE_TEMPLATE); - } else { - sql = stringSubstitutor.replace(LINEAR_ANALYZE_TEMPLATE); + // DUJ1 is the only sample algorithm that collects hot values, and it derives them + // from the grouped sample it already builds for the NDV estimate, so only it needs + // the top-N count. + params.put("hotValueCollectCount", String.valueOf(SessionVariable.getHotValueCollectCount())); + template = DUJ1_ANALYZE_TEMPLATE; } + String sql = new StringSubstitutor(params).replace(template); LOG.info("Analyze param: algorithm {}, partitionColumnTooMany {}, keyColumnTooMany {}", collectInfo.algorithm, partitionColumnSampleTooManyRows, keyColumnSampleTooManyRows); LOG.debug(sql); @@ -312,7 +313,6 @@ public class OlapAnalysisTask extends BaseAnalysisTask { params.put("rowCount", String.valueOf(tableRowCount)); params.put("type", col.getType().toString()); params.put("limit", ""); - params.put("subStringColName", getStringTypeColName(col)); // For agg table and mor unique table, set PREAGGOPEN preAggHint. if (((OlapTable) tbl).getKeysType().equals(KeysType.AGG_KEYS) @@ -362,11 +362,13 @@ public class OlapAnalysisTask extends BaseAnalysisTask { protected void setSampleParamsByAlgorithm(Map<String, String> params, long tableRowCount, AnalyzeSampleAlgorithm algorithm) { if (algorithm == AnalyzeSampleAlgorithm.DUJ1) { + // DUJ1 is the only sample algorithm that collects hot values: it derives them from + // the grouped sample it already builds for the NDV estimate below. + params.put("subStringColName", getStringTypeColName(col)); params.put("ndvFunction", getNdvFunction(String.valueOf(tableRowCount))); params.put("dataSizeFunction", getDataSizeFunction(col, true)); params.put("rowCount2", "(SELECT SUM(`count`) FROM cte1 WHERE `col_value` IS NOT NULL)"); } else { - params.put("rowCount2", "(SELECT COUNT(1) FROM cte1 WHERE ${colName} IS NOT NULL)"); // For single unique key, use count as ndv. if (isSingleUniqueKey()) { params.put("ndvFunction", String.valueOf(tableRowCount)); diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java index 91690df1eb0..135edbfcc5a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java @@ -63,7 +63,6 @@ public class PluginDrivenSampleAnalysisTask extends ExternalAnalysisTask { params.put("dataSizeFunction", getDataSizeFunction(col, false)); Pair<Double, Long> sampleInfo = getSampleInfo(); params.put("scaleFactor", String.valueOf(sampleInfo.first)); - params.put("hotValueCollectCount", String.valueOf(SessionVariable.getHotValueCollectCount())); if (LOG.isDebugEnabled()) { LOG.debug("Will do sample collection for column {}", col.getName()); } @@ -91,10 +90,10 @@ public class PluginDrivenSampleAnalysisTask extends ExternalAnalysisTask { sb.append(LINEAR_ANALYZE_TEMPLATE); params.put("ndvFunction", "ROUND(NDV(${colName}) * ${scaleFactor})"); params.put("rowCount", "ROUND(COUNT(1) * ${scaleFactor})"); - params.put("rowCount2", "(SELECT COUNT(1) FROM cte1 WHERE ${colName} IS NOT NULL)"); } else { sb.append(DUJ1_ANALYZE_TEMPLATE); params.put("subStringColName", getStringTypeColName(col)); + params.put("hotValueCollectCount", String.valueOf(SessionVariable.getHotValueCollectCount())); params.put("dataSizeFunction", getDataSizeFunction(col, true)); params.put("ndvFunction", getNdvFunction("ROUND(SUM(t1.count) * ${scaleFactor})")); params.put("rowCount", "ROUND(SUM(t1.count) * ${scaleFactor})"); diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java index 1af96763d85..66214b514b3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java @@ -101,7 +101,8 @@ public class ColumnStatistic { public final String updatedTime; /** - * hotValues == null: not collected (e.g. full analyze). Do NOT use for optimization. + * hotValues == null: not collected (e.g. full analyze, or a sample analyze that used + * the linear estimator). Do NOT use for optimization. * hotValues != null && hotValues.isEmpty(): collected but no hot values (not skewed). * hotValues != null && !hotValues.isEmpty(): collected with hot values. */ diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java index ca1c3243ce9..369644bf193 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java @@ -157,7 +157,7 @@ public class AnalyzeTableCommandTest { "ANALYZE TABLE test_db.test_tbl WITH SAMPLE ROWS 100 WITH HOT VALUE"); AnalysisException exception = Assertions.assertThrows(AnalysisException.class, () -> command.validate(ctx)); - Assertions.assertEquals("Sample analyze always collects hot value", exception.getDetailMessage()); + Assertions.assertEquals("Sample analyze does not support WITH HOT VALUE", exception.getDetailMessage()); } private Map<String, String> defaultAnalyzeProperties() { diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java index b6976a5d41f..cbd30fde0c1 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java @@ -162,13 +162,8 @@ public class OlapAnalysisTaskTest { + "ROUND(SUM(CASE WHEN `null` IS NULL THEN 1 ELSE 0 END) * ${scaleFactor}) AS `null_count`, " + "SUBSTRING(CAST('1' AS STRING), 1, 1024) AS `min`, " + "SUBSTRING(CAST('2' AS STRING), 1, 1024) AS `max`, " - + "COUNT(1) * 4 * ${scaleFactor} AS `data_size`, NOW() FROM cte1), " - + "cte3 AS (SELECT IFNULL(GROUP_CONCAT(CONCAT(REPLACE(REPLACE(t.`column_key`, " - + "\":\", \"\\\\:\"), \";\", \"\\\\;\"), \" :\", ROUND(t.`count` / ${rowCount2}, 2)), " - + "\" ;\"), '') as `hot_value` FROM (SELECT ${subStringColName} as `hash_value`, " - + "MAX(`null`) as `column_key`, COUNT(1) AS `count` FROM cte1 " - + "WHERE `null` IS NOT NULL GROUP BY `hash_value` ORDER BY `count` DESC LIMIT 10) t) " - + "SELECT * FROM cte2 CROSS JOIN cte3", sql); + + "COUNT(1) * 4 * ${scaleFactor} AS `data_size`, NOW() AS `update_time`, " + + "null as `hot_value` FROM cte1) SELECT * FROM cte2", sql); return null; }).when(olapAnalysisTask).runQuery(ArgumentMatchers.anyString()); @@ -322,8 +317,9 @@ public class OlapAnalysisTaskTest { new OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.FULL, null)); Assertions.assertEquals("1", params.get("scaleFactor")); Assertions.assertEquals("", params.get("sampleHints")); - Assertions.assertEquals("(SELECT COUNT(1) FROM cte1 WHERE ${colName} IS NOT NULL)", - params.get("rowCount2")); + // The linear estimator does not collect hot values, so it fills no hot value param. + Assertions.assertNull(params.get("rowCount2")); + Assertions.assertNull(params.get("subStringColName")); Assertions.assertEquals("ROUND(NDV(${colName}) * ${scaleFactor})", params.get("ndvFunction")); Assertions.assertNull(params.get("preAggHint")); Assertions.assertEquals("COUNT(1)", params.get("rowCount")); @@ -334,8 +330,8 @@ public class OlapAnalysisTaskTest { new OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.LINEAR, Pair.of(Lists.newArrayList(1L, 2L), 100L))); Assertions.assertEquals("TABLET(1, 2)", params.get("sampleHints")); - Assertions.assertEquals("(SELECT COUNT(1) FROM cte1 WHERE ${colName} IS NOT NULL)", - params.get("rowCount2")); + Assertions.assertNull(params.get("rowCount2")); + Assertions.assertNull(params.get("subStringColName")); Assertions.assertEquals("ROUND(NDV(${colName}) * ${scaleFactor})", params.get("ndvFunction")); params.clear(); @@ -348,6 +344,7 @@ public class OlapAnalysisTaskTest { Assertions.assertEquals("TABLET(1, 2)", params.get("sampleHints")); Assertions.assertEquals("(SELECT SUM(`count`) FROM cte1 WHERE `col_value` IS NOT NULL)", params.get("rowCount2")); + Assertions.assertEquals("${colName}", params.get("subStringColName")); Assertions.assertTrue(params.get("ndvFunction").contains("`t1`.`col_value`"), "DUJ1 ndvFunction must reference cte1 output, got: " + params.get("ndvFunction")); params.clear(); @@ -983,7 +980,7 @@ public class OlapAnalysisTaskTest { } @Test - public void testDoSampleAlwaysCollectsHotValue() throws Exception { + public void testDoSampleHotValueByAlgorithm() throws Exception { CatalogIf catalogIf = Mockito.mock(CatalogIf.class); DatabaseIf databaseIf = Mockito.mock(DatabaseIf.class); OlapTable tableIf = Mockito.mock(OlapTable.class); @@ -999,16 +996,6 @@ public class OlapAnalysisTaskTest { Mockito.doReturn(new ResultRow(Lists.newArrayList("1", "2"))).when(task).collectMinMax(); Mockito.doNothing().when(task).getSampleParams(ArgumentMatchers.any(), ArgumentMatchers.anyLong(), ArgumentMatchers.any()); - Mockito.doAnswer(invocation -> { - String sql = invocation.getArgument(0); - Assertions.assertTrue(sql.contains("as `hot_value`"), sql); - Assertions.assertTrue(sql.contains("cte3"), sql); - Assertions.assertTrue(sql.contains("CROSS JOIN cte3"), sql); - Assertions.assertTrue(sql.contains("LIMIT 10"), sql); - Assertions.assertFalse(sql.contains("null as `hot_value`"), sql); - return null; - }).when(task).runQuery(Mockito.anyString()); - task.col = new Column("testCol", Type.fromPrimitiveType(PrimitiveType.INT), true, null, null, null); task.tbl = tableIf; @@ -1021,10 +1008,35 @@ public class OlapAnalysisTaskTest { task.db = databaseIf; task.tableSample = new TableSample(false, 100L); + // The linear estimator does not collect hot values: the extra GROUP BY over the sampled + // column is the dominant cost of LINEAR_ANALYZE_TEMPLATE. + Mockito.doAnswer(invocation -> { + String sql = invocation.getArgument(0); + Assertions.assertTrue(sql.contains("null as `hot_value`"), sql); + Assertions.assertFalse(sql.contains("cte3"), sql); + Assertions.assertFalse(sql.contains("CROSS JOIN cte3"), sql); + Assertions.assertFalse(sql.contains("GROUP BY `hash_value`"), sql); + // None of the hot value params is left for the linear template to fill. + Assertions.assertFalse(sql.contains("${hotValueCollectCount}"), sql); + Assertions.assertFalse(sql.contains("${subStringColName}"), sql); + Assertions.assertFalse(sql.contains("${rowCount2}"), sql); + return null; + }).when(task).runQuery(Mockito.anyString()); Mockito.doReturn(new OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.LINEAR, Pair.of(Lists.newArrayList(1L, 2L), 100L))) .when(task).getSampleCollectInfo(Mockito.anyLong()); task.doSample(); + + // DUJ1 still collects hot values: it derives them from the grouped sample it already + // builds for the NDV estimate. + Mockito.doAnswer(invocation -> { + String sql = invocation.getArgument(0); + Assertions.assertTrue(sql.contains("as `hot_value`"), sql); + Assertions.assertTrue(sql.contains("CROSS JOIN cte3"), sql); + Assertions.assertTrue(sql.contains("LIMIT 10"), sql); + Assertions.assertFalse(sql.contains("null as `hot_value`"), sql); + return null; + }).when(task).runQuery(Mockito.anyString()); Mockito.doReturn(new OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.DUJ1, Pair.of(Lists.newArrayList(1L, 2L), 100L))) .when(task).getSampleCollectInfo(Mockito.anyLong()); diff --git a/regression-test/suites/statistics/test_full_analyze_hot_value.groovy b/regression-test/suites/statistics/test_full_analyze_hot_value.groovy index ad7df099e8d..c709f20ed42 100644 --- a/regression-test/suites/statistics/test_full_analyze_hot_value.groovy +++ b/regression-test/suites/statistics/test_full_analyze_hot_value.groovy @@ -133,42 +133,54 @@ suite("test_full_analyze_hot_value") { // All 100 rows have the same value " : ;a", so it should appear with ratio 1.0 assertEquals("' : ;a':1.0", result[0][17]) - // Test 5: Sample analyze collects hot_value by default. + // Test 5: A sample analyze whose sample request exceeds the table row count falls back to a + // full scan and uses the linear estimator, which does not collect hot_value. sql """drop stats full_hot_skew""" sql """analyze table full_hot_skew with sample rows 400 with sync""" result = sql """show column stats full_hot_skew(value1)""" logger.info("Test5 result: " + result) assertEquals(1, result.size()) - assertTrue(result[0][17].contains(":"), "Sample analyze should also collect hot_value") - hotValues = result[0][17].split(";") - assertEquals(2, hotValues.length) - assertTrue(hotValues[0].trim() == "'1':0.5" || hotValues[0].trim() == "'0':0.5") - assertTrue(hotValues[1].trim() == "'1':0.5" || hotValues[1].trim() == "'0':0.5") + assertEquals("null", String.valueOf(result[0][17])) - // Test 6: Sample analyze rejects WITH HOT VALUE because it always collects hot_value. + // Test 6: Sample analyze rejects WITH HOT VALUE because it cannot honor the flag: inside a + // sample analyze hot values only come from the DUJ1 estimator. test { sql """analyze table full_hot_skew with sample rows 400 with sync with hot value""" - exception "Sample analyze always collects hot value" + exception "Sample analyze does not support WITH HOT VALUE" } - // Test 7: Explicit full analyze produces same hot_value as sample analyze for same data. - sql """drop stats full_hot_skew""" - sql """analyze table full_hot_skew with sync with hot value""" - def fullResult = sql """show column stats full_hot_skew(value1)""" + // Test 7: Explicit full analyze produces the same hot_value as a sample analyze that really + // samples. The column value is uniform, so the sampled rows always carry the same + // distribution as the whole table, and the table is larger than the requested sample rows, + // so sample analyze uses the DUJ1 estimator instead of falling back to a full scan. + sql """drop table if exists full_hot_sample_agree""" + sql """CREATE TABLE full_hot_sample_agree ( + key1 int NULL, + value1 varchar(50) NULL + )ENGINE=OLAP + DUPLICATE KEY(`key1`) + COMMENT "OLAP" + DISTRIBUTED BY HASH(`key1`) BUCKETS 2 + PROPERTIES ( + "replication_num" = "1" + ) + """ + sql """insert into full_hot_sample_agree select number, " : ;a" from numbers("number"="10000")""" + wait_row_count_reported("test_full_analyze_hot_value", "full_hot_sample_agree", 0, 4, "10000") + + sql """analyze table full_hot_sample_agree with sync with hot value""" + def fullResult = sql """show column stats full_hot_sample_agree(value1)""" logger.info("Test7 full result: " + fullResult) assertEquals(1, fullResult.size()) - assertTrue(fullResult[0][17].contains(":")) - def fullParts = fullResult[0][17].split(";").collect { it.trim() }.sort() + assertEquals("' : ;a':1.0", fullResult[0][17]) - sql """drop stats full_hot_skew""" - sql """analyze table full_hot_skew with sample rows 40000 with sync""" - def sampleResult = sql """show column stats full_hot_skew(value1)""" + sql """drop stats full_hot_sample_agree""" + sql """analyze table full_hot_sample_agree with sample rows 400 with sync""" + def sampleResult = sql """show column stats full_hot_sample_agree(value1)""" logger.info("Test7 sample result: " + sampleResult) assertEquals(1, sampleResult.size()) - assertTrue(sampleResult[0][17].contains(":")) - // Both full and sample should produce the same hot_value entries (order may differ) - def sampleParts = sampleResult[0][17].split(";").collect { it.trim() }.sort() - assertEquals(fullParts, sampleParts) + // Both full and sample should produce the same hot_value entry. + assertEquals(fullResult[0][17], sampleResult[0][17]) // Test 8: Explicit full analyze on empty table should produce an empty hot_value string. sql """drop table if exists full_hot_empty""" diff --git a/regression-test/suites/statistics/test_hot_value.groovy b/regression-test/suites/statistics/test_hot_value.groovy index 9759adb8ec1..55d89766e85 100644 --- a/regression-test/suites/statistics/test_hot_value.groovy +++ b/regression-test/suites/statistics/test_hot_value.groovy @@ -136,6 +136,8 @@ suite("test_hot_value") { assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5") sql """drop stats test1""" + // The requested sample rows are larger than the table, so sample analyze falls back to a + // full scan with the linear estimator, which does not collect hot_value. sql """analyze table test1 with sample rows 40000 with sync""" result = sql """show column stats test1(key1)""" logger.info("result " + result) @@ -143,22 +145,14 @@ suite("test_hot_value") { assertEquals("10000.0", result[0][2]) result = sql """show column stats test1(value1)""" logger.info("2. result " + result) - logger.info("result.size()" + result.size()) assertEquals(1, result.size()) - logger.info("result[0][2] " + result[0][2]) assertEquals("10000.0", result[0][2]) - hotValues = result[0][17].split(";") - assertEquals(2, hotValues.length) - assertTrue(hotValues[0] == "'1':0.5" || hotValues[0] == "'0':0.5") - assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5") + assertEquals("null", String.valueOf(result[0][17])) result = sql """show column cached stats test1(value1)""" logger.info("result " + result) assertEquals(1, result.size()) assertEquals("10000.0", result[0][2]) - hotValues = result[0][17].split(";") - assertEquals(2, hotValues.length) - assertTrue(hotValues[0] == "'1':0.5" || hotValues[0] == "'0':0.5") - assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5") + assertEquals("null", String.valueOf(result[0][17])) sql """alter table test1 modify column value1 set stats ('row_count'='100.0', 'ndv'='5.0', 'num_nulls'='0.0', 'data_size'='34.0', 'min_value'='AFRICA', 'max_value'='MIDDLE EAST', 'hot_values'='aaa :0.8');""" result = sql """show column stats test1(value1)""" --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
