morningman opened a new pull request, #69311:
URL: https://github.com/apache/doris/pull/69311
### What problem does this PR solve?
Issue Number: None
Related PR: #69238, #69240, #69242
Problem Summary:
**In short.** Row binlog (`binlog.format = 'ROW'`), the change stream that
Table Stream and incremental materialized views (IVM) read, is renamed to
**changelog**, and it gets its own configuration instead of riding on CCR's
`BinlogConfig`. "Binlog" names a change stream in one fixed format; changelog
is the general notion, and a binlog can later be built on top of it. The
feature exists only on master and is documented as experimental, so this is a
hard cut: there is no code that reads data written under the old names.
Instead, guardrails make leftovers from an earlier master build fail with a
clear message instead of crashing a process or being silently misread. CCR
binlog and MySQL binlog CDC do not change.
**Background.**
- A row binlog table writes every change of a base tablet into a companion
tablet. The companion schema carries the key, the after values, three hidden
columns `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`,
and, with `binlog.need_historical_value`, a before-image column
`__BEFORE__<col>__` per value column. The changes are read with the
`binlog(...)` table function, `@incr`, Table Stream and IVM refresh.
- It was switched on with CCR's properties: `binlog.enable = true` plus
`binlog.format = ROW`. It shared `binlog.enable` and `binlog.ttl_seconds` with
CCR, and `enable_feature_binlog` gated CCR, row binlog table creation, TSO
allocation, Table Stream INSERT and TTL cleanup alike.
- None of this is on a release branch (3.1 / 4.0 / 4.1 / 4.2). The
documentation marks it experimental, and `mtmv/ivm/AGENTS.md` allows
incompatible changes before 2027-01. Master already made incompatible changes
to it in #64133, #65810 and #66889.
**The problem, and what it cost.**
1. **One word for two features.** CCR binlog (cross-cluster replication) and
row binlog share property names, a switch, a configuration object and most code
names, so every reader has to work out which one a `binlog` means.
2. **Row binlog leaks into every table.** Every OLAP table created on master
persists `binlog.format` and `binlog.need_historical_value` and prints them in
SHOW CREATE TABLE, CCR tables included; a 4.x CCR downstream rejects that DDL
as unknown properties.
3. **The shared keys interact badly.**
- `ALTER DATABASE ... SET ("binlog.enable" = "true")` on a database
whose default is ROW validates the tables with the CCR rules, but what takes
effect is the ROW default.
- A table with `binlog.format = ROW` and `binlog.enable = false` can
never be enabled: the error says "can't disable binlog when format is [Row]".
4. **Before-image columns clash with user columns** (reproduced on master
before this change). `__BEFORE__<col>__` is not under the reserved `__DORIS_`
prefix, and schema change matched it with `contains("__BEFORE__")`:
| User column | What happened |
|---|---|
| `__BEFORE__v1__` next to `v1` | the companion index had two columns named
`__BEFORE__v1__`; `@incr` DETAIL returned after values in the UPDATE_BEFORE
rows; the column could not be named in `binlog()` ("ambiguous"); as a key
column every `@incr` read failed; after `DROP COLUMN v1` every load failed |
| `x__BEFORE__y` as the last value column | after `ADD COLUMN v2` the
companion index put `v2` before it, so the two columns swapped values in every
row written afterwards, without any error; with different types every later
load failed with a bad cast |
5. FE and BE had two families of change codes with the same names and
different values (`DELETE` is 1 in FE and 2 in BE).
**How this PR fixes it.**
1. **New names.** User-visible ones:
| Old | New |
|---|---|
| `"binlog.enable" = "true"` + `"binlog.format" = "ROW"` |
`"changelog.enable" = "true"` |
| `binlog.ttl_seconds` (on a ROW table or database),
`binlog.need_historical_value` | `changelog.ttl_seconds`,
`changelog.need_historical_value` |
| internal `__binlog_config_version` | `__changelog_config_version` (no
longer shown in `information_schema.table_properties`) |
| `binlog(...)` table function | `changelog(...)` |
| `__DORIS_BINLOG_{TSO,LSN,OP}__` | `__DORIS_CHANGELOG_{TSO,LSN,OP}__` |
| `__BEFORE__<col>__` | `__DORIS_CHANGELOG_BEFORE_<col>__` |
| companion index `binlog<Row>(t)` | `changelog(t)` |
| SHOW DATA `BinlogSize` / `LocalBinlogSize`, SHOW PARTITIONS `BinlogSize`,
SHOW TABLET(S) / PROC `IsRowBinlog`,
`RowBinlog{Mismatch,Redundant}{Num,Tablets}`, `RowBinlogRepairReason` |
`ChangelogSize`, `LocalChangelogSize`, `IsChangelog`, `Changelog...` |
| IVM fallback reasons `BINLOG_BROKEN`, `BINLOG_NOT_ENABLED` |
`CHANGELOG_BROKEN`, `CHANGELOG_NOT_ENABLED` |
| FE `tablet_binlog_missing_timeout_second`,
`tablet_binlog_missing_max_times` | `tablet_changelog_missing_*` |
| BE `binlog_compaction_*` (6), `max_binlog_compaction_threads`, cloud
`binlog_compaction_thread_num_factor` | `changelog_compaction_*`,
`max_changelog_compaction_threads`, `changelog_compaction_thread_num_factor` |
| metrics `compaction_task_state_total{type="binlog"}`,
`tablet_binlog_max_compaction_score`; bvars `row_binlog_ttl_*`; thread pools |
`type="changelog"`, `tablet_changelog_max_compaction_score`, `changelog_ttl_*`,
`Changelog...` pools |
| HTTP `compact_type=row_binlog_ttl`; compaction policy `"binlog"`; load
profile `RowsetBuilder {id} binlog<row>` | `changelog_ttl`; `"changelog"`;
`RowsetBuilder {id} changelog` |
CCR keeps every name: `binlog.enable / ttl_seconds / max_bytes /
max_history_nums`, `SHOW PROC '/binlog'`, getBinlog / ingest_binlog,
`BinlogManager`.
2. **Separate configuration.**
- New `ChangelogConfig` (`enable`, `ttlSeconds`, `needHistoricalValue`,
`configVersion`), stored as `changelog.*` keys in the table / database property
map. `BinlogConfig` is back to the four CCR fields, so plain and CCR tables
persist and print only the four `binlog.*` keys, as on 4.x.
- The wire format does not change. `TabletBinlogConfig.of` is now the
only place that builds the `TBinlogConfig` / `BinlogConfigPB` sent to BE and
the meta service: a changelog table still sends `enable = true`, `binlog_format
= 1` (now named `CHANGELOG`), its TTL, `need_historical_value` and
`config_version`; plain and CCR tables send exactly what they sent before. The
enum mapping is explicit instead of `valueOf(name())`.
- The TTL generation protocol (short TTL first, published after every
tablet accepted it) moves unchanged to the changelog configuration.
- `changelog.enable` and `changelog.need_historical_value` can only be
set at CREATE; `changelog.ttl_seconds` can be altered. `binlog.format` and
`binlog.need_historical_value` are rejected in every DDL, naming their
replacement.
- In a database whose default is `"changelog.enable" = "true"`, SHOW
CREATE TABLE of a table without changelog prints `"changelog.enable" = "false"`
(output only), so CREATE TABLE LIKE and re-running the DDL keep it out of the
default. Temporary tables never have changelog: an explicit `"changelog.enable"
= "true"` fails, a database default is ignored.
- CCR and changelog stay mutually exclusive, now by explicit checks in
CREATE / ALTER DATABASE, CREATE / ALTER TABLE and RECOVER TABLE: the CCR full
sync relies on backup / restore, which changelog tables do not support, and one
tablet carries only one `BinlogConfigPB`.
- New switch `enable_feature_changelog` (FE and BE) gates changelog
tables, TSO allocation, Table Stream INSERT, TTL cleanup and the changelog part
of SHOW CREATE. `enable_feature_binlog` is back to CCR only.
3. **IDL names** of the row-only elements are renamed; ids, enum values and
types are unchanged, and shared names (`TBinlogConfig`, `BinlogConfigPB`,
`binlog_config`, `binlog_format`) stay:
| File | Old (id) | New |
|---|---|---|
| AgentService.thrift | `TTabletRole.TABLET_ROLE_ROW_BINLOG` (1),
`TBinlogFormat.ROW` (1) | `TABLET_ROLE_CHANGELOG`, `CHANGELOG` |
| Descriptors.thrift | `TOlapTableIndexSchema.row_binlog_id` (7),
`TOlapTableSchemaParam.row_binlog_index_schemas` (17) | `changelog_index_id`,
`changelog_index_schemas` |
| HeartbeatService.thrift | `TMasterInfo.row_binlog_ttl_reference_tso` (15)
| `changelog_ttl_reference_tso` |
| PlanNodes.thrift | `TBinlogScanType`, `TPaloScanRange.binlog_scan_type`
(12), `TOlapScanNode.read_row_binlog` (26) | `TChangelogScanType`,
`changelog_scan_type`, `read_changelog` |
| olap_file.proto | `RowsetMeta(Cloud)PB.is_row_binlog` (58),
`TabletMeta(Cloud)PB.binlog_tablet_id` (37 / 42),
`TabletRolePB.TABLET_ROLE_ROW_BINLOG` (1), `BinlogFormatPB.ROW` (1) |
`is_changelog`, `changelog_tablet_id`, `TABLET_ROLE_CHANGELOG`, `CHANGELOG` |
| descriptors.proto | `POlapTableIndexSchema.row_binlog_id` (6),
`POlapTableSchemaParam.row_binlog_index_schemas` (18) | `changelog_index_id`,
`changelog_index_schemas` |
| internal_service.proto | `PTabletWithPartition.binlog_tablet_id` (3),
`PSyncTabletMetaRequest.discover_row_binlog_ttl` (2) | `changelog_tablet_id`,
`discover_changelog_ttl` |
| cloud.proto | `TabletCompactionJobPB.row_binlog_ttl_seconds` (31) |
`changelog_ttl_seconds` |
4. **Guardrails for leftovers of an earlier master build.**
- FE, on image load, edit log replay and backup restore: plain and CCR
tables and databases drop the three old keys and behave exactly as before. A
table that still says `binlog.format = ROW` with `binlog.enable = true` cannot
be used except by DROP; queries (including `@incr`, `changelog()` and streams),
every load path and DDL fail with `Table <db>.<t> was created with
"binlog.format" = "ROW", which is no longer supported; drop the table and
create it again with "changelog.enable" = "true"`. Background daemons skip it
instead of throwing or rebuilding its companion tablets. A database whose
default was ROW gets `binlog.enable = false` back, with a warning, so it does
not turn into a CCR database.
- BE: publishing a load that an older BE committed on a changelog rowset
without the new hidden columns fails with an error instead of reading column -1
(which crashed the BE on every publish retry). A changelog tablet uses the
changelog compaction policy by its role, whatever policy string its meta
carries; an unknown policy string falls back to size_based with a warning
(looking it up used to throw and abort the BE on every restart); the derive
stage returns an error instead of a `CHECK` when the hidden columns are
missing; a changelog read or compaction without the TSO column fails instead of
writing NULL TSO.
5. **Before-image columns** are `__DORIS_CHANGELOG_BEFORE_<col>__`. A user
cannot create a column under `__DORIS_`, every check uses this prefix, and
`IvmUtil.isCommonHiddenSlot` excludes it (a user column ending in `_COL` would
otherwise look like a common hidden slot).
6. **Change codes** have two name families: FE `CHANGE_TYPE_*` (folded
change types) and BE `CHANGELOG_OP_*` (persisted op values). Values do not
change.
7. Every persisted or externally hard-coded name has an `//
ATTN(changelog-rename): formerly "<old>"` line at its definition, so a later
compatibility layer knows what to map.
**Results.**
| Scenario | Before | After |
|---|---|---|
| SHOW CREATE TABLE of a plain or CCR table | prints `binlog.format`, and
`binlog.need_historical_value` on ROW tables | prints the four CCR `binlog.*`
keys only (plus `"changelog.enable" = "false"` in a database whose default
enables changelog) |
| `ALTER DATABASE ... SET ("binlog.enable"="true")` on a database whose
tables use changelog | CCR checks, ROW default applied | rejected: `Can not
enable binlog in database <db> because table <t> enables changelog` |
| user column `__BEFORE__v1__` / `x__BEFORE__y` with before images | wrong
or swapped values, failing loads | ordinary user columns; values correct
(`test_changelog_before_column_name`) |
| FE and BE of this PR started on metadata written by an earlier master
build (9 ROW tables) | — | both start; one WARN per old table; SHOW / DESC
work; SELECT, INSERT, UPDATE, DELETE, ALTER, TRUNCATE, `@incr`, `changelog()`,
CREATE TABLE LIKE and CREATE STREAM fail with the message above; DROP works and
removes base and companion tablets; no repeating errors afterwards |
| BE restarted on old changelog tablets with policy `"binlog"` | looking the
policy up throws `std::out_of_range` (local mode) | the tablets load with role
changelog and policy `"changelog"` |
**Upgrade from an earlier master build** (none of this applies to a release):
- Before upgrading, drop every table created with `"binlog.format" = "ROW"`
together with its Table Streams and incremental MVs, and set `"binlog.enable" =
"false"` on databases whose default was ROW. Recreate them with
`"changelog.enable" = "true"` afterwards. Do not create or alter changelog
tables while FEs of both versions run; downgrade is not supported.
- Add `enable_feature_changelog = true` to fe.conf and be.conf;
`enable_feature_binlog` no longer enables changelog or TSO.
- Renamed FE / BE configs are not aliased: old names in conf files are
ignored. Update dashboards that use the renamed metrics, bvars, thread-pool
labels and SHOW columns.
- SQL and views that reference `binlog(...)`, the hidden columns or the
before columns must use the new names. Before-image columns now carry the
reserved `__DORIS_` prefix, so `CREATE VIEW` / CTAS / MTMV `AS SELECT * FROM
changelog(...)` on a table with `changelog.need_historical_value` has to list
its columns (as it already had to for the TSO / LSN / OP columns). Flink Doris
Connector incremental reads need a connector release that knows the new hidden
column names.
### Release note
Rename the experimental row binlog to changelog: `"changelog.enable" =
"true"` replaces `"binlog.enable" = "true"` with `"binlog.format" = "ROW"`,
`changelog.ttl_seconds` and `changelog.need_historical_value` replace the ROW
uses of `binlog.ttl_seconds` and `binlog.need_historical_value`, the `binlog()`
table function becomes `changelog()`, hidden columns become
`__DORIS_CHANGELOG_{TSO,LSN,OP}__` and before-image columns
`__DORIS_CHANGELOG_BEFORE_<col>__`, and the new switch
`enable_feature_changelog` (FE and BE) replaces `enable_feature_binlog` for
changelog. Tables created by earlier master builds with `"binlog.format" =
"ROW"` must be dropped and recreated.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- FE UT: the 81 changed test classes, 1186 tests, pass. BE UT: the 62
changed suites, 692 tests, pass (3 unrelated skips).
- Regression on a local cluster: `changelog_p0` 25/25, `table_stream_p0`
16/16, `mtmv_p0/ivm` 109/109, `mtmv_ivm_restart_p0` 2/2, `time_travel_p0` 5/5,
`tso_p0` 3/3, `auth_p0/test_changelog_tvf_auth`,
`query_p0/system/test_table_properties`,
`show_p0/test_show_create_table_and_views_nereids`,
`bloom_filter_p0/test_bloom_filter_index`. `unique_with_mow_p0` 107/109 and
`temp_table_p0/test_temp_table` fail only on the local object storage
credentials (S3 LOAD / CREATE REPOSITORY); the binlog checks of
`test_temp_table` pass by hand. `cloud_p0` was not run locally.
- `.out` regenerated: `test_changelog_property_keys`,
`test_changelog_cluster_key`, the new `test_changelog_before_column_name`,
`test_table_properties` (only the rows of the three removed keys go away) and
`test_show_create_table_and_views_nereids` (also drops 8 blocks the suite has
not produced since #68449). `test_group_commit_interval_ms_property.out` is
deleted: the suite has no `qt_` call, so the file was never read, and it still
held the removed `binlog.format` key. Result tags that said binlog are renamed,
with regenerated `.out` showing only the tag lines changed.
- Manual: the upgrade check in the Results table, on metadata written by
master `15eb3f4e432`.
- `CreateViewTest.testChangelogTtlViewAnalysis` now waits for the TSO
daemon to calibrate after it turns the switch on; the `@incr` view it creates
needs a TSO, and the test raced with the daemon (the same race exists on master
with `enable_feature_binlog`).
- Behavior changed:
- [ ] No.
- [x] Yes. The experimental row binlog is renamed to changelog
(properties, table function, hidden columns, configs, metrics, SHOW columns);
see the upgrade notes above.
- Does this need documentation?
- [ ] No.
- [x] Yes. The doris-website changelog / Table Stream pages follow in a
separate PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
