This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5567a556813 [refactor](row-binlog) Derive binlog column indexes from
column names (#69242)
5567a556813 is described below
commit 5567a556813e5a66046a03ada5690e45a2f2dd2d
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Oct 19 18:38:38 2026 +0100
[refactor](row-binlog) Derive binlog column indexes from column names
(#69242)
### What problem does this PR solve?
Issue Number: None
Related PR: #65810, #69238
Problem Summary:
**In short.** Row binlog tablets keep the positions of their three
hidden columns (`__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`,
`__DORIS_BINLOG_OP__`) in `TabletSchema`. Since #65810 the FE no longer
sends them, so every row binlog tablet created since then has -1 in its
tablet meta. Nothing reads that -1 today, but any new reader would. Only
the BE unit tests still fill these fields, and that takes them down a
path production never runs. This PR makes the BE derive the three
positions from the column names whenever a schema is loaded, and deletes
the thrift and protobuf fields that used to carry them. It also removes
the `reserved` lines and `// N: deprecated` comments that #69238 added
for other deleted row binlog fields: none of these ids was ever in a
release, and only ids that shipped in a release need to stay put.
Nothing that runs today changes.
**Background.**
- A row binlog tablet stores change events. Besides the user columns,
its schema has three hidden columns, `__DORIS_BINLOG_TSO__`,
`__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps
their positions in `_binlog_{tso,lsn,op}_col_idx`.
- The positions came from thrift `TTabletSchema` 28-30 when a tablet was
created, and were persisted in `TabletSchemaPB` 41-43
(`TabletSchemaCloudPB` 42-44).
- Schemas built column by column already derive them from the names:
`TabletSchema::append_column` and `build_current_tablet_schema`. Only
`TabletSchema::init_from_pb`, which loads a schema from its PB, took the
stored values.
**The problem.**
- #65810 stopped the FE from setting `TTabletSchema` 28-30
(`CreateReplicaTask`, and `CloudInternalCatalog` for cloud). Every row
binlog tablet created since then has -1 in:
- its tablet meta schema;
- the meta service schema entry, in cloud mode;
- every schema loaded from those, such as cloud rowsets synced from the
meta service and compaction outputs.
- Nothing reads these -1 today. Every reader gets a schema whose
positions are derived from the names: loads
(`build_current_tablet_schema`), scans and the publish-phase writer
(`append_column`), and the rowset written by the same load. Still:
1. Any new code that reads the positions from the tablet schema gets -1.
2. If a load ever came with an index schema without column unique ids,
`RowsetBuilder` would fall back to copying the tablet schema, and row
binlog derivation would hit `CHECK(idx >= 0)`.
3. The BE unit tests (`create_row_binlog_tablet_schema` and others)
still set 28-30 or the PB fields by hand, so they exercise a path
production never takes.
**How this PR fixes it.**
1. `TabletSchema::init_from_pb` derives the three positions with
`field_index(<column name>)`, as `append_column` does. A schema now has
the same positions whether it was built column by column or loaded from
a PB.
2. The stored values are no longer needed, so the fields are deleted.
They never shipped in a release, so their ids are not reserved:
| File | Message | Ids | Deleted fields |
|---|---|---|---|
| AgentService.thrift | `TTabletSchema` | 28, 29, 30 | `binlog_tso_idx`,
`binlog_lsn_idx`, `binlog_op_idx` |
| olap_file.proto | `TabletSchemaPB` | 41, 42, 43 |
`binlog_tso_col_idx`, `binlog_lsn_col_idx`, `binlog_op_col_idx` |
| olap_file.proto | `TabletSchemaCloudPB` | 42, 43, 44 | same |
`TabletSchema::to_schema_pb`, `TabletMeta` and the cloud `pb_convert`
stop writing and copying them.
3. #69238 retired the row binlog fields that earlier reworks had
deleted, with `reserved` in protobuf and a `// N: deprecated <name>`
comment in thrift. For the same reason, a separate commit removes those
markers. A `reserved` line or a comment generates no code, so nothing
else changes:
| File | Message | Ids | Old fields |
|---|---|---|---|
| olap_file.proto | `TabletMetaPB` | 42, 43, 44 | `row_binlog_schema`,
`row_binlog_rs_metas`, `row_binlog_schema_hash` |
| olap_file.proto | `TabletMetaCloudPB` | 43, 44 |
`row_binlog_rs_metas`, `row_binlog_schema_hash` |
| olap_file.proto | `TabletSchemaPB` / `TabletSchemaCloudPB` | 37, 38 /
38, 39 | `binlog_timestamp_col_idx`, `binlog_lsn_col_idx` |
| olap_file.proto | `DeleteBitmapPB` | 5 | `is_binlog_delvec` |
| AgentService.thrift | `TCloneReq` / `TSnapshotRequest` | 14 / 15 |
`copy_type` |
| BackendService.thrift | `TTabletStat` | 12, 13 | `binlog_size`,
`binlog_file_num` |
| MasterService.thrift | `TTabletInfo` | 26, 27 | `binlog_size`,
`binlog_file_num` |
| PaloInternalService.thrift | `TTabletWithPartition` | 3 |
`binlog_tablet_id` |
4. The BE unit tests stop setting them. A new
`TabletSchemaTest.test_binlog_col_idx_from_pb_column_names` checks the
positions:
- right after `init_from_pb`;
- after a round trip through the PB;
- on a schema without these columns, where they stay -1.
**Compatibility.**
- **Releases.** None of the ids deleted or un-reserved here exists on
branch-3.1, branch-4.0, branch-4.1 or branch-4.2 (checked with `git
show` on each branch). No release writes or sends them, so an upgrade
from a release never meets them. This PR reuses no id.
- **Meta written by an older master build** still loads. Protobuf keeps
these ids as unknown fields, thrift skips them, and the positions are
derived from the column names anyway.
- **An older master BE that reads meta written by this one** gets -1 for
the positions. In tablet meta schemas that is what it already reads for
every row binlog tablet created since #65810. The rowsets written here
are the difference: their schemas used to carry the positions. Of the
older code that reads these positions, only one place can see a rowset
loaded back from its PB: the publish-phase conflict rewrite of a
merge-on-write load with row binlog, when a local-mode BE restarts with
that load committed but not yet published (`BaseTablet` reads
`binlog_lsn_col_idx()` of the attached row binlog rowset). Before
rolling a local-mode cluster back to an older master build, let pending
loads publish.
- **The ids are no longer reserved.** A later change may put a new field
on one of them. A cluster that ran a master build keeps them in its
persisted meta, for example `TabletSchemaPB` 41-43
(`TabletSchemaCloudPB` 42-44) in every tablet and rowset schema it
wrote, -1 for ordinary tables. Such a cluster has to rebuild its tables
before it upgrades to a build that reuses one of these ids with another
meaning. Releases are not affected.
- With the positions derived, `is_tso_enabled()` is now also true on a
row binlog tablet's own schema. Two callers read it from the tablet
schema, and neither changes behavior:
- `CloudTablet::try_make_committed_rs_visible` returned without doing
anything for a row binlog tablet before, and it still does. Row binlog
tablets are not merge-on-write and never put rowsets into the committed
rowset manager.
- The cloud delete task never reaches a row binlog tablet, because
predicate DELETE is rejected on these tables.
---
be/src/cloud/pb_convert.cpp | 36 -------------------
be/src/storage/tablet/tablet_meta.cpp | 9 -----
be/src/storage/tablet/tablet_schema.cpp | 10 +++---
.../storage/row_binlog_vmerge_compaction_test.cpp | 1 -
.../storage/rowset/segment_flusher_format_test.cpp | 7 ----
be/test/storage/tablet/tablet_schema_test.cpp | 41 ++++++++++++++++++++++
.../storage/transform/row_binlog_derive_test.cpp | 18 ----------
be/test/testutil/creators.h | 3 --
gensrc/proto/olap_file.proto | 30 ----------------
gensrc/thrift/AgentService.thrift | 5 ---
gensrc/thrift/BackendService.thrift | 2 --
gensrc/thrift/MasterService.thrift | 2 --
gensrc/thrift/PaloInternalService.thrift | 1 -
13 files changed, 45 insertions(+), 120 deletions(-)
diff --git a/be/src/cloud/pb_convert.cpp b/be/src/cloud/pb_convert.cpp
index 2ff1e83d80e..7bb5fede003 100644
--- a/be/src/cloud/pb_convert.cpp
+++ b/be/src/cloud/pb_convert.cpp
@@ -511,15 +511,6 @@ void doris_tablet_schema_to_cloud(TabletSchemaCloudPB*
out, const TabletSchemaPB
if (in.has_row_lsn_col_idx()) {
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
}
- if (in.has_binlog_tso_col_idx()) {
- out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
- }
- if (in.has_binlog_lsn_col_idx()) {
- out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
- }
- if (in.has_binlog_op_col_idx()) {
- out->set_binlog_op_col_idx(in.binlog_op_col_idx());
- }
if (in.has___split_schema()) {
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -575,15 +566,6 @@ void doris_tablet_schema_to_cloud(TabletSchemaCloudPB*
out, TabletSchemaPB&& in)
if (in.has_row_lsn_col_idx()) {
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
}
- if (in.has_binlog_tso_col_idx()) {
- out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
- }
- if (in.has_binlog_lsn_col_idx()) {
- out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
- }
- if (in.has_binlog_op_col_idx()) {
- out->set_binlog_op_col_idx(in.binlog_op_col_idx());
- }
if (in.has___split_schema()) {
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -652,15 +634,6 @@ void cloud_tablet_schema_to_doris(TabletSchemaPB* out,
const TabletSchemaCloudPB
if (in.has_row_lsn_col_idx()) {
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
}
- if (in.has_binlog_tso_col_idx()) {
- out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
- }
- if (in.has_binlog_lsn_col_idx()) {
- out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
- }
- if (in.has_binlog_op_col_idx()) {
- out->set_binlog_op_col_idx(in.binlog_op_col_idx());
- }
if (in.has___split_schema()) {
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -717,15 +690,6 @@ void cloud_tablet_schema_to_doris(TabletSchemaPB* out,
TabletSchemaCloudPB&& in)
if (in.has_row_lsn_col_idx()) {
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
}
- if (in.has_binlog_tso_col_idx()) {
- out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
- }
- if (in.has_binlog_lsn_col_idx()) {
- out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
- }
- if (in.has_binlog_op_col_idx()) {
- out->set_binlog_op_col_idx(in.binlog_op_col_idx());
- }
if (in.has___split_schema()) {
out->mutable___split_schema()->CopyFrom(in.__split_schema());
diff --git a/be/src/storage/tablet/tablet_meta.cpp
b/be/src/storage/tablet/tablet_meta.cpp
index 5c72b0ad97a..f39f7daca97 100644
--- a/be/src/storage/tablet/tablet_meta.cpp
+++ b/be/src/storage/tablet/tablet_meta.cpp
@@ -403,15 +403,6 @@ void TabletMeta::init_schema_from_thrift(const
TTabletSchema& tablet_schema,
if (tablet_schema.__isset.row_lsn_col_idx) {
tablet_schema_pb->set_row_lsn_col_idx(tablet_schema.row_lsn_col_idx);
}
- if (tablet_schema.__isset.binlog_tso_idx) {
- tablet_schema_pb->set_binlog_tso_col_idx(tablet_schema.binlog_tso_idx);
- }
- if (tablet_schema.__isset.binlog_lsn_idx) {
- tablet_schema_pb->set_binlog_lsn_col_idx(tablet_schema.binlog_lsn_idx);
- }
- if (tablet_schema.__isset.binlog_op_idx) {
- tablet_schema_pb->set_binlog_op_col_idx(tablet_schema.binlog_op_idx);
- }
auto p_seq_map = tablet_schema_pb->mutable_seq_map(); // ColumnGroupsPB
for (auto& it : tablet_schema.seq_map) { // std::vector<
::doris::TColumnGroup>
uint32_t key = it.sequence_column;
diff --git a/be/src/storage/tablet/tablet_schema.cpp
b/be/src/storage/tablet/tablet_schema.cpp
index c34af2526a5..5fceaa50594 100644
--- a/be/src/storage/tablet/tablet_schema.cpp
+++ b/be/src/storage/tablet/tablet_schema.cpp
@@ -1124,9 +1124,10 @@ void TabletSchema::init_from_pb(const TabletSchemaPB&
schema, bool ignore_extrac
_skip_bitmap_col_idx = schema.skip_bitmap_col_idx();
_commit_tso_col_idx = schema.commit_tso_col_idx();
_row_lsn_col_idx = schema.row_lsn_col_idx();
- _binlog_tso_col_idx = schema.binlog_tso_col_idx();
- _binlog_lsn_col_idx = schema.binlog_lsn_col_idx();
- _binlog_op_col_idx = schema.binlog_op_col_idx();
+ // The binlog column cids are not persisted; derive them from the column
names.
+ _binlog_tso_col_idx = field_index(BINLOG_TSO_COL);
+ _binlog_lsn_col_idx = field_index(BINLOG_LSN_COL);
+ _binlog_op_col_idx = field_index(BINLOG_OP_COL);
_sort_type = schema.sort_type();
_sort_col_num = schema.sort_col_num();
_compression_type = schema.compression_type();
@@ -1463,9 +1464,6 @@ void TabletSchema::to_schema_pb(TabletSchemaPB*
tablet_schema_pb) const {
tablet_schema_pb->set_skip_bitmap_col_idx(_skip_bitmap_col_idx);
tablet_schema_pb->set_commit_tso_col_idx(_commit_tso_col_idx);
tablet_schema_pb->set_row_lsn_col_idx(_row_lsn_col_idx);
- tablet_schema_pb->set_binlog_tso_col_idx(_binlog_tso_col_idx);
- tablet_schema_pb->set_binlog_lsn_col_idx(_binlog_lsn_col_idx);
- tablet_schema_pb->set_binlog_op_col_idx(_binlog_op_col_idx);
tablet_schema_pb->set_inverted_index_storage_format(_inverted_index_storage_format);
tablet_schema_pb->mutable_row_store_column_unique_ids()->Assign(
_row_store_column_unique_ids.begin(),
_row_store_column_unique_ids.end());
diff --git a/be/test/storage/row_binlog_vmerge_compaction_test.cpp
b/be/test/storage/row_binlog_vmerge_compaction_test.cpp
index a1cb340d733..04bb9e951b6 100644
--- a/be/test/storage/row_binlog_vmerge_compaction_test.cpp
+++ b/be/test/storage/row_binlog_vmerge_compaction_test.cpp
@@ -136,7 +136,6 @@ protected:
tso->set_length(8);
tso->set_is_nullable(false);
tso->set_visible(false);
- pb.set_binlog_tso_col_idx(2);
auto schema = std::make_shared<TabletSchema>();
schema->init_from_pb(pb);
diff --git a/be/test/storage/rowset/segment_flusher_format_test.cpp
b/be/test/storage/rowset/segment_flusher_format_test.cpp
index b2a92ce9cd5..6b5dd448610 100644
--- a/be/test/storage/rowset/segment_flusher_format_test.cpp
+++ b/be/test/storage/rowset/segment_flusher_format_test.cpp
@@ -937,13 +937,10 @@ ComplexRowBinlogSchemas
create_complex_row_binlog_schemas() {
}
auto* tso = add_column(&row_binlog_pb, 4, BINLOG_TSO_COL, kKeyTypes[4],
false, true);
tso->set_visible(false);
- row_binlog_pb.set_binlog_tso_col_idx(4);
auto* lsn = add_column(&row_binlog_pb, 5, BINLOG_LSN_COL, kKeyTypes[4],
false, false);
lsn->set_visible(false);
- row_binlog_pb.set_binlog_lsn_col_idx(5);
auto* op = add_column(&row_binlog_pb, 6, BINLOG_OP_COL, kKeyTypes[4],
false, false);
op->set_visible(false);
- row_binlog_pb.set_binlog_op_col_idx(6);
row_binlog_pb.set_next_column_unique_id(7);
auto source = std::make_shared<TabletSchema>();
@@ -3166,11 +3163,7 @@ protected:
binlog_request.tablet_schema.columns.insert(
binlog_request.tablet_schema.columns.begin() +
binlog_tso_idx,
std::move(before_v1));
- binlog_tso_idx += 2;
}
- binlog_request.tablet_schema.__set_binlog_tso_idx(binlog_tso_idx);
- binlog_request.tablet_schema.__set_binlog_lsn_idx(binlog_tso_idx + 1);
- binlog_request.tablet_schema.__set_binlog_op_idx(binlog_tso_idx + 2);
RuntimeProfile profile("SegmentFlusherTransformFormatTest");
EXPECT_TRUE(_engine->create_tablet(request, &profile).ok());
diff --git a/be/test/storage/tablet/tablet_schema_test.cpp
b/be/test/storage/tablet/tablet_schema_test.cpp
index 34ad97e6305..9a135370418 100644
--- a/be/test/storage/tablet/tablet_schema_test.cpp
+++ b/be/test/storage/tablet/tablet_schema_test.cpp
@@ -32,6 +32,7 @@
#include "storage/schema.h"
#include "storage/tablet/tablet_schema_helper.h"
#include "storage/tablet_info.h"
+#include "storage/utils.h"
#include "util/json/path_in_data.h"
namespace doris {
@@ -70,6 +71,46 @@ TEST_F(TabletSchemaTest,
test_commit_tso_col_idx_from_current_index_schema) {
EXPECT_EQ(1, current.commit_tso_col_idx());
}
+TEST_F(TabletSchemaTest, test_binlog_col_idx_from_pb_column_names) {
+ TabletSchemaPB schema_pb;
+ schema_pb.set_keys_type(DUP_KEYS);
+ for (const std::string& name :
+ {std::string("k1"), BINLOG_TSO_COL, BINLOG_LSN_COL, BINLOG_OP_COL}) {
+ ColumnPB* column = schema_pb.add_column();
+ column->set_unique_id(schema_pb.column_size() - 1);
+ column->set_name(name);
+ column->set_type("BIGINT");
+ column->set_is_key(name == "k1");
+ column->set_length(8);
+ column->set_aggregation("NONE");
+ }
+
+ // The binlog column cids are not stored in the PB, so they must come from
the column names
+ // both for a new schema and after a round trip through the PB.
+ TabletSchema schema;
+ schema.init_from_pb(schema_pb);
+ EXPECT_EQ(1, schema.binlog_tso_col_idx());
+ EXPECT_EQ(2, schema.binlog_lsn_col_idx());
+ EXPECT_EQ(3, schema.binlog_op_col_idx());
+
+ TabletSchemaPB round_trip_pb;
+ schema.to_schema_pb(&round_trip_pb);
+ TabletSchema round_trip;
+ round_trip.init_from_pb(round_trip_pb);
+ EXPECT_EQ(1, round_trip.binlog_tso_col_idx());
+ EXPECT_EQ(2, round_trip.binlog_lsn_col_idx());
+ EXPECT_EQ(3, round_trip.binlog_op_col_idx());
+ EXPECT_TRUE(schema == round_trip);
+
+ TabletSchemaPB data_schema_pb;
+ data_schema_pb.add_column()->CopyFrom(schema_pb.column(0));
+ TabletSchema data_schema;
+ data_schema.init_from_pb(data_schema_pb);
+ EXPECT_EQ(-1, data_schema.binlog_tso_col_idx());
+ EXPECT_EQ(-1, data_schema.binlog_lsn_col_idx());
+ EXPECT_EQ(-1, data_schema.binlog_op_col_idx());
+}
+
TEST_F(TabletSchemaTest, test_tablet_column_init_from_pb) {
ColumnPB column_pb;
column_pb.set_unique_id(1001);
diff --git a/be/test/storage/transform/row_binlog_derive_test.cpp
b/be/test/storage/transform/row_binlog_derive_test.cpp
index 07fc267d211..512bf53a558 100644
--- a/be/test/storage/transform/row_binlog_derive_test.cpp
+++ b/be/test/storage/transform/row_binlog_derive_test.cpp
@@ -120,11 +120,6 @@ protected:
add_col(5, BINLOG_TSO_COL, "BIGINT", false, true);
add_col(6, BINLOG_LSN_COL, "BIGINT", false, false);
add_col(7, BINLOG_OP_COL, "BIGINT", false, false);
- // init_from_pb reads this straight from the PB (it is not inferred
from
- // the column name).
[k1,v1,v2,__BEFORE__v1__,__BEFORE__v2__,TSO,LSN,OP]
- pb.set_binlog_tso_col_idx(5);
- pb.set_binlog_lsn_col_idx(6);
- pb.set_binlog_op_col_idx(7);
auto schema = std::make_shared<TabletSchema>();
schema->init_from_pb(pb);
@@ -167,9 +162,6 @@ protected:
add_col(1, BINLOG_TSO_COL, "BIGINT", false, true);
add_col(2, BINLOG_LSN_COL, "BIGINT", false, false);
add_col(3, BINLOG_OP_COL, "BIGINT", false, false);
- pb.set_binlog_tso_col_idx(1);
- pb.set_binlog_lsn_col_idx(2);
- pb.set_binlog_op_col_idx(3);
auto schema = std::make_shared<TabletSchema>();
schema->init_from_pb(pb);
@@ -283,9 +275,6 @@ protected:
add_col(2, BINLOG_TSO_COL, "BIGINT", false, true);
add_col(3, BINLOG_LSN_COL, "BIGINT", false, false);
add_col(4, BINLOG_OP_COL, "BIGINT", false, false);
- pb.set_binlog_tso_col_idx(2);
- pb.set_binlog_lsn_col_idx(3);
- pb.set_binlog_op_col_idx(4);
auto schema = std::make_shared<TabletSchema>();
schema->init_from_pb(pb);
@@ -485,9 +474,6 @@ TEST_F(RowBinlogDeriveTest,
PlainMapsHiddenKeysAndSkipsHiddenNonKeys) {
add_system_column(10, BINLOG_TSO_COL, true);
add_system_column(11, BINLOG_LSN_COL, false);
add_system_column(12, BINLOG_OP_COL, false);
- binlog_pb.set_binlog_tso_col_idx(4);
- binlog_pb.set_binlog_lsn_col_idx(5);
- binlog_pb.set_binlog_op_col_idx(6);
auto binlog_schema = std::make_shared<TabletSchema>();
binlog_schema->init_from_pb(binlog_pb);
@@ -684,10 +670,6 @@ TEST_F(RowBinlogDeriveTest,
MowHiddenKeyColumnIsPartOfTheProbeKey) {
add_col(&binlog_pb, 4, BINLOG_TSO_COL, "BIGINT", false, false, true);
add_col(&binlog_pb, 5, BINLOG_LSN_COL, "BIGINT", false, false, false);
add_col(&binlog_pb, 6, BINLOG_OP_COL, "BIGINT", false, false, false);
- // init_from_pb reads these straight from the PB, not from the column names
- binlog_pb.set_binlog_tso_col_idx(4);
- binlog_pb.set_binlog_lsn_col_idx(5);
- binlog_pb.set_binlog_op_col_idx(6);
auto binlog_schema = std::make_shared<TabletSchema>();
binlog_schema->init_from_pb(binlog_pb);
ASSERT_EQ(binlog_schema->binlog_lsn_col_idx(), 5);
diff --git a/be/test/testutil/creators.h b/be/test/testutil/creators.h
index 6756a62dfb2..2fe8e0d84bc 100644
--- a/be/test/testutil/creators.h
+++ b/be/test/testutil/creators.h
@@ -223,13 +223,10 @@ inline TTabletSchema
create_row_binlog_tablet_schema(const TTabletSchema& base_s
row_binlog_schema.columns.push_back(create_tablet_column(
{BINLOG_TSO_COL, TPrimitiveType::BIGINT, false, true,
TAggregationType::NONE, true}));
- row_binlog_schema.__set_binlog_tso_idx(row_binlog_schema.columns.size() -
1);
row_binlog_schema.columns.push_back(create_tablet_column(
{BINLOG_LSN_COL, TPrimitiveType::BIGINT, false, true,
TAggregationType::NONE}));
- row_binlog_schema.__set_binlog_lsn_idx(row_binlog_schema.columns.size() -
1);
row_binlog_schema.columns.push_back(create_tablet_column(
{BINLOG_OP_COL, TPrimitiveType::BIGINT, false, true,
TAggregationType::NONE}));
- row_binlog_schema.__set_binlog_op_idx(row_binlog_schema.columns.size() -
1);
return row_binlog_schema;
}
diff --git a/gensrc/proto/olap_file.proto b/gensrc/proto/olap_file.proto
index 4e74d04683e..a32c1feeb9e 100644
--- a/gensrc/proto/olap_file.proto
+++ b/gensrc/proto/olap_file.proto
@@ -572,24 +572,12 @@ message TabletSchemaPB {
// storage_format; the fallback path lives in TabletSchema::init_from_pb.
optional TabletStorageFormatPB storage_format = 36;
- reserved 37; // deprecated binlog_timestamp_col_idx
- reserved 38; // deprecated binlog_lsn_col_idx
-
// cid of the time-travel commit tso column (__DORIS_COMMIT_TSO_COL__).
optional int32 commit_tso_col_idx = 39 [default = -1];
// cid of the base-table row lsn column (__DORIS_ROW_LSN_COL__).
optional int32 row_lsn_col_idx = 40 [default = -1];
- // cid of the binlog tso column (__DORIS_BINLOG_TSO__).
- optional int32 binlog_tso_col_idx = 41 [default = -1];
-
- // cid of the binlog lsn column (__DORIS_BINLOG_LSN__).
- optional int32 binlog_lsn_col_idx = 42 [default = -1];
-
- // cid of the binlog op column (__DORIS_BINLOG_OP__).
- optional int32 binlog_op_col_idx = 43 [default = -1];
-
optional SplitSchemaPB __split_schema = 1000; // A special field, DO NOT
change it.
}
@@ -637,24 +625,12 @@ message TabletSchemaCloudPB {
// Persisted TStorageFormat (V2=2, V3=3); see
TabletSchemaPB::storage_format.
optional TabletStorageFormatPB storage_format = 37;
- reserved 38; // deprecated binlog_timestamp_col_idx
- reserved 39; // deprecated binlog_lsn_col_idx
-
// cid of the time-travel commit tso column (__DORIS_COMMIT_TSO_COL__).
optional int32 commit_tso_col_idx = 40 [default = -1];
// cid of the base-table row lsn column (__DORIS_ROW_LSN_COL__).
optional int32 row_lsn_col_idx = 41 [default = -1];
- // cid of the binlog tso column (__DORIS_BINLOG_TSO__).
- optional int32 binlog_tso_col_idx = 42 [default = -1];
-
- // cid of the binlog lsn column (__DORIS_BINLOG_LSN__).
- optional int32 binlog_lsn_col_idx = 43 [default = -1];
-
- // cid of the binlog op column (__DORIS_BINLOG_OP__).
- optional int32 binlog_op_col_idx = 44 [default = -1];
-
optional bool is_dynamic_schema = 100 [default=false];
// FIXME(gavin): deprecate and remove in the future
@@ -769,9 +745,6 @@ message TabletMetaPB {
optional TabletRolePB tablet_role = 36 [default = TABLET_ROLE_DATA];
// Used by a base tablet to find its paired row-binlog tablet.
optional int64 binlog_tablet_id = 37;
- reserved 42; // deprecated row_binlog_schema
- reserved 43; // deprecated row_binlog_rs_metas
- reserved 44; // deprecated row_binlog_schema_hash
// For cloud
optional int64 index_id = 1000;
@@ -836,8 +809,6 @@ message TabletMetaCloudPB {
optional TabletRolePB tablet_role = 41 [default = TABLET_ROLE_DATA];
// Used by a base tablet to find its paired row-binlog tablet.
optional int64 binlog_tablet_id = 42;
- reserved 43; // deprecated row_binlog_rs_metas
- reserved 44; // deprecated row_binlog_schema_hash
// Use for selectdb-cloud
optional string table_name = 101;
@@ -857,7 +828,6 @@ message DeleteBitmapPB {
repeated int64 versions = 3;
// Serialized roaring bitmaps indexed with {rowset_id, segment_id, version}
repeated bytes segment_delete_bitmaps = 4;
- reserved 5; // deprecated is_binlog_delvec
}
// Local-mode only binlog metadata.
diff --git a/gensrc/thrift/AgentService.thrift
b/gensrc/thrift/AgentService.thrift
index a0cfa3e6eb6..e1ac111f550 100644
--- a/gensrc/thrift/AgentService.thrift
+++ b/gensrc/thrift/AgentService.thrift
@@ -54,9 +54,6 @@ struct TTabletSchema {
25: optional list<Types.TColumnGroup> seq_map
26: optional i32 commit_tso_col_idx = -1
27: optional i32 row_lsn_col_idx = -1
- 28: optional i32 binlog_tso_idx = -1
- 29: optional i32 binlog_lsn_idx = -1
- 30: optional i32 binlog_op_idx = -1
}
// this enum stands for different storage format in src_backends
@@ -387,7 +384,6 @@ struct TCloneReq {
11: optional Types.TReplicaId replica_id = 0
12: optional i64 partition_id
13: optional i64 table_id = -1
- // 14: deprecated copy_type
}
struct TCompactionReq {
@@ -467,7 +463,6 @@ struct TSnapshotRequest {
12: optional Types.TVersion end_version
13: optional bool is_copy_binlog
14: optional Types.TTabletId ref_tablet_id
- // 15: deprecated copy_type
}
struct TReleaseSnapshotRequest {
diff --git a/gensrc/thrift/BackendService.thrift
b/gensrc/thrift/BackendService.thrift
index cb7c54291e2..19d80620702 100644
--- a/gensrc/thrift/BackendService.thrift
+++ b/gensrc/thrift/BackendService.thrift
@@ -40,8 +40,6 @@ struct TTabletStat {
9: optional i64 local_segment_size = 0 // .dat
10: optional i64 remote_index_size = 0 // .idx
11: optional i64 remote_segment_size = 0 // .dat
- // 12: deprecated binlog_size
- // 13: deprecated binlog_file_num
}
struct TTabletStatResult {
diff --git a/gensrc/thrift/MasterService.thrift
b/gensrc/thrift/MasterService.thrift
index 43526916b0d..da65873eb6e 100644
--- a/gensrc/thrift/MasterService.thrift
+++ b/gensrc/thrift/MasterService.thrift
@@ -51,8 +51,6 @@ struct TTabletInfo {
23: optional i64 local_segment_size = 0 // .dat
24: optional i64 remote_index_size = 0 // .idx
25: optional i64 remote_segment_size = 0 // .dat
- // 26: deprecated binlog_size
- // 27: deprecated binlog_file_num
// For cloud
1000: optional bool is_persistent
diff --git a/gensrc/thrift/PaloInternalService.thrift
b/gensrc/thrift/PaloInternalService.thrift
index 4dc63f53a91..7bd2cb43d50 100644
--- a/gensrc/thrift/PaloInternalService.thrift
+++ b/gensrc/thrift/PaloInternalService.thrift
@@ -678,7 +678,6 @@ struct TFoldConstantParams {
struct TTabletWithPartition {
1: required i64 partition_id
2: required i64 tablet_id
- // 3: deprecated binlog_tablet_id
}
struct TFetchDataResult {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
