[
https://issues.apache.org/jira/browse/HBASE-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-30542:
------------------------------------
Description:
Incremental backup after a table is deleted and recreated restores data of the
deleted table
h3. Problem
The backup system does not notice when a table is deleted and recreated with
the same name. The next incremental backup treats the new table as a
continuation of the old one, so restoring that backup image restores data of
the deleted table.
Found while working on HBASE-29905, by code inspection. I have not reproduced
this with a test yet.
h3. Scenario
# A full backup F1 includes table T in backup root R. This adds T to the
incremental backup set of R ({{{}incrbackupset:{}}} row) and stores the WAL
timestamps of T in its {{trslm:}} row.
# Rows are written to T.
# T is deleted. {{BackupMasterObserver#postDeleteTable}} only removes bulk
load rows, so T stays in the incremental backup set of R and its {{trslm:}} row
stays as well.
# T is recreated with the same name and the same column families, and new rows
are written to it.
# An incremental backup I1 runs on R. Incremental backups cover the whole
incremental backup set, and T exists again, so I1 includes T. The column family
check in {{IncrementalTableBackupClient}} only compares column family names, so
it passes.
# T is restored from I1.
h3. Expected
Either the restored T only contains the rows of the recreated table, or I1
refuses to back up T until a new full backup of T is taken.
h3. Actual
Restoring I1 starts from the snapshot of the deleted table in F1 and then
applies the incremental data. Deleting a table produces no delete markers, so
the restored T contains:
* all rows of the deleted table from F1,
* the edits made to the deleted table between F1 and its deletion, which I1
reads from the WALs because the {{trslm:}} timestamps of T still point to F1,
* the rows of the recreated table.
If the recreated table has different column families, I1 fails with
{{ColumnFamilyMismatchException}} instead, so that variant does not corrupt
data.
{{postTruncateTable}} has the same gap: truncating also removes data without
delete markers. I have not verified the truncate case.
h3. Possible fix
Treat a deleted table as no longer backed up. In
{{{}BackupMasterObserver#postDeleteTable{}}}, remove the table from the
incremental backup set of every backup root and delete its {{trslm:}} rows. The
next incremental backup then skips the recreated table, and a request that
names it explicitly fails with "haven't gone through full backup". The existing
backups of the deleted table stay restorable: restore reads the backup images
and manifests, and point-in-time restore reads the backup history and the
continuous backup table set, so neither uses these rows. As a side effect, this
also removes the stale {{trslm:}} rows that HBASE-29905 works around in
BackupLogCleaner.
was:
*{*}Summary:{*}* Incremental backup after a table is deleted and recreated
restores data of the deleted table
h3. Problem
The backup system does not notice when a table is deleted and recreated with
the same name. The next incremental backup treats the new table as a
continuation of the old one, so restoring that backup image restores data of
the deleted table.
Found while working on HBASE-29905, by code inspection. I have not reproduced
this with a test yet.
h3. Scenario
# A full backup F1 includes table T in backup root R. This adds T to the
incremental backup set of R ({{{}incrbackupset:{}}} row) and stores the WAL
timestamps of T in its {{trslm:}} row.
# Rows are written to T.
# T is deleted. {{BackupMasterObserver#postDeleteTable}} only removes bulk
load rows, so T stays in the incremental backup set of R and its {{trslm:}} row
stays as well.
# T is recreated with the same name and the same column families, and new rows
are written to it.
# An incremental backup I1 runs on R. Incremental backups cover the whole
incremental backup set, and T exists again, so I1 includes T. The column family
check in {{IncrementalTableBackupClient}} only compares column family names, so
it passes.
# T is restored from I1.
h3. Expected
Either the restored T only contains the rows of the recreated table, or I1
refuses to back up T until a new full backup of T is taken.
h3. Actual
Restoring I1 starts from the snapshot of the deleted table in F1 and then
applies the incremental data. Deleting a table produces no delete markers, so
the restored T contains:
* all rows of the deleted table from F1,
* the edits made to the deleted table between F1 and its deletion, which I1
reads from the WALs because the {{trslm:}} timestamps of T still point to F1,
* the rows of the recreated table.
If the recreated table has different column families, I1 fails with
{{ColumnFamilyMismatchException}} instead, so that variant does not corrupt
data.
{{postTruncateTable}} has the same gap: truncating also removes data without
delete markers. I have not verified the truncate case.
h3. Possible fix
Treat a deleted table as no longer backed up. In
{{{}BackupMasterObserver#postDeleteTable{}}}, remove the table from the
incremental backup set of every backup root and delete its {{trslm:}} rows. The
next incremental backup then skips the recreated table, and a request that
names it explicitly fails with "haven't gone through full backup". The existing
backups of the deleted table stay restorable: restore reads the backup images
and manifests, and point-in-time restore reads the backup history and the
continuous backup table set, so neither uses these rows. As a side effect, this
also removes the stale {{trslm:}} rows that HBASE-29905 works around in
BackupLogCleaner.
> Recreating a table can corrupt incremental backups
> --------------------------------------------------
>
> Key: HBASE-30542
> URL: https://issues.apache.org/jira/browse/HBASE-30542
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
> Reporter: Dieter De Paepe
> Priority: Critical
>
> Incremental backup after a table is deleted and recreated restores data of
> the deleted table
> h3. Problem
> The backup system does not notice when a table is deleted and recreated with
> the same name. The next incremental backup treats the new table as a
> continuation of the old one, so restoring that backup image restores data of
> the deleted table.
> Found while working on HBASE-29905, by code inspection. I have not reproduced
> this with a test yet.
> h3. Scenario
> # A full backup F1 includes table T in backup root R. This adds T to the
> incremental backup set of R ({{{}incrbackupset:{}}} row) and stores the WAL
> timestamps of T in its {{trslm:}} row.
> # Rows are written to T.
> # T is deleted. {{BackupMasterObserver#postDeleteTable}} only removes bulk
> load rows, so T stays in the incremental backup set of R and its {{trslm:}}
> row stays as well.
> # T is recreated with the same name and the same column families, and new
> rows are written to it.
> # An incremental backup I1 runs on R. Incremental backups cover the whole
> incremental backup set, and T exists again, so I1 includes T. The column
> family check in {{IncrementalTableBackupClient}} only compares column family
> names, so it passes.
> # T is restored from I1.
> h3. Expected
> Either the restored T only contains the rows of the recreated table, or I1
> refuses to back up T until a new full backup of T is taken.
> h3. Actual
> Restoring I1 starts from the snapshot of the deleted table in F1 and then
> applies the incremental data. Deleting a table produces no delete markers, so
> the restored T contains:
> * all rows of the deleted table from F1,
> * the edits made to the deleted table between F1 and its deletion, which I1
> reads from the WALs because the {{trslm:}} timestamps of T still point to F1,
> * the rows of the recreated table.
> If the recreated table has different column families, I1 fails with
> {{ColumnFamilyMismatchException}} instead, so that variant does not corrupt
> data.
> {{postTruncateTable}} has the same gap: truncating also removes data without
> delete markers. I have not verified the truncate case.
> h3. Possible fix
> Treat a deleted table as no longer backed up. In
> {{{}BackupMasterObserver#postDeleteTable{}}}, remove the table from the
> incremental backup set of every backup root and delete its {{trslm:}} rows.
> The next incremental backup then skips the recreated table, and a request
> that names it explicitly fails with "haven't gone through full backup". The
> existing backups of the deleted table stay restorable: restore reads the
> backup images and manifests, and point-in-time restore reads the backup
> history and the continuous backup table set, so neither uses these rows. As a
> side effect, this also removes the stale {{trslm:}} rows that HBASE-29905
> works around in BackupLogCleaner.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)