[
https://issues.apache.org/jira/browse/HBASE-30524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144048#comment-18144048
]
Hudson commented on HBASE-30524:
--------------------------------
Results for branch branch-3.0
[build #73 on
builds.a.o|https://ci-hbase.apache.org/job/HBase-Integration-Test/job/branch-3.0/73/]:
(/) *{color:green}+1 overall{color}*
----
details (if available):
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.5 with shaded hadoop
client{color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.3.6 with shaded hadoop
client{color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.0 with shaded hadoop
client{color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
(/) {color:green}+1 client integration test for 3.4.1 with shaded hadoop
client{color}
(/) {color:green}+1 client integration test for 3.4.2 {color}
(/) {color:green}+1 client integration test for 3.4.2 with shaded hadoop
client{color}
(/) {color:green}+1 client integration test for 3.4.3 {color}
(/) {color:green}+1 client integration test for 3.4.3 with shaded hadoop
client{color}
> MOB cleaner archives MOB files still referenced after region split or merge
> ---------------------------------------------------------------------------
>
> Key: HBASE-30524
> URL: https://issues.apache.org/jira/browse/HBASE-30524
> Project: HBase
> Issue Type: Bug
> Components: mob
> Reporter: Xiao Liu
> Assignee: Xiao Liu
> Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.1.0, 3.0.1, 2.6.8, 2.5.17
>
>
> {\{MobFileCleanupUtil}} is run by the master \{{MobFileCleanerChore}}. It
> archives a MOB file when two conditions hold:
> * the region named in the file name suffix no longer exists in the table
> directory;
> * the file is older than \{{hbase.mob.min.age.archive}}.
> It collects the MOB files referenced by the current store files
> (\{{allActiveMobFileName}}), but only logs them. HBASE-26969 replaced the
> reference check with the region check.
> After a split or merge, the store files of the new regions still reference
> the MOB files of the parent regions. The compaction that removes the
> reference files is a system compaction, and a system compaction copies MOB
> references unchanged. Only a user major compaction rewrites the MOB cells,
> for example one run by \{{MobFileCompactionChore}}. Split, merge and the
> parent region GC do not handle MOB files.
> The parent MOB files are archived when all of the following hold:
> * the parent region directory has been GCed;
> * the new regions still reference the parent MOB files;
> * the files are older than \{{hbase.mob.min.age.archive}} (1 hour by default);
> * the cleaner runs before the MOB cells are rewritten.
> With the default settings this is likely: the cleaner runs daily, while the
> MOB compaction chore runs weekly. In I/O optimized mode, MOB files larger
> than \{{hbase.mob.compactions.max.file.size}} are never rewritten, so the new
> regions keep referencing them indefinitely.
> The HFileCleaner deletes the archived files after
> \{{hbase.master.hfilecleaner.ttl}} (5 minutes by default). After that, reads
> of the affected rows fail with \{{DoNotRetryIOException}} caused by
> \{{FileNotFoundException}}. MOB compactions of the new regions fail as well.
> Steps to reproduce, as in the attached test:
> # Write and flush MOB data into a single region table.
> # Split the region, and wait for the daughters to compact away their
> reference files.
> # Run the CatalogJanitor so that it GCs the parent region.
> # Run \{{MobFileCleanupUtil.cleanupObsoleteMobFiles}}, then the HFileCleaner.
> # Scan the table: it fails with the exception above.
> Merging two regions leads to the same result.
> The fix is to archive a MOB file only when it is also not referenced by any
> store file.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
