[
https://issues.apache.org/jira/browse/HBASE-29774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044691#comment-18044691
]
Dieter De Paepe commented on HBASE-29774:
-----------------------------------------
[~hgromer] does this ring a bell for you?
> incremental backup fails on empty WAL files
> -------------------------------------------
>
> Key: HBASE-29774
> URL: https://issues.apache.org/jira/browse/HBASE-29774
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
> Affects Versions: 2.6.4
> Reporter: Jan Van Besien
> Priority: Major
>
> Incremental backup fails during
> {{IncrementalTableBackupClient.convertWALsToHFiles}} when one of these WAL
> files is empty (zero bytes).
> The map tasks fail like this:
> {code:java}
> 2025-12-12 06:22:21,785 INFO [main]
> org.apache.hadoop.hbase.mapreduce.WALInputFormat: Opening
> hdfs://hdfsns/hbase/hbase/oldWALs/hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.regiongroup-0.1764754214409
> for
> hdfs://hdfsns/hbase/hbase/oldWALs/hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.regiongroup-0.1764754214409
> (-9223372036854775808:9223372036854775807) length:0
> 2025-12-12 06:22:21,810 INFO [main]
> org.apache.hadoop.hbase.mapreduce.WALInputFormat: Closing reader
> 2025-12-12 06:22:21,811 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2025-12-12 06:22:21,815 INFO [main] org.apache.hadoop.io.compress.CodecPool:
> Got brand-new compressor [.deflate]
> 2025-12-12 06:22:21,904 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.hbase.regionserver.wal.WALHeaderEOFException: EOF while
> reading PB WAL magic
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.readHeader(AbstractProtobufWALReader.java:221)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.init(AbstractProtobufWALReader.java:147)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createStreamReader(WALFactory.java:360)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createStreamReader(WALFactory.java:481)
> at
> org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.openReader(WALInputFormat.java:162)
> at
> org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.openReader(WALInputFormat.java:204)
> at
> org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.initialize(WALInputFormat.java:197)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:561)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
> at
> java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
> Caused by: java.io.EOFException
> at java.base/java.io.DataInputStream.readFully(DataInputStream.java:210)
> at java.base/java.io.DataInputStream.readFully(DataInputStream.java:179)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.readHeader(AbstractProtobufWALReader.java:219)
> ... 14 more
> {code}
> The file mentioned in the above log snippet is indeed zero bytes.
> The calling code fails like this:
> {code}
> 2025-12-12 06:22:45,140 ERROR
> org.apache.hadoop.hbase.backup.impl.TableBackupClient: Unexpected exception
> in incremental-backup: incremental copy backup_1765519365442WAL Player failed
> java.io.IOException: WAL Player failed
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.walToHFiles(IncrementalTableBackupClient.java:448)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.convertWALsToHFiles(IncrementalTableBackupClient.java:414)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:311)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:594)
> {code}
> The javadoc in {{WALHeaderEOFException}} says "This usually means the WAL
> file just contains nothing and we are safe to skip over it.". So maybe that
> is indeed what needs to happen here?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
