Hey Sanjeev,
Allright. Thank you once more. This is clear.
However, this poses an issue then. If during the two years, disk drives
develop bad blocks but do not necessarily fail to the point that they
cannot be mounted, that checksum would have changed since those
filesystem blocks can no longer be read. However, from an HDFS
perspective, since no checks are done regularly, that is not known. So
HDFS still reports that the file is fine, in other words, no missing
blocks. For example, if a disk is going bad, but those files are not
read for two years, the system won't know that there is a problem. Even
when removing a data node temporarily and re-adding the datanode, HDFS
isn't checking because that HDFS file isn't read.
So let's assume this scenario. Data nodes *dn01* to *dn10* exist. Each
data node has 10 x 10TB drives.
And let's assume that there is one large file on those drives and it's
replicated to factor of X3.
If during the two years the file isn't read, and 10 of those drives
develop bad blocks or other underlying hardware issues, then it is
possible that HDFS will still report everything fine, even with a
replication factor of 3. Because with 10 disks failing, it's possible a
block or sector has failed under each of the 3 copies of the data. But
HDFS would NOT know since nothing triggered a read of that HDFS file.
Based on everything below, then corruption is very much possible even
with a replication of factor X3. A this point the file is unreadable
but HDFS still reports no missing blocks.
Similarly, if once I take a data node out, I adjust one of the files on
the data disks, HDFS will not know and still report everything fine.
That is until someone read's the file.
Sounds like this is a very real possibility.
Thx,
TK
On 10/21/2020 10:26 AM, संजीव (Sanjeev Tripurari) wrote:
Hi Tom
Therefore, if I write a file to HDFS but access it two years later,
then the checksum will be computed only twice, at the beginning of the
two years and again at the end when a client connects? Correct? As
long as no process ever accesses the file between now and two years
from now, the checksum is never redone and compared to the two year
old checksum in the fsimage?
yes, Exactly unless data is read checksum is not verified. (when data
is written and when the data is read),
if checksum is mismatched, there is no way to correct it, you will
have to re-write that file.
When datanode is added back in, there is no real read operation on
the files themselves. The datanode just reports the blocks but
doesn't really read the blocks that are there to re-verify the files
and ensure consistency?
yes, Exactly, datanode maintains list of files and their blocks, which
it reports, along with total disk size and used size.
Namenode only has list of blocks, unless datanodes is connected it
wont know where the blocks are stored.
Regards
-Sanjeev
On Wed, 21 Oct 2020 at 18:31, TomK <[email protected]
<mailto:[email protected]>> wrote:
Hey Sanjeev,
Thank you very much again. This confirms my suspision.
Therefore, if I write a file to HDFS but access it two years
later, then the checksum will be computed only twice, at the
beginning of the two years and again at the end when a client
connects? Correct? As long as no process ever accesses the file
between now and two years from now, the checksum is never redone
and compared to the two year old checksum in the fsimage?
When datanode is added back in, there is no real read operation
on the files themselves. The datanode just reports the blocks but
doesn't really read the blocks that are there to re-verify the
files and ensure consistency?
Thx,
TK
On 10/21/2020 12:38 AM, संजीव (Sanjeev Tripurari) wrote:
Hi Tom,
Every datanode sends heartbeat to namenode, on its list of blocks
it has.
When a datanode which is disconnected for a while, after
connecting will send heartbeat to namenode, with list of blocks
it has (till then namenode will have under-replicated blocks).
As soon as the datanode is connected to namenode, it will clear
under-replicatred blocks.
*When a client connects to read or write a file, it will run
checksum to validate the file.*
There is no independent process running to do checksum, as it
will be heavy process on each node.
Regards
-Sanjeev
On Wed, 21 Oct 2020 at 00:18, Tom <[email protected]
<mailto:[email protected]>> wrote:
Thank you. That part I understand and am Ok with it.
What I would like to know next is when again the CRC32C
checksum is ran and checked against the fsimage that the
block file has not changed or become corrupted?
For example, if I take a datanode out, and within 15 minutes,
plug it back in, does HDF rerun the CRC 32C on all data disks
on that node to make sure blocks are ok?
Cheers,
TK
Sent from my iPhone
On Oct 20, 2020, at 1:39 PM, संजीव (Sanjeev Tripurari)
<[email protected]
<mailto:[email protected]>> wrote:
its done as sson as a file is stored on disk..
Sanjeev
On Tuesday, 20 October 2020, TomK <[email protected]
<mailto:[email protected]>> wrote:
Thanks again.
At what points is the checksum validated (checked) after
that? For example, is it done on a daily basis or is it
done only when the file is accessed?
Thx,
TK
On 10/20/2020 10:18 AM, संजीव (Sanjeev Tripurari) wrote:
As soon as the file is written first time checksum is
calculated and updated in fsimage (first in edit logs),
and same is replicated other replicas.
On Tue, 20 Oct 2020 at 19:15, TomK <[email protected]
<mailto:[email protected]>> wrote:
Hi Sanjeev,
Thank you. It does help.
At what points is the checksum calculated?
Thx,
TK
On 10/20/2020 3:03 AM, संजीव (Sanjeev Tripurari) wrote:
For Missing blocks and corrupted blocks, do check
if all the datanode services are up, non of the
disks where hdfs data is stored is accessible and
have no issues, hosts are reachable from namenode,
If you are able to re-generate the data and write
its great, otherwise hadoop cannot correct itself.
Could you please elaborate on this? Does it mean I
have to continuously access a file for HDFS to be
able to detect corrupt blocks and correct itself?
*"Does HDFS check that the data node is up, data
disk is mounted, path to
the file exists and file can be read?"*
-- yes, only after it fails it will say missing
blocks.
*Or does it also do a filesystem check on that
data disk as well as
perhaps a checksum to ensure block integrity?*
-- yes, every file cheksum is maintained and cross
checked, if it fails it will say corrupted blocks.
hope this helps.
-Sanjeev
*
*
On Tue, 20 Oct 2020 at 09:52, TomK
<[email protected] <mailto:[email protected]>>
wrote:
Hello,
HDFS Missing Blocks / Corrupt Blocks Logic:
What are the specific
checks done to determine a block is bad and
needs to be replicated?
Does HDFS check that the data node is up, data
disk is mounted, path to
the file exists and file can be read?
Or does it also do a filesystem check on that
data disk as well as
perhaps a checksum to ensure block integrity?
I've googled on this quite a bit. I don't see
the exact answer I'm
looking for. I would like to know exactly
what happens during file
integrity verification that then constitutes
missing blocks or corrupt
blocks in the reports.
--
Thank You,
TK.
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]
<mailto:[email protected]>
For additional commands, e-mail:
[email protected]
<mailto:[email protected]>
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
