[
https://issues.apache.org/jira/browse/HDFS-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18065031#comment-18065031
]
ASF GitHub Bot commented on HDFS-17800:
---------------------------------------
CapMoon commented on PR #7754:
URL: https://github.com/apache/hadoop/pull/7754#issuecomment-4037974751
Great job, after further thinking, when check EC data in
`hasSufficientReplicas()`, we should considered the issue of duplicate indices
in Locations needs, like this:
```
private boolean hasSufficientReplicas(LocatedBlock block,
ErasureCodingPolicy ecPolicy) {
DatanodeInfo[] locations = block.getLocations();
if (locations == null || locations.length == 0) {
return false;
}
// Replicated block
if (!block.isStriped()) {
return true;
}
// Striped (EC) block
LocatedStripedBlock stripedBlock = (LocatedStripedBlock) block;
int requiredDataBlocks = (int) Math.min(ecPolicy.getNumDataUnits(),
(block.getBlockSize() - 1) / ecPolicy.getCellSize() + 1);
byte[] indices = stripedBlock.getBlockIndices();
boolean[] seen = new boolean[ecPolicy.getNumDataUnits() +
ecPolicy.getNumParityUnits()];
int count = 0;
for (byte idx : indices) {
int i = idx & 0xFF;
if (!seen[i]) {
seen[i] = true;
if (++count >= requiredDataBlocks) {
return true;
}
}
}
return false;
}
```
> Observer node can return insufficient blocks for erasure coded files
> --------------------------------------------------------------------
>
> Key: HDFS-17800
> URL: https://issues.apache.org/jira/browse/HDFS-17800
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.4.1
> Reporter: Adam Binford
> Priority: Major
> Labels: pull-request-available
>
> Similar to https://issues.apache.org/jira/browse/HDFS-13924,
> https://issues.apache.org/jira/browse/HDFS-16732, and
> https://issues.apache.org/jira/browse/HDFS-17768, except for erasure coded
> files. A single block may not be enough to fulfill an erasure coded read, as
> you need at least the number of data shards of blocks to read an erasure
> coded file. The checks need to be updated to factor in erasure coded files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
