[
https://issues.apache.org/jira/browse/HDFS-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075941#comment-18075941
]
ASF GitHub Bot commented on HDFS-17800:
---------------------------------------
ZanderXu commented on code in PR #7754:
URL: https://github.com/apache/hadoop/pull/7754#discussion_r3136421721
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##########
@@ -9206,16 +9197,75 @@ private boolean isObserver() {
return haEnabled && haContext != null &&
haContext.getState().getServiceState() == OBSERVER;
}
+ private boolean hasSufficientReplicas(LocatedBlock block,
+ ErasureCodingPolicy ecPolicy) {
+ DatanodeInfo[] locations = block.getLocations();
+ if (locations == null) {
+ return false;
+ }
+
+ if (block.isStriped()) {
+ LocatedStripedBlock stripedBlock = (LocatedStripedBlock) block;
Review Comment:
```
LocatedStripedBlock stripedBlock = (LocatedStripedBlock) block;
// For erasure coded files, require enough data units to reconstruct
// the block, bounded by the number of cells in the block.
long numCells = (block.getBlockSize() - 1) / ecPolicy.getCellSize() +
1;
int minRequiredIndices = (int) Math.min(ecPolicy.getNumDataUnits(),
numCells);
// Units can be over-replicated, so need to account for unique indices
byte[] blockIndices = stripedBlock.getBlockIndices();
boolean[] seenIndices = new boolean[ecPolicy.getNumDataUnits() +
ecPolicy.getNumParityUnits()];
int uniqueIndexCount = 0;
for (byte idx : blockIndices) {
int index = idx & 0xFF;
if (!seenIndices[index]) {
seenIndices[index] = true;
if (++uniqueIndexCount >= minRequiredIndices) {
return true;
}
}
}
return false;
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##########
@@ -9206,16 +9197,75 @@ private boolean isObserver() {
return haEnabled && haContext != null &&
haContext.getState().getServiceState() == OBSERVER;
}
+ private boolean hasSufficientReplicas(LocatedBlock block,
Review Comment:
hasSufficientBlockLocations
> Observer node can return insufficient blocks for erasure coded files
> --------------------------------------------------------------------
>
> Key: HDFS-17800
> URL: https://issues.apache.org/jira/browse/HDFS-17800
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.4.1
> Reporter: Adam Binford
> Priority: Major
> Labels: pull-request-available
>
> Similar to https://issues.apache.org/jira/browse/HDFS-13924,
> https://issues.apache.org/jira/browse/HDFS-16732, and
> https://issues.apache.org/jira/browse/HDFS-17768, except for erasure coded
> files. A single block may not be enough to fulfill an erasure coded read, as
> you need at least the number of data shards of blocks to read an erasure
> coded file. The checks need to be updated to factor in erasure coded files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
