[doris] branch branch-2.0 updated: [fix](broker) fix export job failed for that currentStreamOffset may be different with request offset (#23133)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
     new 3e49a38509 [fix](broker) fix export job failed for that 
currentStreamOffset may be different with request offset (#23133)
3e49a38509 is described below

commit 3e49a38509cfe61ddccfd7844385d0507dae3c51
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Fri Aug 18 14:32:36 2023 +0800

    [fix](broker) fix export job failed for that currentStreamOffset may be 
different with request offset (#23133)
    
    Co-authored-by: caiconghui1 <caicongh...@jd.com>when export job encounter 
heavy pressure, the failed export job may see the following message
    current outputstream offset is 423597 not equal to request 421590, cause 
by: null,
    because the broker pwrite operation may retry for timeout, so we just skip 
it instead of throw broker exception.
---
 .../doris/broker/hdfs/FileSystemManager.java       | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git 
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
 
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
index 41b57f2784..d86df86fe0 100644
--- 
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
+++ 
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
@@ -1279,16 +1279,24 @@ public class FileSystemManager {
                         "errors while get file pos from output stream");
             }
             if (currentStreamOffset != offset) {
-                throw new 
BrokerException(TBrokerOperationStatusCode.INVALID_INPUT_OFFSET,
+                // it's ok, it means that last pwrite succeed finally
+                if (currentStreamOffset == offset + data.length) {
+                    logger.warn("invalid offset, but current outputstream 
offset is "
+                        + currentStreamOffset + ", data length is " + 
data.length + ", the sum is equal to request offset "
+                        + offset + ", so just skip it");
+                } else {
+                    throw new 
BrokerException(TBrokerOperationStatusCode.INVALID_INPUT_OFFSET,
                         "current outputstream offset is {} not equal to 
request {}",
                         currentStreamOffset, offset);
-            }
-            try {
-                fsDataOutputStream.write(data);
-            } catch (IOException e) {
-                logger.error("errors while write data to output stream", e);
-                throw new 
BrokerException(TBrokerOperationStatusCode.TARGET_STORAGE_SERVICE_ERROR,
+                }
+            } else {
+                try {
+                    fsDataOutputStream.write(data);
+                } catch (IOException e) {
+                    logger.error("errors while write data to output stream", 
e);
+                    throw new 
BrokerException(TBrokerOperationStatusCode.TARGET_STORAGE_SERVICE_ERROR,
                         e, "errors while write data to output stream");
+                }
             }
         }
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org