Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1764033379 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -57,6 +64,14 @@ class S3InputStream extends SeekableInputStream implements RangeRea

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1764030252 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -195,14 +230,20 @@ private void openStream() throws IOException { } } - p

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763890714 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java: ## @@ -0,0 +1,194 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763885839 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java: ## @@ -0,0 +1,194 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

danielcweeks commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763770630 ## core/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1762163236 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -178,18 +183,23 @@ private void positionStream() throws IOException { } priv

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

SandeepSinghGahir commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2353901444 > > > > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2334992007 > > > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

ookumuso commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2334900818 > > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way we can se

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

jackye1995 commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-236419 > I believe https://github.com/apache/iceberg/pull/8221 aimed at addressing both input and output streams. Is there an existing PR for output stream retries? @edgarRd based on

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

jackye1995 commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-234653 The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions: #11052, is there a way we can see how we coul

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

edgarRd commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2321827281 @amogh-jahagirdar Any reason why not handling retries in `S3OutputStream` as well? We've internally seen issues with `UncheckedIOExceptions` during `putObject` for manifest objects. I b

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

nastra commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1726971204 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -92,13 +93,13 @@ public void seek(long newPos) { public int read() throws IOException {

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

nastra commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1726966943 ## gradle/libs.versions.toml: ## @@ -37,6 +37,7 @@ delta-standalone = "3.1.0" delta-spark = "3.2.0" esotericsoftware-kryo = "4.0.3" errorprone-annotations = "2.27.0

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725991357 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -139,7 +140,11 @@ private InputStream readRange(String range) { S3RequestUtil

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725989680 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725989680 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725982688 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1716246358 ## api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

SandeepSinghGahir commented on PR #10433: URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2282666300 When will this merged? I'm getting this issue while reading iceberg tables in glue. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

singhpk234 commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625032460 ## api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

danielcweeks commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625018098 ## api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625017201 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -139,7 +140,11 @@ private InputStream readRange(String range) { S3RequestUtil

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

singhpk234 commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625011756 ## api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

danielcweeks commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624953864 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -139,7 +140,11 @@ private InputStream readRange(String range) { S3RequestUtil.con

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624930096 ## gradle/libs.versions.toml: ## @@ -37,6 +37,7 @@ delta-standalone = "3.1.0" delta-spark = "3.2.0" esotericsoftware-kryo = "4.0.3" errorprone-annotations

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624927489 ## gradle/libs.versions.toml: ## @@ -95,7 +96,7 @@ antlr-antlr4 = { module = "org.antlr:antlr4", version.ref = "antlr" } antlr-runtime = { module = "org.an

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624925629 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -139,7 +140,11 @@ private InputStream readRange(String range) { S3RequestUtil

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624919847 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624919847 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar commented on code in PR #10433: URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624920900 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java: ## @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]

amogh-jahagirdar opened a new pull request, #10433: URL: https://github.com/apache/iceberg/pull/10433 This is an alternative approach to https://github.com/apache/iceberg/pull/4912/files and https://github.com/apache/iceberg/pull/8221/files#diff-0b632866a3b10fac55c442b08178ec0ac72b3b6008782