Tavisca-vinayak-bhadage commented on issue #8155: URL: https://github.com/apache/iceberg/issues/8155#issuecomment-1746581142
This compareWithFileList would be good solution for AWS S3 based iceberg tables also. As we are facing below exception with default remove orphan file implementation : `org.apache.iceberg.exceptions.RuntimeIOException: java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;)` Is there any way to use compareWithFileList parameter from pyspark or spark-sql. As we are having AWS Glue as spark engine which is not supporting java Data Set api. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
