[GitHub] [iceberg] JonasJ-ap commented on pull request #6746: AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client

[via GitHub]

JonasJ-ap commented on PR #6746:
URL: https://github.com/apache/iceberg/pull/6746#issuecomment-1423889742

   Sorry for the late update. It took me sometime to construct the EKS 
environment properly.
   ## Test Environment
   AWS EKS: 1.24, Spark 3.1.2
   ## Test spark job / k8s job config:
   ```java
   import org.apache.spark.sql.SparkSession
   
   object GlueApp {
   
     def main(sysArgs: Array[String]) {
       val spark: SparkSession = SparkSession.builder.
         config("spark.master", "local").
         config("spark.sql.extensions", 
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions").
         config("spark.sql.catalog.demo", 
"org.apache.iceberg.spark.SparkCatalog").
         config("spark.sql.catalog.demo.io-impl", 
"org.apache.iceberg.aws.s3.S3FileIO").
         config("spark.sql.catalog.demo.warehouse", 
"s3a://gluetestjonas/warehouse").
         config("spark.sql.catalog.demo.catalog-impl", 
"org.apache.iceberg.aws.glue.GlueCatalog").
         config("spark.sql.catalog.demo.http-client.type", "apache").
         getOrCreate()
   
       val data = spark.range(0, 100)
       data.writeTo("demo.default.table100").createOrReplace()
       spark.sql("SELECT * FROM demo.default.table100").show()
   
       // read using SQL
       // spark.sql("SELECT * FROM demo.reviews.book_reviews").show()
     }
   }
   ```
   
   ```bash
   name: spark
             image: jonasjiang/spark-eks:v3.1.2
             args: [
                 "/bin/sh",
                 "-c",
                 "/opt/spark/bin/spark-submit \
               --master k8s://https://kubernetes.default.svc.cluster.local:443 \
               --deploy-mode cluster \
               --name spark-eks \
               --class GlueApp \
               --jars 
s3a://gluetestjonas/jars/iceberg-spark-runtime-3.1_bug_fix_aws_httpclient_conflict_prefix_map.jar
 \
               --packages 
software.amazon.awssdk:apache-client:2.17.257,software.amazon.awssdk:s3:2.17.257,software.amazon.awssdk:glue:2.17.257,software.amazon.awssdk:kms:2.17.257,software.amazon.awssdk:iam:2.17.257,software.amazon.awssdk:sts:2.17.257,software.amazon.awssdk:dynamodb:2.17.257
 \
               ...
               --conf 
spark.kubernetes.container.image=jonasjiang/spark-eks:v3.1.2 \
               ...
               --conf 
spark.kubernetes.authenticate.driver.serviceAccountName=spark \
               --conf 
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider
 \
               --conf 
spark.kubernetes.authenticate.submission.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 \
               --conf 
spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
 \
               s3a://gluetestjonas/jars/scalar-glue_apache.jar
   ```
   ### Test result:
   Current master branch had the following error:
   ![Screen Shot 2023-02-09 at 03 57 
22](https://user-images.githubusercontent.com/55990951/217773207-827b8866-1b55-4823-8797-b9ea50591305.png)
   This PR was fine:
   ![Screen Shot 2023-02-09 at 04 01 
00](https://user-images.githubusercontent.com/55990951/217773583-e7fa151e-2719-4e6a-a9d9-fee0bf4a29e8.png)
   
   Indicating that we no longer need `urlconnection` when set http-client.type 
to `apache`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org