Fokko commented on code in PR #2491: URL: https://github.com/apache/iceberg-python/pull/2491#discussion_r2368727309
########## dev/spark-defaults.conf: ########## @@ -16,20 +16,35 @@ # spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions + +# Configure Iceberg REST catalog spark.sql.catalog.rest org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.rest.type rest spark.sql.catalog.rest.uri http://rest:8181 spark.sql.catalog.rest.io-impl org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.rest.warehouse s3://warehouse/rest/ spark.sql.catalog.rest.s3.endpoint http://minio:9000 +spark.sql.catalog.rest.cache-enabled false + +# Configure Iceberg Hive catalog spark.sql.catalog.hive org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.hive.type hive -spark.sql.catalog.hive.uri http://hive:9083 +spark.sql.catalog.hive.uri thrift://hive:9083 spark.sql.catalog.hive.io-impl org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.hive.warehouse s3://warehouse/hive/ spark.sql.catalog.hive.s3.endpoint http://minio:9000 + +# Configure Spark's default session catalog (spark_catalog) to use Iceberg backed by the Hive Metastore +spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog +spark.sql.catalog.spark_catalog.type hive +spark.sql.catalog.spark_catalog.uri thrift://hive:9083 +spark.hadoop.fs.s3a.endpoint http://minio:9000 +spark.sql.catalogImplementation hive +spark.sql.warehouse.dir s3a://warehouse/hive/ Review Comment: It requires the `SparkSessionCatalog` 👍 ########## dev/Dockerfile: ########## @@ -36,25 +36,38 @@ ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$ RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME} && mkdir -p /home/iceberg/spark-events WORKDIR ${SPARK_HOME} -# Remember to also update `tests/conftest`'s spark setting ENV SPARK_VERSION=3.5.6 -ENV ICEBERG_SPARK_RUNTIME_VERSION=3.5_2.12 -ENV ICEBERG_VERSION=1.9.1 +ENV SCALA_VERSION=2.12 +ENV ICEBERG_SPARK_RUNTIME_VERSION=3.5_${SCALA_VERSION} +ENV ICEBERG_VERSION=1.9.2 ENV PYICEBERG_VERSION=0.10.0 +ENV HADOOP_VERSION=3.3.4 +ENV AWS_SDK_VERSION=1.12.753 Review Comment: Nice, this is much better 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
