kevinjqliu commented on code in PR #3924: URL: https://github.com/apache/iceberg-python/pull/3924#discussion_r3974063180
########## dev/hive/Dockerfile: ########## @@ -13,30 +13,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM apache/hive:4.0.0 - -# Dependency versions - changing these invalidates the JAR download layer -ARG HADOOP_VERSION=3.3.6 -ARG AWS_SDK_BUNDLE=1.12.753 -ARG MAVEN_MIRROR=https://repo1.maven.org/maven2 +FROM apache/hive:4.2.1 USER root -# Install curl (separate layer - rarely changes) -RUN apt-get update -qq && \ - apt-get -qq -y install --no-install-recommends curl && \ - apt-get clean && \ - rm -rf /var/lib/apt/lists/* - -# Download JARs with retry logic (slow layer - only changes when versions change) -RUN curl -fsSL --retry 3 --retry-delay 5 \ - -o /opt/hive/lib/hadoop-aws-${HADOOP_VERSION}.jar \ - "${MAVEN_MIRROR}/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar" && \ - curl -fsSL --retry 3 --retry-delay 5 \ - -o /opt/hive/lib/aws-java-sdk-bundle-${AWS_SDK_BUNDLE}.jar \ - "${MAVEN_MIRROR}/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_BUNDLE}/aws-java-sdk-bundle-${AWS_SDK_BUNDLE}.jar" +# Link the hadoop-aws and AWS SDK jars that the image ships into the metastore classpath +RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-*.jar /opt/hive/lib/ && \ + ln -s /opt/hadoop/share/hadoop/tools/lib/bundle-*.jar /opt/hive/lib/ Review Comment: nice! it ships with the image now and we can skip downloading ########## pyiceberg/catalog/hive.py: ########## @@ -379,9 +382,21 @@ def _create_hive_table(self, open_client: Client, hive_table: HiveTable) -> None except AlreadyExistsException as e: raise TableAlreadyExistsError(f"Table {hive_table.dbName}.{hive_table.tableName} already exists") from e + def _fetch_hive_table(self, open_client: Client, database_name: str, table_name: str) -> HiveTable: + # Hive 4.0.1 removed get_table, and Hive 2 does not have get_table_req + if self._hive2_compatible: + return open_client.get_table(dbname=database_name, tbl_name=table_name) + return open_client.get_table_req(GetTableRequest(dbName=database_name, tblName=table_name)).table Review Comment: nit: now that we're using hive 4 in the integration tests, i think the hive2 path is untested not an issue for this pr, just calling it out ########## dev/spark/Dockerfile: ########## @@ -23,6 +23,7 @@ ARG ICEBERG_VERSION=1.11.0 ARG ICEBERG_SPARK_RUNTIME_VERSION=4.0_2.13 ARG HADOOP_VERSION=3.4.1 ARG AWS_SDK_VERSION=2.24.6 +ARG HIVE_METASTORE_VERSION=4.0.1 Review Comment: according to my agent heres the compatibility matrix: Client jar Compiled for 4.0.1 Java 8 4.1.0 Java 17 4.2.1 Java 21 Spark 4.0.1 image runs Java 17, so either `4.0.1` or `4.1.0` are ok 👍 ########## dev/hive/Dockerfile: ########## @@ -13,30 +13,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM apache/hive:4.0.0 - -# Dependency versions - changing these invalidates the JAR download layer -ARG HADOOP_VERSION=3.3.6 -ARG AWS_SDK_BUNDLE=1.12.753 -ARG MAVEN_MIRROR=https://repo1.maven.org/maven2 +FROM apache/hive:4.2.1 USER root -# Install curl (separate layer - rarely changes) -RUN apt-get update -qq && \ - apt-get -qq -y install --no-install-recommends curl && \ - apt-get clean && \ - rm -rf /var/lib/apt/lists/* - -# Download JARs with retry logic (slow layer - only changes when versions change) -RUN curl -fsSL --retry 3 --retry-delay 5 \ - -o /opt/hive/lib/hadoop-aws-${HADOOP_VERSION}.jar \ - "${MAVEN_MIRROR}/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar" && \ - curl -fsSL --retry 3 --retry-delay 5 \ - -o /opt/hive/lib/aws-java-sdk-bundle-${AWS_SDK_BUNDLE}.jar \ - "${MAVEN_MIRROR}/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_BUNDLE}/aws-java-sdk-bundle-${AWS_SDK_BUNDLE}.jar" +# Link the hadoop-aws and AWS SDK jars that the image ships into the metastore classpath +RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-*.jar /opt/hive/lib/ && \ + ln -s /opt/hadoop/share/hadoop/tools/lib/bundle-*.jar /opt/hive/lib/ -# Copy configuration last (changes more frequently than JARs) -COPY core-site.xml /opt/hadoop/etc/hadoop/core-site.xml +# The entrypoint links this directory into the Hive config directory, over its own core-site.xml +ENV HIVE_CUSTOM_CONF_DIR=/opt/hive/custom-conf +COPY core-site.xml ${HIVE_CUSTOM_CONF_DIR}/core-site.xml Review Comment: 👍 sets `HIVE_CUSTOM_CONF_DIR` env var so the file gets picked up ########## dev/spark/Dockerfile: ########## @@ -23,6 +23,7 @@ ARG ICEBERG_VERSION=1.11.0 ARG ICEBERG_SPARK_RUNTIME_VERSION=4.0_2.13 ARG HADOOP_VERSION=3.4.1 ARG AWS_SDK_VERSION=2.24.6 +ARG HIVE_METASTORE_VERSION=4.0.1 Review Comment: so it looks like `4.2.1` requires java 21, so we're using an older version that still speaks the same request-based protocol -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
