[
https://issues.apache.org/jira/browse/HADOOP-19877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080725#comment-18080725
]
ASF GitHub Bot commented on HADOOP-19877:
-----------------------------------------
pan3793 commented on code in PR #8467:
URL: https://github.com/apache/hadoop/pull/8467#discussion_r3236630788
##########
.github/workflows/tmpl_cloud_aws.yml:
##########
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name: s3a integration
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 17
+ toolchain_branch:
+ required: false
+ type: string
+ description: Branch to use for toolchain image build
+ default: trunk
+ os:
+ required: false
+ type: string
+ description: OS for container to run the build in
+ default: ubuntu_24
+ runner_os:
+ required: false
+ type: string
+ description: OS tag for runner (e.g., Linux, ubuntu-24.04)
+ default: ubuntu_24.04
+
+# Security: Minimal defaults for workflow.
+permissions: {}
+
+concurrency:
+ group: >-
+ cloud-aws
+ ${{ github.workflow }}
+ ${{ github.repository == 'apache/hadoop' && github.run_id || github.ref }}
+ ${{ inputs.java }}
+ ${{ inputs.toolchain_branch }}
+ ${{ inputs.os }}
+ cancel-in-progress: true
+
+env:
+ BUCKET_NAME: hadoop-ci
+
+jobs:
+ precondition:
+ runs-on: ${{ inputs.runner_os }}
+ outputs:
+ build_image_url: ${{ steps.img.outputs.build_image_url }}
+ steps:
+ - uses: actions/checkout@v6
+ with:
+ # Full fetch so build image URL can be computed for any branch
+ fetch-depth: 0
+ - uses: ./.github/actions/build_image_url
+ id: img
+ with:
+ os: ${{ inputs.os }}
+ branch: ${{ inputs.toolchain_branch }}
+ - name: debug base_image_url
+ run: |
+ echo "precondition url: ${{ steps.img.outputs.build_image_url }}"
+
+ build-image:
+ name: Toolchain image (JDK${{ inputs.java }}, ${{ inputs.os }}-${{
inputs.toolchain_branch }})
+ runs-on: ${{ inputs.runner_os }}
+ needs: [ precondition ]
+ permissions:
+ packages: write
+ outputs:
+ uid: ${{ steps.build_img.outputs.uid }}
+ steps:
+ - name: debug build url
+ run: |
+ echo "Build image URL: ${{
needs.precondition.outputs.build_image_url }}"
+ - uses: actions/checkout@v6
+ - uses: ./.github/actions/build_image
Review Comment:
> If I push an update to my S3A PR, there is no reason it should take more
than 10-15 minutes.
Spark did this by adding `dev/is-changed.py -m <module>` test in the
precondition step, and running other jobs conditionally.
> run s3a integration tests in CI
> -------------------------------
>
> Key: HADOOP-19877
> URL: https://issues.apache.org/jira/browse/HADOOP-19877
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Priority: Major
> Labels: pull-request-available
>
> * Get a decent portion of hadoop-aws (s3a) integration tests running in CI.
> * Use localstack (OSS license) or other S3 emulator as a target.
> * Update docs as needed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
