This is an automated email from the ASF dual-hosted git repository.
kevinjqliu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-go.git
The following commit(s) were added to refs/heads/main by this push:
new a2b91aee2 ci: replace MinIO with RustFS in integration tests (#2055)
a2b91aee2 is described below
commit a2b91aee208daeffc3d955d5bb54a549a9149df6
Author: Sreesh Maheshwar <[email protected]>
AuthorDate: Thu Sep 24 23:43:39 2026 +0100
ci: replace MinIO with RustFS in integration tests (#2055)
* ci: replace MinIO with RustFS in integration tests
Use RustFS 1.0.0 and AWS CLI bucket setup for the shared S3 test fixture.
Generated-by: Codex
Signed-off-by: Sreesh Maheshwar <[email protected]>
* ci: apply the shared S3 configuration to Spark 3.5
Copy spark-defaults.conf into the Spark 3.5 image so it uses object-store
instead of the inherited MinIO endpoint.
Generated-by: Codex
Signed-off-by: Sreesh Maheshwar <[email protected]>
* ci: preserve the Spark 3.5 image catalog configuration
Override only the inherited S3 endpoint, retaining the demo catalog and
history-server settings.
Generated-by: Codex
Signed-off-by: Sreesh Maheshwar <[email protected]>
* ci: make RustFS fixture startup reliable and observable
Expose server logs, retry bucket initialization, and rebuild the Spark
fixture so local images pick up its endpoint configuration.
Generated-by: Codex
Signed-off-by: Sreesh Maheshwar <[email protected]>
* ci: empty the warehouse bucket on setup
Clear any existing objects in the warehouse bucket during create-bucket
so re-running integration setup starts from a clean state, as before.
Also update a leftover object store reference in the CLI docs.
Co-authored-by: Copilot App <[email protected]>
* ci: share spark-defaults.conf with the Spark 3.5 image
Copy the repo's spark-defaults.conf into the Spark 3.5 image, as the
Spark 4 image already does, so both use one catalog configuration
instead of rewriting the base image's config.
Co-authored-by: Copilot App <[email protected]>
---------
Signed-off-by: Sreesh Maheshwar <[email protected]>
Co-authored-by: Kevin Liu <[email protected]>
Co-authored-by: Copilot App <[email protected]>
---
.github/workflows/go-integration.yml | 2 +-
CONTRIBUTING.md | 2 +-
Makefile | 4 +-
README.md | 2 +-
catalog/README.md | 8 +--
...ation_test.go => hadoop_s3_integration_test.go} | 36 ++++++------
internal/recipe/Dockerfile | 1 +
internal/recipe/docker-compose.yml | 68 ++++++++++++++--------
internal/recipe/provision.py | 2 +-
internal/recipe/spark-defaults.conf | 2 +-
io/gocloud/s3/s3_integration_test.go | 4 +-
website/src/cli.md | 2 +-
12 files changed, 76 insertions(+), 57 deletions(-)
diff --git a/.github/workflows/go-integration.yml
b/.github/workflows/go-integration.yml
index 818f5867a..5e77bd377 100644
--- a/.github/workflows/go-integration.yml
+++ b/.github/workflows/go-integration.yml
@@ -70,7 +70,7 @@ jobs:
- name: Setup environment variables
run: |
- echo "AWS_S3_ENDPOINT=http://$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minio):9000" >>
$GITHUB_ENV
+ echo "AWS_S3_ENDPOINT=http://$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' object-store):9000"
>> $GITHUB_ENV
echo "SPARK_CONTAINER_ID=$(docker ps -qf 'name=spark-iceberg')" >>
$GITHUB_ENV
echo "DOCKER_API_VERSION=$(docker version -f
'{{.Server.APIVersion}}')" >> $GITHUB_ENV
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index bad108f58..c81b48983 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -60,6 +60,6 @@ make test-race # runs all unit tests with race detector
Integration tests require Docker and are gated behind a build tag:
```bash
-docker compose -f internal/recipe/docker-compose.yml up -d rest minio mc --wait
+docker compose -f internal/recipe/docker-compose.yml up -d rest object-store
create-bucket --wait
go test -tags integration ./...
```
diff --git a/Makefile b/Makefile
index c19a48dfb..741393ac7 100644
--- a/Makefile
+++ b/Makefile
@@ -39,7 +39,7 @@ lint-install:
integration-setup:
mkdir -p /tmp/iceberg-hadoop-warehouse
- docker compose -f internal/recipe/docker-compose.yml up -d --wait
+ docker compose -f internal/recipe/docker-compose.yml up -d --build
--wait
docker compose -f internal/recipe/docker-compose.yml exec -T
spark-iceberg ipython ./provision.py
integration-setup-spark4:
@@ -55,7 +55,7 @@ integration-logs:
docker compose -f internal/recipe/docker-compose.yml logs
integration-env:
- @echo "export AWS_S3_ENDPOINT=http://$$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minio):9000"
+ @echo "export AWS_S3_ENDPOINT=http://$$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' object-store):9000"
@echo "export AWS_REGION=us-east-1"
@echo "export SPARK_CONTAINER_ID=$$(docker ps -qf 'name=spark-iceberg')"
@echo "export DOCKER_API_VERSION=$$(docker version -f
'{{.Server.APIVersion}}')"
diff --git a/README.md b/README.md
index 23888222e..23414a28b 100644
--- a/README.md
+++ b/README.md
@@ -70,7 +70,7 @@ make lint-install
2. Export the required environment variables:
```shell
- export AWS_S3_ENDPOINT=http://$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minio):9000
+ export AWS_S3_ENDPOINT=http://$(docker inspect -f
'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' object-store):9000
export AWS_REGION=us-east-1
export SPARK_CONTAINER_ID=$(docker ps -qf 'name=spark-iceberg')
export DOCKER_API_VERSION=$(docker version -f '{{.Server.APIVersion}}')
diff --git a/catalog/README.md b/catalog/README.md
index 37912d27d..265d66715 100644
--- a/catalog/README.md
+++ b/catalog/README.md
@@ -25,11 +25,11 @@ in the `cmd/iceberg` folder.
### REST Catalog
To test the REST catalog implementation, we have a docker configuration
-for a Minio container and tabulario/iceberg-rest container.
+for RustFS and the Apache Iceberg REST fixture.
-You can spin up the local catalog by going to the `dev/` folder and running
-`docker-compose up`. You can then follow the steps of the Iceberg
[Quickstart](https://iceberg.apache.org/spark-quickstart/#creating-a-table)
-tutorial, which we've summarized below.
+You can spin up the local catalog from the repository root by running
+`make integration-setup`. You can then follow the steps of the Iceberg
+[Quickstart](https://iceberg.apache.org/spark-quickstart/#creating-a-table)
tutorial, which we've summarized below.
#### Setup your Iceberg catalog
diff --git a/catalog/hadoop/hadoop_minio_integration_test.go
b/catalog/hadoop/hadoop_s3_integration_test.go
similarity index 84%
rename from catalog/hadoop/hadoop_minio_integration_test.go
rename to catalog/hadoop/hadoop_s3_integration_test.go
index 7d69e1141..184282799 100644
--- a/catalog/hadoop/hadoop_minio_integration_test.go
+++ b/catalog/hadoop/hadoop_s3_integration_test.go
@@ -36,7 +36,7 @@ import (
"github.com/stretchr/testify/suite"
)
-type HadoopMinIOIntegrationSuite struct {
+type HadoopS3IntegrationSuite struct {
suite.Suite
ctx context.Context
@@ -45,9 +45,9 @@ type HadoopMinIOIntegrationSuite struct {
warehouse string
}
-func (s *HadoopMinIOIntegrationSuite) SetupTest() {
+func (s *HadoopS3IntegrationSuite) SetupTest() {
s.ctx = context.Background()
- s.warehouse = fmt.Sprintf("s3a://warehouse/hadoop-minio/%s",
uuid.NewString())
+ s.warehouse = fmt.Sprintf("s3a://warehouse/hadoop-s3/%s",
uuid.NewString())
s.props = iceberg.Properties{
"allow-unsafe-commits": "true",
icebergio.S3Region: "local",
@@ -57,12 +57,12 @@ func (s *HadoopMinIOIntegrationSuite) SetupTest() {
// from `make integration-env` or recipe.Start.
}
- cat, err := hadoop.NewCatalog("hadoop_minio_test", s.warehouse, s.props)
+ cat, err := hadoop.NewCatalog("hadoop_s3_test", s.warehouse, s.props)
s.Require().NoError(err)
s.cat = cat
}
-func (s *HadoopMinIOIntegrationSuite) testSchema() *iceberg.Schema {
+func (s *HadoopS3IntegrationSuite) testSchema() *iceberg.Schema {
return iceberg.NewSchema(
1,
iceberg.NestedField{ID: 1, Name: "id", Type:
iceberg.PrimitiveTypes.Int64, Required: true},
@@ -70,7 +70,7 @@ func (s *HadoopMinIOIntegrationSuite) testSchema()
*iceberg.Schema {
)
}
-func (s *HadoopMinIOIntegrationSuite) TearDownTest() {
+func (s *HadoopS3IntegrationSuite) TearDownTest() {
fs, err := icebergio.LoadFS(s.ctx, s.props, s.warehouse)
if err != nil {
return
@@ -81,9 +81,9 @@ func (s *HadoopMinIOIntegrationSuite) TearDownTest() {
}
}
-func (s *HadoopMinIOIntegrationSuite) TestNamespaceTableRoundTrip() {
- ns := table.Identifier{"minio_ns"}
- ident := table.Identifier{"minio_ns", "tbl"}
+func (s *HadoopS3IntegrationSuite) TestNamespaceTableRoundTrip() {
+ ns := table.Identifier{"s3_ns"}
+ ident := table.Identifier{"s3_ns", "tbl"}
s.Require().NoError(s.cat.CreateNamespace(s.ctx, ns, nil))
@@ -94,7 +94,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestNamespaceTableRoundTrip() {
tbl, err := s.cat.CreateTable(s.ctx, ident, s.testSchema())
s.Require().NoError(err)
s.NotNil(tbl)
- s.True(strings.HasPrefix(tbl.MetadataLocation(),
s.warehouse+"/minio_ns/tbl/metadata/v1.metadata.json"))
+ s.True(strings.HasPrefix(tbl.MetadataLocation(),
s.warehouse+"/s3_ns/tbl/metadata/v1.metadata.json"))
loaded, err := s.cat.LoadTable(s.ctx, ident)
s.Require().NoError(err)
@@ -115,7 +115,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestNamespaceTableRoundTrip() {
s.False(tableExists)
}
-func (s *HadoopMinIOIntegrationSuite) TestListTablesEmptyNamespace() {
+func (s *HadoopS3IntegrationSuite) TestListTablesEmptyNamespace() {
ns := table.Identifier{"empty_ns"}
s.Require().NoError(s.cat.CreateNamespace(s.ctx, ns, nil))
@@ -127,7 +127,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestListTablesEmptyNamespace() {
s.Empty(tables)
}
-func (s *HadoopMinIOIntegrationSuite) TestListTablesMissingNamespace() {
+func (s *HadoopS3IntegrationSuite) TestListTablesMissingNamespace() {
for _, err := range s.cat.ListTables(s.ctx,
table.Identifier{"missing_ns"}) {
s.ErrorIs(err, catalog.ErrNoSuchNamespace)
@@ -135,7 +135,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestListTablesMissingNamespace() {
}
}
-func (s *HadoopMinIOIntegrationSuite)
TestListTablesOnlyDirectNamespaceTables() {
+func (s *HadoopS3IntegrationSuite) TestListTablesOnlyDirectNamespaceTables() {
parent := table.Identifier{"parent_ns"}
child := table.Identifier{"parent_ns", "child_ns"}
directTable := table.Identifier{"parent_ns", "direct_tbl"}
@@ -156,7 +156,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestListTablesOnlyDirectNamespaceTables()
s.Equal([]table.Identifier{directTable}, tables)
}
-func (s *HadoopMinIOIntegrationSuite)
TestDropTableRemovesTableFromListingAndLoad() {
+func (s *HadoopS3IntegrationSuite)
TestDropTableRemovesTableFromListingAndLoad() {
ns := table.Identifier{"drop_ns"}
ident := table.Identifier{"drop_ns", "tbl"}
@@ -177,7 +177,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestDropTableRemovesTableFromListingAndLoa
s.Empty(tables)
}
-func (s *HadoopMinIOIntegrationSuite) TestPurgeTableRemovesTableRoot() {
+func (s *HadoopS3IntegrationSuite) TestPurgeTableRemovesTableRoot() {
ns := table.Identifier{"purge_ns"}
ident := table.Identifier{"purge_ns", "tbl"}
@@ -209,7 +209,7 @@ func (s *HadoopMinIOIntegrationSuite)
TestPurgeTableRemovesTableRoot() {
s.False(exists)
}
-func (s *HadoopMinIOIntegrationSuite) TestDropNamespaceAfterDropTable() {
+func (s *HadoopS3IntegrationSuite) TestDropNamespaceAfterDropTable() {
ns := table.Identifier{"drop_table_then_ns"}
ident := table.Identifier{"drop_table_then_ns", "tbl"}
@@ -225,6 +225,6 @@ func (s *HadoopMinIOIntegrationSuite)
TestDropNamespaceAfterDropTable() {
s.Require().NoError(s.cat.DropNamespace(s.ctx, ns))
}
-func TestHadoopIntegrationMinIO(t *testing.T) {
- suite.Run(t, new(HadoopMinIOIntegrationSuite))
+func TestHadoopIntegrationS3(t *testing.T) {
+ suite.Run(t, new(HadoopS3IntegrationSuite))
}
diff --git a/internal/recipe/Dockerfile b/internal/recipe/Dockerfile
index 073a276b1..e2ef77bc6 100644
--- a/internal/recipe/Dockerfile
+++ b/internal/recipe/Dockerfile
@@ -19,6 +19,7 @@ RUN pip3 install -q ipython
RUN pip3 install pyiceberg[s3fs,hive]
RUN pip3 install pyarrow
+COPY spark-defaults.conf ${SPARK_HOME}/conf/spark-defaults.conf
COPY provision.py .
COPY validation.py .
COPY hadoop_validation.py .
diff --git a/internal/recipe/docker-compose.yml
b/internal/recipe/docker-compose.yml
index b5328f1ab..be0ac0b6c 100644
--- a/internal/recipe/docker-compose.yml
+++ b/internal/recipe/docker-compose.yml
@@ -25,7 +25,7 @@ services:
iceberg_net:
depends_on:
- rest
- - minio
+ - object-store
volumes:
- ./warehouse:/home/iceberg/warehouse
- ./notebooks:/home/iceberg/notebooks/notebooks
@@ -46,6 +46,9 @@ services:
retries: 12
start_period: 30s
rest:
+ depends_on:
+ create-bucket:
+ condition: service_completed_successfully
image: apache/iceberg-rest-fixture:1.10.1
container_name: iceberg-rest
networks:
@@ -58,8 +61,11 @@ services:
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_ENDPOINT=http://object-store:9000
rest-scan-planning:
+ depends_on:
+ create-bucket:
+ condition: service_completed_successfully
# 1.10.1 predates the fixture's scan-planning implementation. Pin the
# multi-platform image built from apache/iceberg@b0df3ca so the Java wire
# compatibility tests are reproducible.
@@ -78,26 +84,32 @@ services:
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_ENDPOINT=http://object-store:9000
- CATALOG_INCLUDE__CREDENTIALS=true
# Use a non-S3 credential so vending can be asserted without changing the
- # static MinIO credentials used by the fixture's backing FileIO.
+ # static S3 credentials used by the fixture's backing FileIO.
- CATALOG_GCS_OAUTH2_TOKEN=scan-planning-integration-token
- minio:
- image: quay.io/minio/minio
- container_name: minio
+ object-store:
+ image: rustfs/rustfs:1.0.0
+ container_name: object-store
environment:
- - MINIO_ROOT_USER=admin
- - MINIO_ROOT_PASSWORD=password
- - MINIO_DOMAIN=minio
+ - RUSTFS_ACCESS_KEY=admin
+ - RUSTFS_SECRET_KEY=password
+ - RUSTFS_SERVER_DOMAINS=object-store
+ - RUSTFS_OBS_LOG_STDOUT_ENABLED=true
networks:
iceberg_net:
aliases:
- - warehouse.minio
+ - warehouse.object-store
ports:
- 9001:9001
- 9000:9000
- command: ["server", "/data", "--console-address", ":9001"]
+ command: ["/data"]
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:9000/health/ready";]
+ interval: 5s
+ timeout: 5s
+ retries: 12
azurite:
image: mcr.microsoft.com/azure-storage/azurite
container_name: azurite
@@ -105,25 +117,31 @@ services:
ports:
- 11000:11000
command: ["azurite-blob", "--loose", "--blobHost", "0.0.0.0"
,"--blobPort", "11000", "--skipApiVersionCheck"]
- mc:
+ create-bucket:
depends_on:
- - minio
- image: quay.io/minio/mc
- container_name: mc
+ object-store:
+ condition: service_healthy
+ image: amazon/aws-cli:2.36.44
networks:
iceberg_net:
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- - AWS_REGION=us-east-1
- entrypoint: >
- /bin/sh -c "
- until (/usr/bin/mc alias set minio http://minio:9000 admin password) do
echo '...waiting...' && sleep 1; done;
- /usr/bin/mc rm -r --force minio/warehouse;
- /usr/bin/mc mb minio/warehouse;
- /usr/bin/mc policy set public minio/warehouse;
- tail -f /dev/null
- "
+ - AWS_DEFAULT_REGION=us-east-1
+ - AWS_ENDPOINT_URL=http://object-store:9000
+ entrypoint: /bin/sh
+ command:
+ - -ec
+ - |
+ for attempt in 1 2 3 4 5; do
+ if { aws s3api head-bucket --bucket warehouse >/dev/null 2>&1 || aws
s3api create-bucket --bucket warehouse; } \
+ && aws s3 rm s3://warehouse --recursive --quiet; then
+ exit 0
+ fi
+ sleep 2
+ done
+ echo "Failed to create warehouse bucket." >&2
+ exit 1
clickhouse:
image: clickhouse/clickhouse-server
container_name: clickhouse
diff --git a/internal/recipe/provision.py b/internal/recipe/provision.py
index 8a8227163..53f9047ec 100644
--- a/internal/recipe/provision.py
+++ b/internal/recipe/provision.py
@@ -33,7 +33,7 @@ catalogs = {
**{
"type": "rest",
"uri": "http://rest:8181";,
- "s3.endpoint": "http://minio:9000";,
+ "s3.endpoint": "http://object-store:9000";,
"s3.access-key-id": "admin",
"s3.secret-access-key": "password",
},
diff --git a/internal/recipe/spark-defaults.conf
b/internal/recipe/spark-defaults.conf
index 5051bd5be..b5507408a 100644
--- a/internal/recipe/spark-defaults.conf
+++ b/internal/recipe/spark-defaults.conf
@@ -18,7 +18,7 @@ spark.sql.catalog.spark_catalog
org.apache.iceberg.spark.Sp
spark.sql.catalog.spark_catalog.type rest
spark.sql.catalog.spark_catalog.uri http://rest:8181
spark.sql.catalog.spark_catalog.io-impl
org.apache.iceberg.aws.s3.S3FileIO
-spark.sql.catalog.spark_catalog.s3.endpoint http://minio:9000
+spark.sql.catalog.spark_catalog.s3.endpoint http://object-store:9000
spark.sql.catalog.spark_catalog.s3.path-style-access true
spark.sql.catalog.spark_catalog.warehouse s3://warehouse
spark.sql.defaultCatalog spark_catalog
diff --git a/io/gocloud/s3/s3_integration_test.go
b/io/gocloud/s3/s3_integration_test.go
index 3bae04c11..0f02f6c26 100644
--- a/io/gocloud/s3/s3_integration_test.go
+++ b/io/gocloud/s3/s3_integration_test.go
@@ -34,7 +34,7 @@ import (
"github.com/uptrace/bun/driver/sqliteshim"
)
-func TestMinioWarehouse(t *testing.T) {
+func TestS3Warehouse(t *testing.T) {
_, err := recipe.Start(t)
require.NoError(t, err)
@@ -66,7 +66,7 @@ func TestMinioWarehouse(t *testing.T) {
require.NotNil(t, tbl)
}
-func TestMinioWarehouseNoLocation(t *testing.T) {
+func TestS3WarehouseNoLocation(t *testing.T) {
_, err := recipe.Start(t)
require.NoError(t, err)
diff --git a/website/src/cli.md b/website/src/cli.md
index f799e7705..440b370d0 100644
--- a/website/src/cli.md
+++ b/website/src/cli.md
@@ -104,7 +104,7 @@ All commands accept `--output text` (default,
human-readable) or `--output json`
Note: only the identity transform is supported for `--partition-spec` at this
moment.
```shell
-# Create a simple table with REST catalog and Minio
+# Create a simple table with REST catalog and object store
./iceberg create table default.table-1 \
--properties write.format.default=parquet \
--partition-spec foo \
