zoobst opened a new issue, #958:
URL: https://github.com/apache/iceberg-go/issues/958
### Feature Request / Improvement
Drafted from a production migration of a several TB table on
AWS Glue + S3, using iceberg-go v0.5.0 as the catalog client and write
path.
The pilot migration wrote 7.94B rows across 11 days, 6,869 Parquet files, and
exercised the full `Transaction.AddFiles` → `AddDataFiles` →
`Transaction.Commit` path against AWS Glue. One incident (a 50-minute
CPU-pinned wedge during `AddFiles`) and one full recovery surfaced the
issue below.
**Problem.** `io/gocloud/s3.go` calls `s3.NewFromConfig(*awscfg, ...)`
without overriding the AWS SDK's HTTP client. The SDK falls back to
`awshttp.NewBuildableClient()` which inherits Go's `http.DefaultTransport`
defaults — including `MaxIdleConnsPerHost = 2`. At any non-trivial
file count, every `ReadAt` opens a fresh TCP+TLS connection because
the connection pool is saturated.
**Evidence.** The wedge goroutine dump showed 15+ idle
`(*persistConn).readLoop` / `writeLoop` goroutines, each backed by its
own `dialConn` invocation — i.e., the runtime had created hundreds of
connections but couldn't reuse them across calls because of the
per-host idle cap.
After installing a tuned `*http.Client` via the existing
`utils.WithAwsConfig(ctx, awsCfg)` plumbing — `MaxIdleConnsPerHost = 256`,
`MaxConnsPerHost = 256`, `IdleConnTimeout = 90s` — per-file footer read
time dropped from ~30s to <1s.
**Proposed change.** In `io/gocloud/s3.go::createS3Bucket`, when the
caller hasn't provided an explicit `aws.Config.HTTPClient`, install a
sensible default for footer-read workloads. Suggested floor:
```go
http.Transport{
MaxIdleConns: 256,
MaxIdleConnsPerHost: 256,
MaxConnsPerHost: 256,
IdleConnTimeout: 90 * time.Second,
// Other fields inherited from DefaultTransport
}
```
Alternatively (or additionally): document the
`utils.WithAwsConfig(ctx, ...)` injection path prominently in the
`io/gocloud` README. Currently it's a discovery-via-source-reading
experience.
**Backwards compatibility.** Defaults-only change. Callers who already
inject an AWS Config keep their current behavior. The defaults purely
relax connection-reuse constraints.
**Expected impact.** Roughly 30× per-file footer-read latency
improvement under the existing serial `AddFiles` path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
