[
https://issues.apache.org/jira/browse/THRIFT-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylwester Lachiewicz updated THRIFT-6234:
-----------------------------------------
Description:
Every Linux job in the GitHub Actions workflows ({{build.yml}}, {{cmake.yml}},
{{sca.yml}}, {{make-dist.yml}}) installs packages with {{apt-get}} at apt's
defaults.
The GitHub runner images point the sources at a mirror list
({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then
archive.ubuntu.com, then security.ubuntu.com). apt only moves an item to the
next mirror once it has failed on the current one, and on a transient failure
(timeout, connection refused) it first retries the same mirror
{{Acquire::Retries}} times, which defaults to 3. The http method additionally
reconnects and resends once before it reports a failure. With the default 30
second inactivity timeout a stalled mirror therefore costs 4 attempts x 2 x 30
s plus back-off, about four minutes per item, before apt falls over.
Example: [Build with CMake run
34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542]
on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3 minutes
between single downloads, and hit the 10 minute step timeout before compiling
anything. The same workflow passed on the commits before and after.
Proposal: ship one apt configuration fragment in the repository and copy it
into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:
{code}
Acquire::Retries "0";
Acquire::http::Timeout "20";
Acquire::https::Timeout "20";
DPkg::Lock::Timeout "120";
{code}
Without same-mirror retries and with the shorter timeout a stalled item falls
over to the next mirror after 40 seconds (measured with a local mirror that
accepts connections and never answers: first fetch from the fallback mirror at
40 s on Ubuntu 22.04 and 24.04, versus 60 s with the timeout alone and more
than 300 s with the defaults). The three mirrors in the list provide the
redundancy that the retries were providing before. The lock timeout makes
apt-get wait for a concurrent apt process instead of failing immediately.
was:
Every Linux job in the GitHub Actions workflows ({{build.yml}}, {{cmake.yml}},
{{sca.yml}}, {{make-dist.yml}}) installs packages with {{apt-get}} at apt's
defaults: no retries and a 120 second inactivity timeout per connection.
The GitHub runner images point {{sources.list}} at a mirror list
({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then
archive.ubuntu.com, then security.ubuntu.com). apt only moves to the next
mirror when a fetch fails, so a stalled primary mirror runs the step into its
{{timeout-minutes}} instead of failing over.
Example: [Build with CMake run
34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542]
on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3 minutes
between single downloads, and hit the 10 minute step timeout before compiling
anything. The same workflow passed on the commits before and after.
Proposal: ship one apt configuration fragment in the repository and copy it
into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:
{code}
Acquire::Retries "3";
Acquire::http::Timeout "30";
Acquire::https::Timeout "30";
DPkg::Lock::Timeout "120";
{code}
The shorter timeout lets the mirror fallback engage, the retries cover
transient failures, and the lock timeout waits for unattended-upgrades instead
of failing immediately when it holds the dpkg lock.
> Configure apt retries and timeouts in GitHub Actions workflows
> --------------------------------------------------------------
>
> Key: THRIFT-6234
> URL: https://issues.apache.org/jira/browse/THRIFT-6234
> Project: Thrift
> Issue Type: Improvement
> Components: Build Process
> Reporter: Sylwester Lachiewicz
> Priority: Major
>
> Every Linux job in the GitHub Actions workflows ({{build.yml}},
> {{cmake.yml}}, {{sca.yml}}, {{make-dist.yml}}) installs packages with
> {{apt-get}} at apt's defaults.
> The GitHub runner images point the sources at a mirror list
> ({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then
> archive.ubuntu.com, then security.ubuntu.com). apt only moves an item to the
> next mirror once it has failed on the current one, and on a transient failure
> (timeout, connection refused) it first retries the same mirror
> {{Acquire::Retries}} times, which defaults to 3. The http method additionally
> reconnects and resends once before it reports a failure. With the default 30
> second inactivity timeout a stalled mirror therefore costs 4 attempts x 2 x
> 30 s plus back-off, about four minutes per item, before apt falls over.
> Example: [Build with CMake run
> 34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542]
> on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3
> minutes between single downloads, and hit the 10 minute step timeout before
> compiling anything. The same workflow passed on the commits before and after.
> Proposal: ship one apt configuration fragment in the repository and copy it
> into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:
> {code}
> Acquire::Retries "0";
> Acquire::http::Timeout "20";
> Acquire::https::Timeout "20";
> DPkg::Lock::Timeout "120";
> {code}
> Without same-mirror retries and with the shorter timeout a stalled item falls
> over to the next mirror after 40 seconds (measured with a local mirror that
> accepts connections and never answers: first fetch from the fallback mirror
> at 40 s on Ubuntu 22.04 and 24.04, versus 60 s with the timeout alone and
> more than 300 s with the defaults). The three mirrors in the list provide the
> redundancy that the retries were providing before. The lock timeout makes
> apt-get wait for a concurrent apt process instead of failing immediately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
