slachiewicz commented on PR #3841: URL: https://github.com/apache/thrift/pull/3841#issuecomment-5634676902
The Azure apt mirror has been stalling intermittently all morning, so the Build workflow runs since 10:30 UTC give a same-conditions comparison of the apt install steps (45 Linux jobs per run) with and without this change. | Build run | median | p90 | worst step | steps over 60 s | failed | |---|---|---|---|---|---| | master 10:35, without | 31 s | 52 s | 622 s (lib-netstd) | 3 | 0 | | master 10:46, without | 33 s | 52 s | 203 s (lib-dart 3.2.0) | 3 | 0 | | this PR run 1, retries 0 | 32 s | 74 s | 134 s (cross-test cpp) | 5 | 1 | | this PR run 2, retries 1 | 30 s | 46 s | 179 s (cross-test nodejs) | 3 | 0 | | THRIFT-4244-doc 12:12, without | 33 s | 53 s | 501 s (cross-test cpp) | 2 | 0 | | THRIFT-6235 12:22, without | 31 s | 55 s | 323 s (lib-python 3.12) | 2 | 0 | Medians are the same everywhere, so the normal path is unchanged. The difference is the tail. Without the change a stalled package is retried on the same mirror four times and nothing is logged: the [cross-test cpp step in the THRIFT-4244-doc run](https://github.com/apache/thrift/actions/runs/34597719237/job/103259428567) waited 311 s and then 155 s on two packages from azure.archive.ubuntu.com, 501 s in total, and the [lib-netstd step on master](https://github.com/apache/thrift/actions/runs/34589946308/job/103233317543) took 622 s. With the change the [worst step](https://github.com/apache/thrift/actions/runs/34597144251/job/103257633897) shows both stalled packages abandoned after the one retry and fetched from the next mirror, 179 s in total. The one failure was run 1 with retries at 0: one package stalled on all three mirror names within 80 s and apt had nothing left to try. Retries 1 covers that at the cost of a slower failover per package (about 80 s instead of 40 s). @Jens-G @fishy could you take a look and decide? Open points from my side are the retry count (1 as pushed, or something else), whether the `DPkg::Lock::Timeout` addition is welcome, and whether a 20 s inactivity timeout is acceptable for the runners. *This comment was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
