On Thu, Jan 21, 2021 at 03:27:40PM +0000, Witold Baryluk wrote: > Package: apt > Version: 2.1.18 > Followup-For: Bug #973861 > X-Debbugs-Cc: witold.bary...@gmail.com, mir...@init7.net > > I started getting this error recently too. > > To the point that using live-build (that downloads ~6100 package in one > go) is not possible, because some package will fail to download and fail > entire apt install request. > > > This is related to keepalive. > > For example most nginx server has keepalive_requests 100 by default. That > is, after serving 100 request, nginx server will close the connection. > This is because some dynamic memory allocated by nginx is allocated on > arenas that are tied to TCP connection, and they are never freed until > the connection is closed. This is a high performance design, but to > prevent leaks and abuse, the connection must be closed after some number > of requests. > > Most tools (wget), respect the response from the server, close the > existing connection and open a new one. It is easy to see with wget -i - > and feeding the same file 10000 times. Every 100th request will start a > new TCP connection. > > > In apt this manifests as some XX00 request failing. > > Get:4500 http://mirror.init7.net/debian bullseye/main amd64 libplymouth5 > amd64 0.9.5-2 [108 kB] > Err:4500 http://mirror.init7.net/debian bullseye/main amd64 libplymouth5 > amd64 0.9.5-2 > Undetermined Error [IP: 2001:1620::1620 80] > > > Get:1600 http://mirror.init7.net/debian bullseye/main amd64 libastyle3 amd64 > 3.1-2+b1 [109 kB] > Err:1600 http://mirror.init7.net/debian bullseye/main amd64 libastyle3 amd64 > 3.1-2+b1] > Undetermined Error [IP: 2001:1620::1620 80] > > > > This is repeatable and reproducible. I can try 10 times doing the > download, and every time the same file and number will fail. (Sometimes I > might get lucky and one of them will succeed, but not ther other). > > I don't know why files 100, 200, 300, ... doesn't fail, but it looks like > some bug in apt in general. > > (The above files do exist and are served correctly by the server). > > > Does apt use maybe HTTP pipelining in addition to keepalive, that causes > issues? Or maybe in general the retry logic is broken somehow.
Yes - pipelining is absolutely vital to get sensible performance. Did you enable retries? Anyway, go compile apt and add debugging code in appropriate places and figure out where it's broken. I don't have a reproducer and nor the time nor the ability to investigate where the breakage is - it generally starts working before I've really had a chance to fix it. > > I wasn't able to reproduce this issue with Fastly CDN or Apache 2.4, > probably because they use different keepalive request limit policy. It > might be a bug in nginx too. I am not sure. The server I was using > appears to be using nginx/1.14.2. More likely a function of MTU or network speed or something than the server software, and the concrete list of packages and the order in which they are downloaded is important too -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
