On Sun, Jan 14, 2018 at 06:06:28PM +0100, Philipp Kern wrote:
> That said, you just went the other way. Not that I liked the cURL https
> transport, given that it was fairly stupid^Wstraightforward. But at
^^^ you must be joking…¹
> least it used a library. So I guess I'm curious if you'd be open to
> picking a library and then rewriting everything to use that single one
> for all of HTTP(S) 1.1/2. (Assuming such a library exists by now.)Well, curl wasn't a good choice for implementing https for us nowadays as it is big library supporting lots of things we don't want (apt downloads over IMAPS after a http redirect, Transport-Encoding: gzip, …) and not supporting many things we do (SRV records, working with hashes …). Many improvements to https were only possible by sidestepping curl – that isn't how you want to interact with a library. If there is a library which would give us what we ended up with by combining our http with a TLS wrapping we could change to that (assuming its not too obscure, linux-only, … and so on). There is still a lot of code to be written for the actual implementation of the transport as we want to be in control of a lot of things: Redirects e.g. can't be handled internally by the library: We like to limit what is possible (e.g. no https→http redirects) and want to keep an instance single-connection: a transport can't change the server it talks to at runtime as it has no access to the configuration applying potentially for this server (auth, settings, …). Beside, it isn't very realistic that the library does play as much with hashes as we do from terminating the connection if the content-length didn't match what we expect to fixing pipelining problems at runtime simply because that isn't how the rest of the internet works… Of course, that sounds like NIH, but on the other hand we didn't want to implement TLS by hand nor am I particular interested in hand-rolling the binary parts of http2 if it can be helped, so from my viewpoint its not NIH but "just" the guesstimate that there will be no off-the-shelf solution. A transport is "just" a lot more than a wget wrapped in C code. ¹ yes, I am allowed to "bash" curl as I am due to a very minor contribution in relation to SOCKS² error reporting considered a contributor as I recently found out. So apt leaving curl had at least some advantage for both sides I guess. ;) ² something I thought I wouldn't implement myself either, so perhaps there is some hope for me doing http2 NIH-style still ;) > > [HTTP/2 has an unencrypted variant aka "h2c" btw. At least on paper > > – I know that browsers aren't going to implement it.] [That was just a comment on "requires TLS" – on paper it doesn't. I am well aware that this might never really exist in practice and I am not concerned either way.] > >> Happy to hear your thoughts on how to solve this. > > You could do something similar to what httpredir.d.o was doing or work > > with the (now reimplemented) -mirror method. That hard-depends on your > > "behind load-balancer" servers to be reachable directly through. But it > > gets you parallel connections without changing clients (at least in the > > first case, in the second you will need to wait a few years until this > > becomes true I guess). > > That's actually an excellent idea. We'll try that out. I actually wrote > code for it already and it's a pretty straightforward approach, but > we'll take another stab at attempting to reduce time to first byte > latency first. Julian commented on IRC a while ago that I wasn't explaining the second option – using the mirror method – very well. So for the record: Assuming you have 3 mirrors and a load-balancer you could place a mirrorlist file containing the 3 mirrors on the balancer, point your clients to it and be done. In pre-1.6 versions apt will consistently choose one of the mirrors [based on the hostname of the client] – so you get what you have at the moment more or less. With 1.6 onwards apt will download each file from a randomly picked server³, so you will end up with 3 parallel downloads in practice [expect if you are randomly unlucky or if you run 'update' as that will stick to the same mirror by default to avoid sync problems]. See the apt-transport-mirror manpage (in 1.6) for advanced details. ³ more exact: The list of three mirrors will be shuffled randomly for each file and then the download is attempt in this order, so it isn't exactly load-balanced, but close enough for now. Anyhow: What are we going to do with this bugreport now? Close, retitle for http2 support or is there some other action we can take to fix the initial problem? Best regards David Kalnischkies
signature.asc
Description: PGP signature
