Package: apt Version: 0.5.27 Severity: important Hi,
retrieving packages with apt-get via an HTTP proxy (e.g. apt-proxy) that uses "Transfer-Encoding: chunked" leads to random data corruption in the retrieved package files and an error message: |nevermind:~# apt-get install libx11-6 |Failed to fetch http://nevermind.krynn:9999/debian/pool/main/x/xfree86/libx11-6_4.3.0.dfsg.1-10_i386.deb MD5Sum mismatch Using ethereal to capture the network traffic and comparing the garbled package file with a good one I found that data corruption always occurs when a chunk of data starts with a <CR> (0x0d). "strace -f apt-get install ..." revealed that /usr/lib/apt/methods/http is the culprit: >> pid 26438 is /usr/lib/apt/methods/http >> fd 3 is the socket of the server connection >> fd 4 is "/var/cache/apt/archives/partial/libx11-6_4.3.0.dfsg.1-10_i386.deb" [...] [pid 26438] read(3, "ff0\r\n\r\364\6{C\275\351\336L\357p\357d\357\266\336\35"..., 11906) = 1448 ^^^^^^ start of DATA ^^^^ CRLF delimiter ^^^ Data Length Token of the Chunked Transfer-Encoding [pid 26438] read(3, 0x8063656, 10458) = -1 EAGAIN (Resource temporarily unavailable) [pid 26438] select(5, [0 3], [4], NULL, {120, 0}) = 1 (out [4], left {120, 0}) [pid 26438] write(4, "\364\6{C\275\351\336L\357p\357d\357\266\336\35\275\317"..., 1442) = 1442 [...] ^ This block should begin with '\r' and the size should be 1443! (1448 byte returned by read() minus 3 byte chunk length code minus 2 byte CRLF) A look at the source code of the http method revealed that the CircleBuf::WriteTillEl() method is broken. Assuming we entered the routine to read the data length token at the start of a data chunk this is what happens: | for (unsigned long I = OutP; I < InP; I++) | { | if (Buf[I%Size] != '\n') | continue; >> we get here with 'I' pointing to the '\n' after the token, >> having already skipped the preceding '\r' | ++I; >> we skip the delimiter, leaving 'I' pointing to the first DATA >> byte of the chunk! | if (I < InP && Buf[I%Size] == '\r') | ++I; >> IF the first DATA byte happens to be '\r' it will also be skipped! >> Exactly what I have seen in the strace log >> Since the block size is known and the DATA block is delimited >> with '\r\n' The '\r' of the final delimiter will show up in the >> data block, making the block appear to have been rotated by one >> byte. | | if (Single == false) | { | if (Buf[I%Size] != '\n') >> Not relevant here, BUT since 'I' might be equal to InP >> already we might be checking an invalid byte from the buffer! | continue; | ++I; | if (I < InP && Buf[I%Size] == '\r') | ++I; | } At the first glance of the routine the sequence of comparisons '\n', '\r' looked suspicious to me (the delimiter sequence is '\r\n', so the comparisons ought to be in the same order!). So I decided to swap them (directly editing the binary using hexl-mode of emacs ;) so that the loop terminates after it has found a '\r\n' sequence as it should. With this change I have successfully installed a dozen packages that have failed with MD5Sum errors before. Lothar Wassmann -- Package-specific info: -- apt-config dump -- APT ""; APT::Architecture "i386"; APT::Build-Essential ""; APT::Build-Essential:: "build-essential"; APT::Clean-Installed "off"; Dir "/"; Dir::State "var/lib/apt/"; Dir::State::lists "lists/"; Dir::State::cdroms "cdroms.list"; Dir::State::userstatus "status.user"; Dir::State::status "/var/lib/dpkg/status"; Dir::Cache "var/cache/apt/"; Dir::Cache::archives "archives/"; Dir::Cache::srcpkgcache "srcpkgcache.bin"; Dir::Cache::pkgcache "pkgcache.bin"; Dir::Etc "etc/apt/"; Dir::Etc::sourcelist "sources.list"; Dir::Etc::vendorlist "vendors.list"; Dir::Etc::vendorparts "vendors.list.d"; Dir::Etc::main "apt.conf"; Dir::Etc::parts "apt.conf.d"; Dir::Etc::preferences "preferences"; Dir::Bin ""; Dir::Bin::methods "/usr/lib/apt/methods"; Dir::Bin::dpkg "/usr/bin/dpkg"; DPkg ""; DPkg::Pre-Install-Pkgs ""; DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10"; DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true"; DPkg::Tools ""; DPkg::Tools::Options ""; DPkg::Tools::Options::/usr/bin/apt-listchanges ""; DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2"; Acquire ""; Acquire::http ""; Acquire::http::Pipeline-Depth "0"; -- (no /etc/apt/preferences present) -- -- /etc/apt/sources.list -- # This sources.list was generated by apt-sources v2.20 # Copyright(c) 2001 by Danny Rodriguez # deb http://nevermind.krynn:9999/debian testing main contrib non-free deb http://nevermind.krynn:9999/non-US testing/non-US main contrib non-free deb http://nevermind.krynn:9999/security testing/updates main contrib -- System Information: Debian Release: 3.1 APT prefers testing APT policy: (500, 'testing') Architecture: i386 (i686) Kernel: Linux 2.6.8.1 Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Versions of packages apt depends on: ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an ii libgcc1 1:3.4.3-6 GCC support library ii libstdc++5 1:3.3.5-5 The GNU Standard C++ Library v3 -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
