When a newsreader requests article headers from a news server
with an OVER (formerly XOVER) command, the server sends the
headers of each article combined into a single line, with the
contents of the various headers separated by tabs, in this
order (quoting from Section 8.3.2 of RFC 3977
http://tools.ietf.org/html/rfc3977#section-8.3):
"0" or article number
Subject header content
From header content
Date header content
Message-ID header content
References header content
:bytes metadata item
:lines metadata item
For all fields, the value is processed by first removing all
CRLF pairs (that is, undoing any folding and removing the
terminating CRLF) and then replacing each TAB with a single
space.
Since tabs separate the various header content fields of the
line the server sends to the newsreader, obviously those fields
must not contain any tabs. Otherwise the newsreader won't be
able to count the tabs in order to distinguish and separate
the fields, and thus reconstruct the headers. That's why
RFC 3977 says that tabs in any of the header contents must be
replaced by spaces before those contents are combined and sent
to the newsreader.
Unfortunately Astraweb doesn't replace those tabs with spaces.
Tabs occur most frequently in the References header, when the
article is two or more followup levels below the original post.
The Pan newsreader is unable to retrieve, or sometimes even see,
articles from Astraweb whose References header is folded using
tabs. Most newsreaders don't post that way, but some do.
That's why Pan couldn't download from Astraweb three of the
five test articles that I posted. And why, from Astraweb,
Pan can't see all the articles in deeply nested followup
trees, and sometimes sees a tree broken into smaller ones.
This was all determined by capturing the stream of alt.test
headers, from Astraweb and from Blocknews, using the wireshark
network packet analyzer. Blocknews formats the header records
properly, removing tabs from the field values before combining
them.
I'll send another trouble-ticket to Astraweb, but after their
previous performance I don't know if they'll care. It shouldn't
be hard to insert code into Pan to interpret the incoming header
stream defensively and determine when the References field ends,
which is what KNode and other newsreaders must be doing, since
they work OK with Astraweb.
_______________________________________________
Pan-users mailing list
Pan-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/pan-users
