On Tue, 23 Sep 2003, Pat Lashley wrote: > --On Tuesday, September 23, 2003 02:18:50 -0700 Chris Stromsoe > <[EMAIL PROTECTED]> wrote: > > > out of ~40,000 messages I've got handy, between 50 and 60 had some > > variant of envelope-to (either plain or with x- or old-), and 36,906 > > had a return-path header. Return-Path is an rfc mandated header that > > must be handled. > > None of the messages should have had either of those headers when you > recieved them.
It's very unlikely that any of the messages did have a Return-Path when they were handed to me. I wasn't saying that they did. All of the messages containing envelope-to variants came that way. (And, in fact, there is no reason why they shouldn't -- as you said yourself, envelop-to is not a standard header; it could be inserted at either end.) > Yes, the RFC mandates a Return-Path header; but it specificly says > that it is to be generated only at the point of final delivery into > a file. Keep reading the RFC. It also says that the message is considered delivered when it leaves the SMTP environment. Ie, when lmtpd takes it, lmtpd should add the Return-Path header. > >> I keep mentioning Envelope-To because 30+ years of Software > >> Engineering experience tells me that if you handle Return-Path, > >> people will also want Envelope-To. .... > > I'm still not sure how Envelope-To applies to this discussion. > > Please re-read the first quoted paragraph above. Repeat until you > DO understand how Envelope-To applies to this discussion... This seems very much like you're sniping at me. I almost expected the sentence to end with ", junior" or something similar. It's nice that you have a lot of experience doing software development and that you bring your experience to play. However, I don't believe that your experience has anything to do with this issue. The RFC _requires_ Return-Path. It is standard. Envelope-To is something that somebody made up. I can't recall it being mentioned in any smtp rfc that I've read recently. I can't recall seeing it generated by any MTA. You keep asserting that it has something to do with this and that I should somehow "know" how it fits into the discussion. I don't believe that Envelope-To has anything to do with discussing whether or not _lmtpd should make sure that its in-memory data matches what it writes to disk_. > > On Tue, 23 Sep 2003, Pat Lashley wrote: > > > >> --On Monday, September 22, 2003 22:40:00 -0700 Chris Stromsoe > >> <[EMAIL PROTECTED]> wrote: > >> > >> > That, imho, is not correct behavior. > >> > >> What headers, aside from Return-Path, are not cached in RAM? > > > > The Received header that lmtpd adds and the X-Sieve header. There may be > > others in other areas of the code, but those two and the Return-Path > > header are the three that I saw. > > Well, it makes sense taht the X-Sieve header wouldn't be applied until > sieve has processed the message. The x-sieve header is applied regardless of sieve script processing. If you enable sieve with cyrus, it is _always_ added. > As to lmtpd's Received header; I could certainly see an argument for > including it if lmtpd has already sent back a positive response to the > MTA. However, if LMTP runs sieve while the LMTP connection is still > open, then it is right to withold that header until it has actually > accepted the message. Two things: - lmtpd has written the received header to disk. When that message is forwarded or bounced the on disk file is sent. If on-disk format is canonical, then the header cache needs to have the headers too. If the header cache is canonical, then the on-disk representation needs to not have those headers until the message is dispensed with. - sendmail provides a Return-Path header to procmail, the final delivery agent. sendmail could also provide one to lmtpd, but lmtpd provides its own. lmtpd is the "local mail transport protocol". Like a local mailer in sendmail. It is the final delivery step. Sure, other things can happen after that; those things have nothing to do with anything that happened up to that point. When sieve forwards or rejects a message, it invokes sendmail as a client. It doesn't try to relay the message through the next hop or anything like that. It reinjects the message. > > There is no RFC or draft that says that the return-path header should > > not be visible as a header to sieve scripts, whereas 2821 does say > > that the final delivery must add the return-path header. To me, that > > says that return-path should be added by lmtpd and be visible to > > sieve. > > You are assuming that sieve processing occurs -AFTER- final delivery. > The authors of lmtpd apparently consider it to be happening -BEFORE- > final delivery; with the sieve script itself determining whether there > will even BE a final delivery at that point. Or, the authors of lmtpd never considered this situation and it wasn't a big deal having or not having the missing headers in the cache. I'll let them speak for themselves. > It makes no sense to you because you continue to believe that it is a > shortcoming and not a correct behavour. Stop trying to justify your > 'solution' and try to take an objective look at the problem. You will > find that it is solved admirably by the envelope extension. I objectively believe that lmtpd is maintaining an on-disk structure that does not match the structure that it is maintaining in-memory. I objectively believe that this is wrong. I noticed this when I was looking to find out why something I wanted to do wasn't working. The two things are _completely separate_. > >> You're trying to change LMTP to provide a header that WILL NOT EXIST > >> in other sieve environments. It will not be at all obvious to the > >> naive user that that header was inserted at the last minute rather > >> than following the message from its inception. > > > > I still don't see what it is that you are trying to say. You seem to > > be assuming some sort of lmtp/sieve interdependance. Please correct > > me if I'm wrong. > > There is an interdependance -in this implementation- because lmtpd is > the agent that invokes the sieve processing. Your mistake is in > considering the sieve processing to be done -after- lmtpd performs final > delivery. In actual fact, it is done once lmtpd has accepted the > message but -before- it determines whether there will even be a final > delivery. It cannot know whether there will be a final at this point > until the sieve script has been run. Therefore it MUST NOT (in RFC > terminology) add the Return-Path header before sieve processing. lmtpd is the final delivery. Read more of 2821: When the delivery SMTP server makes the "final delivery" of a message, it inserts a return-path line at the beginning of the mail data. This use of return-path is required; mail systems MUST support it. The return-path line preserves the information in the <reverse- path> from the MAIL command. Here, final delivery means the message has left the SMTP environment. Normally, this would mean it had been delivered to the destination user or an associated mail drop, but in some cases it may be further processed and transmitted by another mail system. Which actually implies that lmtpd is wrong to be adding the Return-Path and that the smtp server handing off to lmtpd should be adding it. Which is beside the point. > > As far as being not obvious to the naive user ... when has that > > stopped anything from happening? > > Wasn't the basis for your original 'problem' that naive users could not > understand why they couldn't filter on the Return-Path header? No. I didn't say that anywhere. Are you reading what I'm writing or just looking to argue with me? > >> You are confusing functionality with implementation. A common mistake. > > > > No, I'm not: I need the functionality of being able to filter on all of > > the headers in the message. > > No, that's the implementation that you want, not the functionality that > you need. Sorting those two out is often one of the most difficult > tasks for the software engineer. Say again? I know the functionality of what I want. I don't care how it is implemented. There are headers. I want to be able to filter on all of them. Not a subset of them, but all of them. I'm not overly concerned with how that is implemented, just that it is implemented. > > Return-Path is a header. > > Once final delivery has actually been made, it is a header. Until then > all you have is the envelope sender. final delivery occurs when smtp passes the message to lmtpd. What lmtpd does with it afterward is its own business. > > If there were a separate sieve parser (a la procmail) and lmtpd forked > > off sieve to take action, it would act on the on-disk file and would > > see the Return-Path header. The built-in sieve parser should see the > > same set of headers. > > If there were a separate sieve parser, lmtpd would quite probably feed > it the message (without a Return-Path header) via a pipe; Look into how sendmail feeds procmail. I use procmail to filter on return-path all the time. > and the seive > processor would need to invoke some other mechanism to actually perform > a final delivery. If lmtpd did use the file, it would quite probably > NOT include the Return-Path header in that file. (Because to do so > would be an RFC violation since final delivery has not yet been made.) sieve opens the on-disk file and _feeds it to sendmail_ for redirecting. The on-disk file that _includes the Return-Path_. So, is the on-disk file just an optimization? Or is it the "final delivery" and being used as such by other parts of lmtpd? Have you looked at the code to see what it's doing? Is there some technical fault with what I'm proposing? Or are you being disagreeable because you don't personally like the return-path header or because I've somehow offended your sensibilities? -Chris
