Dear all,
I am using fetchmail to get messages from various accounts into Cyrus imapd.
Generally, this has been working for years now.
Recently, I took a look into my system's log files and wondered why the mail
log files were growing very fast. Further analysis showed that there are email
messages in some of my external server accounts which are malformed. fetchmail
downloads these and tries to deliver them via lmtp to Cyrus imapd; this fails
because they are malformed (probably some part of the header). fetchmail is
aware of the dlivery failure and does not flush the messages on the external
server (which is the correct and desired behaviour).
Consequently, the whole thing restarts with the next poll.
Now, I would like to know if there is any way to get _every_ message into cyrus
via fetchmail, even if it is malformed.
There are several reasons why I want to do this, the most important ones being:
- The log files grow very fast because the poll interval is short, and I have
several dozens of these messages (mainly spam) in each of my external accounts.
- For legal reasons (I am handling my professional email using this system), I
must not lose any message which is sent to me, even if it is spam or it is
malformed.
The latter one is the reason why I can't just delete the respective messages on
the external servers.
I have put a lot of time into debugging and trying to find out what is
happening there and how to get around it, but the respective error messages are
worrying, and debugging cyrus does not work like it is described in the
respective documents (I'll thoroughly describe this in one of my next messages
because it's a different subject from what I am asking here).
My preferred solution would be one which does not involve sendmail or postfix
or the like, but if using one of the "big" MTAs / MDAs is the only way to
achieve what I'd like to, I'll do that.
This is on debian wheezy, cyrus imapd 2.4.16, fetchmail 6.3.21 (both probably
including some debian patches which could cause changes in the behaviour that I
am not aware of).
Some lines from /etc/fetchmailrc:
poll
pop.gmx.net
proto pop3
bad-header accept
user "f...@gmx.de"
ssl
pass "secret"
is "gmx_foo" here
smtphost "/var/run/cyrus/socket/lmtp"
Some lines from /etc/cyrus.conf:
START {
...
# At least one form of LMTP is required for delivery
# (you must keep the Unix socket name in sync with imap.conf)
#lmtp cmd="lmtpd -a" listen="localhost:lmtp" prefork=0
maxchild=20
lmtpunix cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp"
prefork=0 maxchild=20
...
}
The respective log entries are like that (for example, in /etc/mail.log):
May 28 09:38:04 spock fetchmail[2434]: reading message
f...@gmx.de@pop.gmx.net:43 of 80 (2284 octets) (log message incomplete)
May 28 09:38:04 spock fetchmail[2434]: Connection errors for this poll:#012name
0: connection to localhost:smtp [127.0.0.1/25] failed: Connection
refused.#012name 1: connection to localhost:smtp [127.0.0.1/25] failed:
Connection refused.
May 28 09:38:04 spock fetchmail[2434]: not flushed
(Note: It has turned out that this error message is quite worrying; there are
definitely no connection errors with the socket. The socket exists and happily
accepts all messages which are not malformed. But the worrying log message is
probably fetchmail's fault, so a discussion about that would be off-topic.)
What I already have tried:
- All combinations of configuration directives "reject8bit" and "munge8bit" in
/etc/imapd.conf
- Using the lmtpd daemon listening on localhost:lmtp (instead of socket) by
changing /etc/cyrus.conf and changing /etc/fetchmailrc accordingly
This did not change the situation, the log messages are basically the same, and
the malformed messages still don't make their way into cyrus imapd.
Starting fetchmail with the additional option --verbose leads to log entries
like that:
May 27 20:53:00 spock fetchmail[18771]: reading message
f...@gmx.de@pop.gmx.net:39 of 80 (2298 octets) (lo$
May 27 20:53:00 spock fetchmail[18771]: LMTP> MAIL
FROM:<u...@harttweb.hartford.edu> SIZE=2298
May 27 20:53:00 spock fetchmail[18771]: LMTP< 250 2.1.0 ok
May 27 20:53:00 spock fetchmail[18771]: LMTP> RCPT TO:<gmx_foo@localhost>
May 27 20:53:00 spock fetchmail[18771]: LMTP< 250 2.1.5 ok
May 27 20:53:00 spock fetchmail[18771]: LMTP> DATA
May 27 20:53:00 spock fetchmail[18771]: LMTP< 354 go ahead
May 27 20:53:00 spock fetchmail[18771]: LMTP>. (EOM)
May 27 20:53:00 spock fetchmail[18771]: LMTP< 554 5.6.0 Message contains
invalid header
May 27 20:53:00 spock fetchmail[18771]: Trying to connect to
127.0.0.1/25...connection failed.
May 27 20:53:00 spock fetchmail[18771]: connection to localhost:smtp
[127.0.0.1/25] failed: Connection refused.
May 27 20:53:00 spock fetchmail[18771]: Trying to connect to
127.0.0.1/25...connection failed.
May 27 20:53:00 spock fetchmail[18771]: connection to localhost:smtp
[127.0.0.1/25] failed: Connection refused.
May 27 20:53:00 spock fetchmail[18771]: Connection errors for this
poll:#012name 0: connection to localhost:smtp [127.0$
May 27 20:53:00 spock fetchmail[18771]: not flushed
So the reason for failure is that Cyrus lmtpd refuses the message with error
code "554 5.6.0 Message contains invalid header". Unfortunately, I have found
no way to find out more about this (see my next message to this list).
Thank you very much for any ideas,
Recursive
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
