Re: [Mailman-Users] Python process size grows 30x in 8 hours (memory

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 2, 2008, at 11:15 PM, Mark Sapiro wrote:
Yes, I know. I was just about to resend. It is attached here. The  
MUA I
used to send the previous message gives any attachment without an
extension Content-Type: application/octet-stream, so the list's  
content
filtering removed it.

Ah, np.

|> It does the following compared to the normal script.
|
|> The normal script reads the message from the pipe from the MTA and
|> queues it in the 'in' queue for processing by an IncomingRunner.  
This
|> script receives the message and instead queues it in the 'bad'  
queue.
|> It then looks at the size of the 'bad' queue entry (a Python pickle
|> that will be just slightly larger than the message text). If the  
size
|> is less than MAXSIZE bytes (a parameter near the beginning of the
|> script, currently set to 1000000, but which you can change as you
|> desire), it moves the queue entry from the 'bad' queue to the 'in'
|> queue for processing.
|
| I'm not sure 'bad' should be used.  Perhaps a separate queue called
| 'raw'?  It is nice that files > MAXSIZE need only be left in 'bad'.


If we're going to do something like this going forward, we can  
certainly
change the queue. For this 'debug' effort, I wanted to keep it simple
and use an existing mm_cfg queue name.

Excellent point.  A couple of very minor comments on the file, but  
other than that, it looks great.  (I know you copied this from the  
original file, but still I can't resist. ;)

# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software  
Foundation, Inc.

1998-2008


#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
02110-1301, USA.

"""Accept posts to a list and handle them properly.

The main advertised address for a list should be filtered to this  
program,
through the mail wrapper.  E.g. for list [EMAIL PROTECTED]', the  
`test'
alias would deliver to this script.

Stdin is the mail message, and argv[1] is the name of the target  
mailing list.

"""

import os
import sys

import paths
from Mailman import mm_cfg
from Mailman import Utils
from Mailman.i18n import _
from Mailman.Queue.sbcache import get_switchboard
from Mailman.Logging.Utils import LogStdErr

LogStdErr("error", "post")

MAXSIZE = 1000000



def main():
   # TBD: If you've configured your list or aliases so poorly as to  
get
   # either of these first two errors, there's little that can be  
done to
   # save your messages.  They will be lost.  Minimal testing of new  
lists
   # should avoid either of these problems.
   try:
       listname = sys.argv[1]
   except IndexError:
       print >> sys.stderr, _('post script got no listname.')
       sys.exit(1)
   # Make sure the list exists
   if not Utils.list_exists(listname):
       print >> sys.stderr, _('post script, list not found: % 
(listname)s')
       sys.exit(1)
   # Immediately queue the message for the incoming qrunner to  
process.  The
   # advantage to this approach is that messages should never get  
lost --
   # some MTAs have a hard limit to the time a filter prog can run.   
Postfix
   # is a good example; if the limit is hit, the proc is SIGKILL'd  
giving us
   # no chance to save the message.
   bdq = get_switchboard(mm_cfg.BADQUEUE_DIR)
   filebase = bdq.enqueue(sys.stdin.read(),
                          listname=listname,
                          tolist=1, _plaintext=1)

Should probably use True there instead of 1.

   frompath= os.path.join(mm_cfg.BADQUEUE_DIR, filebase + '.pck')
   topath= os.path.join(mm_cfg.INQUEUE_DIR, filebase + '.pck')

Space in front of the =



   if os.stat(frompath).st_size < MAXSIZE:
       os.rename(frompath,topath)

Space after the comma.


if __name__ == '__main__':
   main()

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkhsTbQACgkQ2YZpQepbvXGGigCfe+w4Ynz/qvFEp6VmurbySv42
b6cAoJa9wuSaql8dLUo8/VXT/Sxiu9pW
=Ywsy
-----END PGP SIGNATURE-----
------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9