Re: [Python-Dev] Daily documentation builds

[Neal Norwitz]

I ran 2.6, 3.0, and 3.1 manually.  2.7 should get picked up on the
next run.  The problem is that regrtest.py -R hangs from time to time
which caused the machine to run out of memory.  Does anyone else have
regrtest.py -R hang for them?  Some tests were disabled to try to
prevent the problem, but it still happens from time to time.

n

On Tue, Feb 10, 2009 at 11:33 AM, Raymond Hettinger  wrote:
> The online development docs stopped their nightly rebuilds four days ago.
>
>
> Raymond
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Daily documentation builds

[Neal Norwitz]

On Wed, Feb 11, 2009 at 6:24 AM, A.M. Kuchling  wrote:
> On Tue, Feb 10, 2009 at 09:16:48PM -0800, Neal Norwitz wrote:
>> I ran 2.6, 3.0, and 3.1 manually.  2.7 should get picked up on the
>> next run.  The problem is that regrtest.py -R hangs from time to time
>> which caused the machine to run out of memory.  Does anyone else have
>> regrtest.py -R hang for them?  Some tests were disabled to try to
>> prevent the problem, but it still happens from time to time.
>
> It's also possible that tools/sphinx needs a manual 'svn update'. A
> recent change to sphinxext/pyspecific.py imports a new package,
> sphinx.builders.  I had to do this to keep my source tree building the
> docs.

Ok, I'll take a look later.  Misc/build.sh should be doing a "make
update" which should fix most problems.  There was one problem a while
back I had to fix manually though.  If you see ways to make Misc/build
more robust, feel free to check in changes.  The doc section is at the
bottom of the file.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding support to curses library

[Neal Norwitz]

On Tue, Feb 24, 2009 at 2:18 PM, Heracles
 wrote:
>
> Hello,
>
> I am working on a patch to add to the _cursesmodule.c file of the Python
> core libraries.  I figured I would take on one of the implemented functions
> to try to get my feet wet contributing to the project.  At any rate, I have
> the following function defined in the 2.7.a version updated from SVN this
> morning:

I'm glad you are interested in developing Python.  I'm not sure if
this is the best forum.  OTOH, I'm not sure if comp.lang.python would
be appropriate either.

I'd suggest making a proper patch and submitting it to http://bugs.python.org

> - Snippet ---
> // Insert new method color_set Steve Owens 2/24/2009
> //   The curses library color_set function has the following signature:
> //       int color_set(short color_pair_number, void* opts);
> static PyObject *
> PyCurses_color_set(PyObject *self, PyObject *args)
> {
>   short color_pair_number;
>   void * opts;
>   int erg;
>
>   // These macros ought to be documented in the API docs
>   // but they aren't yet.
>   PyCursesInitialised
>   PyCursesInitialisedColor
>
>   // Per ncurses Man Page:
>   //   The routine color_set sets the current color of the given window to
>   // the foreground/background combination described by the
> color_pair_number.
>   // The parameter opts is reserved for future use, applications must
> supply a
>   // null pointer.
>   switch(PyTuple_Size(args))
>   {
>   case 1:
>           // Dont make them pass a useless null pointer.
>           if (!PyArg_ParseTuple(args, "h", &color_pair_number)) return NULL;
>           break;
>   case 2:
>           // Allow them to pass the opts pointer so that when ncurses is later
> updated.
>           // This method will still work.
>           if (!PyArg_ParseTuple(args, "hO&", &color_pair_number, &opts)) 
> return
> NULL;
>           break;
>   default:
>      PyErr_SetString(PyExc_TypeError, "color_set requires 1 or 2 arguments
> (color_pair_number[, opts]?)");
>          return NULL;
>   }
>
>   erg = color_set(color_pair_number, opts); // Debating on forcing null
> here.
>
>   if (erg == ERR)
>          return PyCursesCheckERR(erg, "color_set");
>   else
>      PyInt_FromLong((long) 1L);

I did a cursory review of the patch and if this is the exact code,
this is a problem.  You are missing a return statement.  The compiler
should have issued a warning for this too.

> }
> -End  Snippet ---
>
> I also have the following added in (see last line of the snippet):
>
> - Snippet ---
> static PyMethodDef PyCurses_methods[] = {
>  {"baudrate",            (PyCFunction)PyCurses_baudrate, METH_NOARGS},
>  {"beep",                (PyCFunction)PyCurses_beep, METH_NOARGS},
>  {"can_change_color",    (PyCFunction)PyCurses_can_change_color,
> METH_NOARGS},
>  {"cbreak",              (PyCFunction)PyCurses_cbreak, METH_VARARGS},
>  {"color_content",       (PyCFunction)PyCurses_Color_Content,
> METH_VARARGS},
>  {"color_pair",          (PyCFunction)PyCurses_color_pair, METH_VARARGS},
>  {"color_set",           (PyCFunction)PyCurses_color_set, METH_VARARGS},
> -End  Snippet ---
>
> The code compiles and installs fine, but when I run the following unit test,
> I get a segmentation fault:
>
> - Snippet ---
> import unittest, curses
> from test import test_support
>
> def testCursesColorSet(stdscrn):
>   curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
>   curses.init_pair(2, curses.COLOR_WHITE, curses.COLOR_BLUE);
>   i = curses.color_set(1, NULL);
>   stdscrn.addstr("RED/BLACK (%0)\n".format(i))
>   i = curses.color_set(2, NULL);
>   stdscrn.print("WHITE/BLUE (%0)\n".format(i))
>   i = curses.color_set(0, NULL);
>   stdscrn.print("Default (%0)\n".format(i))
>
>
> def test_main(stdscrn):
>   curses.savetty()
>   if curses.has_color():
>      testCursesColorSet(stdscrn)
>   else
>      stdscr.addstr( "Test Aborted: Color not supported on this terminal.")
>
>
> if __name__ == '__main__':
>    curses.wrapper(test_main)
> -End  Snippet ---
>
> It turns out that by commenting out this line in the _cursesmodule.c code,
> allows the unit test to run
> obviously reporting the error as expected:
>
> - Snippet ---
> //erg = color_set(color_pair_number, opts); // Debating on forcing null
> here.
> -End  Snippet ---
>
> At any rate I am stuck.  I am still trying to build just a plain C file
> which will test the color_set function
> outside of python, but that is another task.
>
> Any suggestions?

Beyond what I said above, typically you need to go the next step.
Fire up a debugger and determine exactly where and why it's crashing.

Good luck!

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mai

[Python-Dev] cleanup before 3.1 is released

[Neal Norwitz]

Has anyone run valgrind/purify and pychecker/pylint on the 3.1 code
recently?  Both sets of tools should be used before the final release
so we can fix any obvious problems.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Neal Norwitz]

On Wed, Oct 21, 2009 at 6:56 PM, Mike Krell  wrote:
> Is there any possibility of backporting support for the nonlocal keyword
> into a  2.x release?  I see it's not in 2.6, but I don't know if that was an
> intentional design choice or due to a lack of demand / round tuits.  I'm
> also not sure if this would fall under the scope of the proposed moratorium
> on new language features (although my first impression was that it could be
> allowed since it already exists in python 3.

IIRC, Jeremy Hylton tried to backport it during a Pycon sprint a few
years back.  I think it was difficult and he dropped it.  I don't know
if there were any showstoppers or if Jeremy saved his work...or if my
memory is even correct. :-)

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thanks to everyone cleaning up the tests

[Neal Norwitz]

On Mon, Nov 2, 2009 at 2:18 AM, Mark Dickinson  wrote:
> On Sat, Oct 31, 2009 at 9:47 PM, Brett Cannon  wrote:
>> Just wanted to publicly thank everyone who has been causing all the
>> checkins to fix and stabilize the test suite (I think it's mostly
>> Antoine and Mark, but I could be missing somebody; I'm under a
>> deadline so only have marginal higher brain functionality).
>
> Not guilty, your honour.  Blame Antoine and David.  :)
>
> David's new buildslave seems to be making a difference, too.

It's really been awesome to see all the work improving the test suite.
 This will make it easier to implement change without fear of breaking
everything.  These changes will reap benefits for a long time to come.

Kudos for picking up this work and to Brett for calling you out. :-)

Thanks,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of the Buildbot fleet and related bugs

[Neal Norwitz]

On Fri, Nov 6, 2009 at 12:27 AM, Mark Dickinson  wrote:
> On Fri, Nov 6, 2009 at 3:53 AM, R. David Murray  wrote:
>
>>    (2) issue 4970: consistent signal 32 error on the norwitz-x86 Gentoo
>>        buildslave in 3.1 and 3.x.  This may be due to the box
>>        running an old threading library, but it does make one wonder what
>>        changed in 3.x that exposed it.
>
> This error has been happening since well before 3.0 was released.  Asking
> for access to Neal's machine is probably the only sensible way to diagnose
> it. (A less invasive but slower way to debug would be to create a branch
> especially for this bug and do repeated runs to figure out which part of 
> test_os
> is causing the failure.)

IIRC, I spent quite a bit of time trying to nail this down.  I don't
remember finding any useful information on the cause (beyond narrowing
it to some tests).  As Mark said, this has been happening for a long
time.

I'm reticent to provide access to the machine, as it's not really
mine.  I'm not even sure I have access, I haven't logged in for a long
time.

I'd just like to say thanks again to everyone for making the buildbots
more green and also improving the general testing infrastructure for
Python.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Neal Norwitz]

On Mon, Jan 25, 2010 at 2:34 PM, Victor Stinner
 wrote:
>
> Interaction with the Python developers
> ==
>
> I open an issue for each bug found in CPython. I describe how to reproduce it
> and try to write a patch. I have learn to always write an unit test, useful to
> reproduce the bug, and it makes Python commiters happy :-)

Yes! :-)

> The reaction depends on the impacted component, the severity of the bug, the
> complexity of the code reproducing the bug, and the quality of my bug report
> :-) The answer was always quick for core components. But some modules are
> maintained by everyone, which means nobody, like imageop, audioop or
> cProfile/hotshot. Having a module maitainer, like Guilherme Polo aka gpolo for
> Tkiner, does really help!
>
> It looks like fuzzing bugs are not always appreciated by developers, maybe
> because they are always "borderline" cases (not "realist").

As many others have said, I greatly appreciate your work in this area.
 Even if obscure, all of the crashes are important to fix.  Stability
has been a big selling point of Python and will likely remain so in
the future.  While it may not be important in some environments, it's
critical in others.  We should strive to do the best job we can and
Fusil makes it easier to be more robust.

> Sometimes, even if I write a patch, an unit test, explain the problem and the
> solution, I don't get any comment. It doesn't motivate me to continue fuzzing
> :-/

Yeah, I know this can be frustrating.  As Terry mentioned later in
this post, many of your patches have been committed which is very
impressive.  The outstanding ones are at most about a month old.  I
would expect most of those to move along as people find the time to
address them.

If you keep creating more high quality patches (e.g., with unit
tests), you could be on your way to becoming a Python committer which
would help solve this problem.

I definitely hope you continue to find and fix problems in Python.  It
helps everyone who uses Python even those who will never know to thank
you.  Who knows, someone might even write a book about Fusil someday
about a topic as obscure as Beautiful Testing. :-)

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Neal Norwitz]

On Wed, Jan 27, 2010 at 11:58 AM, Ben Finney  wrote:
> Chris Bergstresser  writes:
>
>> On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney  
>> wrote:
>> > Neal Norwitz  writes:
>> >> Who knows, someone might even write a book about Fusil someday
>> >> about a topic as obscure as Beautiful Testing. :-)
>> >
>> > Your suggested title is already taken, though, for exactly this
>> > purpose. The book “Beautiful Testing”, published by O'Reilly, might
>> > help http://oreilly.com/catalog/9780596159825>.
>>
>>    I suspect Neal already knows that, since he cowrote chapter 9
>> "Beautiful is Better than Ugly".
>
> Ah, good.
>
> I did suspect that might be the case, but couldn't easily find Neal's
> name associated with the book. I decided to post anyway, preferring the
> information to be out there at the risk of seeming to miss the joke.
>
> Maybe one day I'll get to read the book :-

Ben, thanks.  Your comment gave me a great laugh and I really
appreciated it. :-)

The chapter is about the general Python development process, including
testing, static analysis, dynamic analysis, including Fusil among
other things.

One of my points to Victor and everyone else like him is that even
though it may seem no one is listening to you or cares, you might be
surprised to find out how many people really are paying attention and
do care.  If you have something you want to do, make it happen.  One
person really can do amazing things.  You may never get recognized for
much of what you do, that doesn't make it any less important.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unary minus bug

[Neal Norwitz]

Do we care about this (after your checkin and with my fix to make
32-63 bit values ints rather than longs):

# 64 bit box
>>> minint = str(-sys.maxint - 1)
>>> minint
'-9223372036854775808'
>>> eval(minint)
-9223372036854775808
>>> eval('-(%s)' % minint[1:])
-9223372036854775808L

n
--
On 7/9/06, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
> The bug was reported by Armin in SF #1333982:
>
> the literal -2147483648 (i.e. the value of -sys.maxint-1) gives
> a long in 2.5, but an int in <= 2.4.
>
> I have a fix but I wonder if it's the right thing to do.  I suppose
> returning a long has the chance of breaking someone code.  Here's
> the test we currently have for a related case:
>
> [...]
> # 32-bit machine
> all_one_bits = '0x'
> self.assertEqual(eval(all_one_bits), 4294967295L)
> self.assertEqual(eval("-" + all_one_bits), -4294967295L)
>
> I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int))
> to that set of tests.
>
>   Neil
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unary minus bug

[Neal Norwitz]

On 7/9/06, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
> On Sun, Jul 09, 2006 at 03:02:06PM -0700, Neal Norwitz wrote:
> > Do we care about this (after your checkin and with my fix to make
> > 32-63 bit values ints rather than longs):
> >
> > # 64 bit box
> > >>>minint = str(-sys.maxint - 1)
> > >>>minint
> > '-9223372036854775808'
> > >>>eval(minint)
> > -9223372036854775808
> > >>>eval('-(%s)' % minint[1:])
> > -9223372036854775808L
>
> I don't think we care.  Python <= 2.4 has the same behavior (i.e.
> adding parens defeats the constant folding since it's done at the
> syntax tree level).

Ok, unless someone care's enough to make a patch, I'm removing this
from the outstanding issues for 2.5.

> Where's your fix regarding 32-63 bit ints?  I'm not familiar with

I just had to finish the test.  It's checked in now and should fix the
buildbot problems.  Your test exposed the issue, but didn't cause it.
It was caused
by the optimization of converting strings to ints.  My approach to fix it was
kinda hacky (just add the proper table for 64-bit machines).  I guess
ideally we would generate this table based off sizeof(long).  Probably
should have added
a TODO.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Missing" 2.5 feature

[Neal Norwitz]

On 7/9/06, Tim Peters <[EMAIL PROTECTED]> wrote:
> Just to make life harder ;-), I should note that code, docs and tests
> for sys._current_frames() are done, on the tim-current_frames branch.
> All tests pass, and there are no leaks in the new code.  It's just a
> NEWS blurb away from being just another hectic release memory :-)

There hasn't been much positive response (in the original thread or
here).  Given you forgot about it for over a year, how important can
it be? :-)

What's the justifcation to add this feature, but none of the others?
Can we defer this to 2.6?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] xml issue in 2.5

[Neal Norwitz]

http://python.org/sf/1513611
xml.sax.ParseException weirdness in python 2.5b1.  The following code
doesn't work:

from xml.sax import make_parser, SAXParseException

parser = make_parser()
try:
parser.parse(StringIO('invalid'))
except SAXParseException:
print 'caught it!'

Any comments?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)

[Neal Norwitz]

On 7/10/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> Jeremy Hylton wrote:
>
> > To express this email in the positive form:
> > 1. Reserved words should be real words.
> > 2. The meaning of the word should be clear.
> > 3. "Put statements in positive form."  (Strunk & White)
> > 4. The word should sound good.
>
> agreed.  a word should describe what a thing is, not what it isn't.

Just looking at a thesaurus, came up with expose.  Not to mention
outward, remote, peripheral and without.  We already have a 'with'
keyword, why not 'without'?  Oh, I guess it fails #3...and #2...and #4
too.  Screw it, just go with alien. :-)

http://thesaurus.reference.com/browse/outer

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] User's complaints

[Neal Norwitz]

On 7/12/06, Armin Rigo <[EMAIL PROTECTED]> wrote:
>
> Only two breakages is certainly nice, and I know that we all try quite
> hard to minimize that; that's probably still two breakages too much.

I agree, but some of this responsibility has to fall to users.
Sometimes these breakages are bugs, pure and simple.  Our tests don't
catch everything.  This is why it's really, really important to get as
many alpha/beta testers as possible.  Had the issues been brought up
before 2.4 was released, we could have addressed them.

Basically, if you care about these things, test before we release.
That way the issues can be addressed. If you don't care enough to test
and get bitten by a problem, quit whining, you had your chance.  I
have no sympathy for someone who is only willing to whine after the
fact, but unwilling to do any work.  That's really not helpful.

n
--
PS  People do this all the time in politics too.  They complain about
someone, when they couldn't even be bothered to vote.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] User's complaints

[Neal Norwitz]

On 7/12/06, Jeroen Ruigrok van der Werven <[EMAIL PROTECTED]> wrote:
> On 7/5/06, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> > For example, we heard grumblings about the releases coming too often.
> > Once we went to an 18 month release schedule, there was minimal
> > complaining.  It should be fairly safe to assume this silence means
> > people think we are doing a good job.  What are the things that could
> > be fixed that would silence the most number of user's complaints?
>
> I frequent a lot of IRC channels either devoted to Python or heaving
> people use Python a lot next to my own use so I think I can,
> hopefully, give some comments. Of course, things are highly
> subjective.

Jeroen,

Thank you very much for your feedback.  It helps.

> The release cycle is not bothering people from what I gather,
> especially not given the fact that the world is still changing when it
> comes to things like XML. What is bothering people is how you have to
> reinstall all your site-packages when you go from one major version to
> another. I understand this might be a difficult problem to tackle, but
> I can also understand the hassle if people have more than 10-15
> modules installed.

If it's pure python, why don't people just copy everything under
site-packages after installing?  They could/should run compileall
after that to recompile the .pyc files.  With 2.5 on 64-bit machines,
C extension modules *must* be recompiled due to lots of internal
changes.

One thing you didn't mention that I've heard from time to time is the
stdlib should be improved.  For example, cleaning up old modules.
Though I'm not really sure everyone has the same thing in mind when it
comes to improving the stdlib.

> Another point is how the module documentation is very terse in a lot
> of areas. Especially when it concerns what kind of exceptions a
> particular function can/will raise. As a professional, part-time,
> technical writer I wonder how to best tackle the entirety of the
> documentation we have. (Note that this is in no way any jibe towards
> Fred Drake, since I bet it's quite a bulk of documentation to work
> with/on every time.)
> Also it can get quite confusing when you need the introductory manual,
> when the language reference and when the module documentation.

Do you think you could help with the doc?  How can we get people,
especially tech writers, interested in improving the doc?  Most people
agree it's important, but few make time to really improve the doc.
We've talked about making it easier for people to contribute to the
docs, perhaps adding something like a wiki/comments.  Do you think
that would help?

> Things that struck me as peculiar is the old:
>
> if __name__ == "__main__":
> whatever()
>
> This is so out of tune with the rest of python it becomes a nuisance.

I'm not sure I understand your point.  Can you provide more info about
what you dislike/expect instead?

> I'll ask around and see what people are reporting to me as their top 3
> or 5 Python complaints though.

Great.  Thanks!

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] User's complaints

[Neal Norwitz]

On 7/13/06, Jeroen Ruigrok van der Werven <[EMAIL PROTECTED]> wrote:
>
> > If it's pure python, why don't people just copy everything under
> > site-packages after installing?  They could/should run compileall
> > after that to recompile the .pyc files.  With 2.5 on 64-bit machines,
> > C extension modules *must* be recompiled due to lots of internal
> > changes.
>
> I wasn't even aware of the compileall step, can you elaborate since
> this is the first time I see it being mentioned.

python -mcompileall -h

It's in the stdlib.  It is used by the build process to recursively
traverse directories and compile all the .py files into .pyc (or .pyo)
files.  So if you wanted to copy everything from site-package in 2.4
to 2.5, you could do something like:

cp -r /usr/lib/python2.{4,5}/site-packages
/usr/bin/python2.5 -mcompileall -d /usr/lib/python2.5/site-packages

That should really be all that's required. It of course doesn't verify
the packages are correct or that you still want to keep all the files.
 This generally works for .so's too.  However a warning will be
generated each time you import the .so since it was built on an old
version of Python..  On 2.5 in 64-bit systems, this will definitely
not work.

> > One thing you didn't mention that I've heard from time to time is the
> > stdlib should be improved.  For example, cleaning up old modules.
> > Though I'm not really sure everyone has the same thing in mind when it
> > comes to improving the stdlib.
>
> How do you envision cleaning up old modules?

For me, manually.  Do you still need moduleX?  There's no way for a
computer to tell you the answer, it depends on what you will use with
the new version of Python.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[Neal Norwitz]

On 7/13/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> Fred> It feels like the release cycle from alpha1 to final has gotten
> Fred> increasingly rushed.

I think that's just because you are getting older and time goes by
faster the less time you've got left. :-)  It seems to be going quite
quick, but I'm not so sure it's really different.

> Same here.  I believe there was some shortening of the 2.5 release cycle two
> or three months ago.  I don't recall why or by how much, but I think the
> acceleration has resulted in a lot of the "can't we please squeeze this one
> little change in?" that's been happening.  Shortening a micro release a bit
> seems reasonably easy to accommodate, but since minor releases occur so
> infrequently, I think it would be better to stretch them out if necessary.

Not exactly.  The PEP was created Feb 7 (r42266) with a very relaxed
schedule to allow enough time for ssize_t branch to solidify.  Martin
was ready sooner, so on Feb 26 (r42577) sped up the schedule to
something more reasonable:

-alpha 1: April 1, 2006 [planned]
-alpha 2: April 29, 2006 [planned]
-beta 1:  June 24, 2006 [planned]
-beta 2:  July 15, 2006 [planned]
-rc 1:August 5, 2006 [planned]
-final:   August 19, 2006 [planned]

The current schedule is:

+alpha 1: April 5, 2006 [completed]
+alpha 2: April 27, 2006 [completed]
+beta 1:  June 20, 2006 [completed]
+beta 2:  July 11, 2006 [completed]
+rc 1:August 1, 2006 [planned]
+final:   August 8, 2006 [planned]

So beta1 was 4 days earlier than the date set at the end of Feb.  The
first beta was  about 19 months after 2.4 was released if I'm counting
correctly.  In 2.4 beta1 to release was 1.5 months.  For 2.5 it is
about the same.  The total 2.4 schedule (from a1 to release) was 4.5
months which included 3 alphas.  For 2.5 we only had 2 alphas and the
total schedule is just over 4 months.  So 2.5 is just a little faster.
 But 2.5 has had a ton more testing than any previous released due to
the buildbots, not to mention Coverity and now Klocwork.

I definitely pushed the schedule, but I don't think it was really all
that different from in the past.

Given that several people here think we should lengthen the schedule
in some way, I suspect we will do something.  I'm not really against
it, but I don't think it will provide much benefit either.  A few more
bugs will be fixed since we have more time.  What I think might really
make a difference if we could leverage the buildbots and copy the
builds (especially WIndows) up to python.org and make them available
for download.  That might make it easier for people to try stuff out.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[Neal Norwitz]

On 7/13/06, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
> On 7/13/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> > On Thu, 13 Jul 2006 11:29:16 -0700, Aahz <[EMAIL PROTECTED]> wrote:
> >
> > >There's been some recent discussion in the PSF wondering where it would
> > >make sense to throw some money to remove grit in the wheels; do you think
> > >this is a case where that would help?
> >
> > Most likely yes.  It's not a huge undertaking, and there are a lot of 
> > people out
> > there in the community with the knowledge of Buildbot to make this happen.
>
> I'm at least willing to set up the buildbots for projects I care about
> (Twisted, pydoctor, whatever), and perhaps help out with the setting
> up the buildmaster.

It would be really great to have someone setup buildbots for other
important/large python projects to test python svn HEAD with.  I can't
even stay on top of the python buildbots, so I'm not volunteering for
more work.  I will greatly appreciate and thank whoever takes this on
though.  :-)

The harder problem is keeping it running.  Setting it up, in some ways
is a lot easier.  It's at least well defined.  Maintaining it for
several months is a different story.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[Neal Norwitz]

On 7/13/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> Neal Norwitz wrote:
>
> > Given that several people here think we should lengthen the schedule
> > in some way, I suspect we will do something.  I'm not really against
> > it, but I don't think it will provide much benefit either.  A few more
> > bugs will be fixed since we have more time.
>
> you're still missing the point of this thread: it isn't bugs in the core
> Python distribution that's the problem, it's compatibility with third-
> party modules and applications.

...

> a longer beta period gives *external* developers more time to catch up,
> and results in less work for the end users.

This is the part I don't get.  For the external developers, if they
care about compatibility, why aren't they testing periodically,
regardless of alpha/beta releases?  How often is the python build
broken or otherwise unusable?  On Unix there's no more or less work to
grab a tarball whether it's a nightly or a release.  For Windows it's
definitely easier to wait for a release.  If there was an installable
windows version made by the buildbots, would that mean there would be
more testing?

Is part of your point that these developers only care about something
called "release" and they won't start testing before then?  If that's
the case why don't we start making (semi-)automated alpha releases
every month?  That will give people lots more time.  Somehow I don't
think it will make a difference.  People mostly work on schedules or
should I say avoid schedules.  Give them 5 months and they will still
wait to the last minute.

Remember I also tried to push for more features to go in early?  That
would have given more time for external testing.  Still features are
coming in.  Python developers weren't happy about having to get things
in earlier.  I don't see a practical way to implement what you propose
(see Anthony's comments).

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[Neal Norwitz]

On 7/15/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Terry Reedy wrote:
> > That is the goal, but when I watched the buildbot results last spring, the
> > degree of stability (greenness) appeared to vary.  Is it possible to tag
> > particular versions as a 'green' version, or the 'most recent green
> > version' worth playing with?
>
> Don't get confused by these colors. The tree compiled nearly all of the
> time, even if some tests were failing for an extended period of time on
> some platforms.

Right.  It's very rare that the interpreter or stdlib is broken.
There are 19 buildbots currently.  7 are not green.  5 of those are
offline, most with known (to me and the person that donated the
machines) hardware issues ranging from lack of air conditioning, to a
bad router, to no power.  1 is cygwin which is running an old version.
 I just (an hour or so ago) got an account on a cygwin box to help
diagnose the status and figure out if anything within Python or the
tests are broken.  That leaves 1 unexplained failure on a Windows bot.
 This started happening recently after being down for a while.  I
haven't had time to investigate.

The reason why it was not very green in spring was because that's when
we were getting it set up!  The master looks like it was installed at
the end of 2005/beginning of 2006.  It took several months to get rid
of many testing issues.  Tests that couldn't be run in a particular
order, tests that couldn't be run simultaneously (socket), bad
compilation of sqlite/bsddb modules (not in python), misconfigured
systems, tests verifying something that was system dependent, signed
addresses, etc.

Of all the problems there were, I only remember a single problem in
Python.  (There were probably more, I just remember one.)  That was in
test code (xxmodule or something like that.  There were a bunch of
problems with ctypes and/or sqlite when they got integrated having to
deal with these new platforms.  That may be what you are recalling.
Right before alpha 2 was a particularly bad time.

What we mean by stable is that at any time/any stage of development,
if a change goes in that breaks the tests, it is likely to be fixed or
reverted within hours.  The amount of time the build or tests are
broken on the majority of platforms is quite small.  It took a while
to get to this point due to a bunch of flaky tests, but those have
mostly been fixed.  The only known problems are mentioned on the wiki.

When the buildbots fail, we get mail to python-checkins.
Unfortunately, that's mostly spam at this point.  I hope to fix that
at some point.  I also hope to change the main page to give more info
in less space.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots -- reprise

[Neal Norwitz]

I have a server up and running.  I still need to polish some stuff
off.  I will mail more info when I get a chance.

n
--

On 7/21/06, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
> On Fri, 21 Jul 2006 10:04:38 -0700, Grig Gheorghiu <[EMAIL PROTECTED]> wrote:
> >Hi,
> >
> >Apart from the goals stated by Glyph, I see this as a very valuable
> >effort in convincing people of the value of automated tests,
> >Python-related or not. A secondary effect I'd like to see would be for
> >these suites of tests to be invoked in a standard fashion -- maybe
> >'python setup.py test'.
> >
> >If PSF can contribute some $$$ towards the hosting of the master
> >server, that would be appreciated, but not required. All that's
> >required is enough interest from the community.
> >
> >Please let me know if you're interested.
> >
>
> This is certainly interesting to me.  If you need any help setting up
> the Twisted buildslave, please let me know.
>
> Jean-Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] outstanding bugs to fix for 2.5

[Neal Norwitz]

There are still a bunch of outstanding bugs.  rc1 is about a week away
and it would be great to fix these.  Many of these are also present in
2.4, but it would be nice to squash them in 2.5.  Here's the list from
PEP 356:

http://python.org/sf/1526585 - SystemError concat long strings (2.4)
http://python.org/sf/1523610 - PyArg_ParseTupleAndKeywords potential
core dump (2.4)
http://python.org/sf/1521947 - mystrtol.c fails with gcc 4.1 (2.4?)
test_compile.test_unary_minus
http://python.org/sf/1519025 - socket timeout crash when receive
signal (2.4)
http://python.org/sf/1517042 - Fix crashers/gc_inspection.py (2.4)
http://python.org/sf/1515471 - stringobject (char buffers)
http://python.org/sf/1513611 - XML: xml.sax.expatreader missing
http://python.org/sf/1511497 - XML: xml.sax.ParseException issue
http://python.org/sf/1475523 - gettext.py bug
http://python.org/sf/1467929 - %-formatting and dicts (2.4)
http://python.org/sf/1333982 - AST
http://python.org/sf/1191458 - AST (test_trace issue mentioned below)

It would be great to fix *all* of these.  In this list, at least 3
(4?) can cause segfaults, and #1521947 can cause incorrect results.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] remaining issues from Klocwork static analysis

[Neal Norwitz]

I've fixed most of the problems (or determined they weren't problems)
from all the warnings issued by Klocwork's static analysis tool.  The
following are outstanding issues.

This first group looks like real problems to me:

# 74 Object/funcobject.c:143Suspicious deref of ptr before NULL check
#169 Modules/threadmodule.c:497 Memory Leak

# 28 Modules/_sre.c:987   Array Index Out of Bounds

Buffer overflow, array index of 'mark' may be outside the
bounds. Array 'mark' of size 200 declared at sre.h:77 may use
index values 0..536870911. Also there are 3 similar errors on
lines 1006, 1225, 1237.  (Try limiting mark on line 589?)

#174 Modules/unicodedata.c:432   Array Index Out of Bounds

Buffer overflow, array index of 'decomp_prefix' may be outside the
bounds. Array 'decomp_prefix' of size 18 declared at
unicodedata_db.h:529 may use index values 18..255. Also there is one
similar error on line 433.

# 36 Modules/cPickle.c:3404   Memory Leak

Memory leak. Dynamic memory stored in 's' allocated through
function 'pystrndup' at line 3384 is lost at line 3404.

s should not be freed on line 3407, but earlier.
PDATA_PUSH can return on error and s will not be freed.

# 61 Modules/_sqlite/cursor.c:599  Null pointer may be dereferenced

Null pointer 'self->statement' that comes from line 674 may be
dereferenced by passing argument 1 to function
'statement_mark_dirty' at line 599.

Most of these seem suspect.  I'm not so sure about them, but I haven't
looked into some at all.  Let me know if you want the details for any
of these or if you can provide an analysis to demonstrate they are
incorrect.

Null pointer may be dereferencedPython/ast.c:641
Null pointer may be dereferencedPython/ast.c:656
Ptr will be derefed after it was positively checked for
NULLPython/compile.c:3020
Null pointer may be passed to function that may dereference
it  Python/compile.c:4459
Array Index Out of Bounds   Modules/_sre.c:987
Array Index Out of Bounds   Object/longobject.c:1787
Array Index Out of Bounds   Object/longobject.c:2475
Array Index Out of Bounds   Python/sysmodule.c:1016

Array Index Out of Bounds   Python/getpath.c:285
Buffer Overflow - Non-null Terminated StringPython/getpath.c:432
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String  Python/getpath.c:431
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String  Python/getpath.c:496
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String  Python/getpath.c:497

Let me know if you want more info about any particular report.  It
would be great to have some help and fix these.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots -- reprise

[Neal Norwitz]

If you want I can send you the build master cfg I setup on python.org
and some simple instructions for how to connect to it.  I don't have
time to focus on this at the moment and probably won't until 2.5 is
out.

n
--

On 7/20/06, Grig Gheorghiu <[EMAIL PROTECTED]> wrote:
> Hi,
>
> This message is in response to Glyph's plea
> ().
>
> Here's what Glyph said:
>
> "I would like to propose, although I certainly don't have time to
> implement, a program by which Python-using projects could contribute
> buildslaves which would run their projects' tests with the latest
> Python trunk.  This would provide two useful incentives: Python code
> would gain a reputation as generally well-tested (since there is a
> direct incentive to write tests for your project: get notified when
> core python changes might break it), and the core developers would have
> instant feedback when a "small" change breaks more code than it was
> expected to."
>
>
> I'm volunteering to organize this effort, is there is enough interest
> on this list. In fact, I've done some prep work already:
>
>  * got a domain name: pybots.org
>  * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access
> and everything); it's available at master.pybots.org, and it's ready to
> be configured as a buildmaster for the pybots
>  * got a mailing list: [EMAIL PROTECTED]
>
> I can start configuring the Ubuntu machine as a buildmaster, and I can
> also add a buildslave on the same machine that will check out the
> latest Python trunk code, build it, then run the automated tests for a
> sample project -- let's say for Twisted, since Glyph was the one
> requesting this. This will also serve as a sample buildslave for other
> people who will be interested in running buildslaves for their own
> projects.
>
> Apart from the goals stated by Glyph, I see this as a very valuable
> effort in convincing people of the value of automated tests,
> Python-related or not. A secondary effect I'd like to see would be for
> these suites of tests to be invoked in a standard fashion -- maybe
> 'python setup.py test'.
>
> If PSF can contribute some $$$ towards the hosting of the master
> server, that would be appreciated, but not required. All that's
> required is enough interest from the community.
>
> Please let me know if you're interested.
>
> Grig
>
> 
> http://agiletesting.blogspot.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] remaining issues from Klocwork static analysis

[Neal Norwitz]

On 7/25/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz wrote:
> > # 74 Object/funcobject.c:143Suspicious deref of ptr before NULL check
>
> Not quite sure what it is complaining about, but
>
> else if (PyTuple_Check(closure)) {
> Py_XINCREF(closure);
> }
>
> looks indeed suspicious: Why do we check for NULL (XINCREF) when
> we know closure can't be NULL (Tuple_Check). Drop the X, and see
> if the warning goes away

Yes, I definitely think dropping the X would make the warning go away.
 Do we want to check for a NULL pointer and raise an exception?  The
docs don't address the issue, so I think if we added a check, ie:  if
(closure && PyTuple_Check(closure)) and got rid of the X that would be
fine as well.

> > #169 Modules/threadmodule.c:497 Memory Leak
>
> Does it say what memory is leaking? Perhaps it complains about
> boot not being released if ident is not -1, however, in that case,
> t_bootstrap will release the memory.

I believe you are right, I never traced through t_bootstrap.   I think
this is a false positive.  There is some memory being leaked on thread
creation as reported by valgrind IIRC.  This doesn't seem to be it
though.

> > #174 Modules/unicodedata.c:432   Array Index Out of Bounds
> >
> > Buffer overflow, array index of 'decomp_prefix' may be outside the
> > bounds. Array 'decomp_prefix' of size 18 declared at
> > unicodedata_db.h:529 may use index values 18..255. Also there is one
> > similar error on line 433.
>
> This limit is enforced by Tools/unicode/makeunicodedata.py. There are
> only 18 decomposition prefixes at the moment, yet we use 8 bits for
> the decomposition prefix (makeunicodedata checks that prefix < 256)

Just to make sure I understand.  The code in question is accessing
decomp_prefix like this:
decomp_prefix[decomp_data[index] & 255]

So decomp_prefix will be accessed with the result of:
decomp_data[index] & 255

The first line of data is (fro unicodedata_db.h) is:

static unsigned int decomp_data[] = {
0, 257, 32, 514, 32, 776, 259, 97, 514, 32, 772, 259, 50, 259, 51, 514,

If index == 2 (or 3-5, 7-10, etc), we have:
decomp_prefix[decomp_data[2] & 255]
decomp_prefix[32 & 255]
decomp_prefix[32]

which is larger than the max size of decomp_prefix (18).  But from
what I think you stated above, index can't equal those values and the
code that prevents it is calculated a few lines above:

index = decomp_index1[(code>>DECOMP_SHIFT)];
index = decomp_index2[(index<http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] remaining issues from Klocwork static analysis

[Neal Norwitz]

On 7/25/06, Georg Brandl <[EMAIL PROTECTED]> wrote:
> Martin v. Löwis wrote:
> > Neal Norwitz wrote:
> >> # 74 Object/funcobject.c:143Suspicious deref of ptr before NULL check
> >
> > Not quite sure what it is complaining about, but
> >
> > else if (PyTuple_Check(closure)) {
> > Py_XINCREF(closure);
> > }
> >
> > looks indeed suspicious: Why do we check for NULL (XINCREF) when
> > we know closure can't be NULL (Tuple_Check). Drop the X, and see
> > if the warning goes away
>
> In comparison, the PyFunction_SetDefaults function does check for
> NULL, and raises an error in this case. However, since it is a C API function
> only, passing NULL is an error anyway.

Heh, that was me that added it 10 days ago. :-)
Might as well do the same here.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] remaining issues from Klocwork static analysis

[Neal Norwitz]

On 7/25/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> >
> > Yes, I definitely think dropping the X would make the warning go away.
> > Do we want to check for a NULL pointer and raise an exception?  The
> > docs don't address the issue, so I think if we added a check, ie:  if
> > (closure && PyTuple_Check(closure)) and got rid of the X that would be
> > fine as well.
>
> The docs do address the issue:
>
> \var{closure} must be \var{Py_None} or a tuple of cell objects.
>
> It doesn't allow for NULL, and None indicates that the closure
> should become NULL. The only caller of it in the core will never
> pass NULL.
>
> If you want to check that this is not NULL on the grounds that
> somebody may call it incorrectly, then you should also check that
> op is not NULL, because somebody may call it incorrectly.

We never really did address this issue did?  A while back we talked
about whether to assert vs check and do PyErr_BadInternalCall().  I
don't remember a clear resolution (though my memory).  I vaguely
remember a preference towards asserting, but I don't know if that was
in all cases or maybe it was just my preference. :-)

I'm happy to assert here too.  But it's really a broader question.  I
guess I'm even happy to just remove the X.  It would be nice to handle
this consistently going forward.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] remaining issues from Klocwork static analysis

[Neal Norwitz]

On 7/25/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz wrote:
> > We never really did address this issue did?  A while back we talked
> > about whether to assert vs check and do PyErr_BadInternalCall().  I
> > don't remember a clear resolution (though my memory).  I vaguely
> > remember a preference towards asserting, but I don't know if that was
> > in all cases or maybe it was just my preference. :-)
> >
> > I'm happy to assert here too.  But it's really a broader question.  I
> > guess I'm even happy to just remove the X.  It would be nice to handle
> > this consistently going forward.
>
> I would just remove the X.

I'll do that here since it's the easiest.

> If we want to handle it consistently, we would have to check all pointer
> parameters in all functions; this would be a huge task (and for little
> value, IMO).

I'm not suggesting changing existing code, unless we find issues.  I
agree that it would be a huge task and of little value.  I was
thinking about for future code.  I guess we aren't writing a lot of
new C APIs in 2.x, so it really doesn't much there matter.  Though for
3k, it would be nice to make it consistent as new APIs are written or
old APIs are cleaned up.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Windows, buildbot] kill_python.c mystery

[Neal Norwitz]

On 7/26/06, Tim Peters <[EMAIL PROTECTED]> wrote:
>
> Today I noticed this happened when the buildbot started to run tests,
> and I'm 100% sure it's due to this code in
> Tools/buildbot/kill_python.c (the buildbot log files showed that
> kill_python.c killed /some/ Python process, and the Python running
> release-build tests in my sandbox was the only plausible candidate):
>
> if ((strstr(path, "build\\pcbuild\\python_d.exe") != NULL) ||
> (strstr(path, "build\\python.exe") != NULL)) {
> printf("Terminating %s (pid %d)\n", path, pids[i]);
> if (!TerminateProcess(hProcess, 1)) {
>
> The second clause in the first `if` looks for a substring match on:
>
> build\python.exe
>
> and that just happens to match a suffix of:
>
> C:\Code\python\PCbuild\python.exe
>
> which is the release-build Python I happen to be running in my sandbox.
>
> Why is the second clause there?  That is, are we /trying/ to kill a
> release-build Python running from the user's sandbox, and if so why?

No, I don't believe that was the intent.  The exe on cygwin uses the
unix convention, not the Windows convention for the filename.  ie,
either a debug
or release build on cygwin are both called python.exe.

So the second clause is there to kill the process when it's running
under cygwin.
It's interesting that the process appears to be running as
./python.exe, but build shows up in filename.  From that I deduce that
it must contain the complete path.  Assuming that is true, we can
change the code to ensure that build is a directory since that's what
buildbot does (add the leading \\):

> (strstr(path, "\\build\\python.exe") != NULL)) {

I tested this change with a different path, so I believe it will work
fine and not catch PCbuild.  I'll check in this change and add some
comments.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix

[Neal Norwitz]

What is the behaviour that was added which broke compliance?  What is
the benefit of the behaviour?

>From your description of fixing the problem, it seems there's some
risk invovled as it's modiyfing import.c, plus adding new features.
What is your recommendation?

n
--

On 7/26/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> I posted last week about a need-for-speed patch that broke PEP 302
> compliance, and asked if it should be fixed or reverted.  I got exactly one
> response which said "yes, it should be fixed or reverted", which
> unfortunately didn't answer my question as to which one we should do.  :)
>
> If we don't revert it, there are two ways to fix it.  One is to just change
> PEP 302 so that the behavior is unbroken by definition.  :)  The other is
> to actually go ahead and fix it by adding PathImporter and NullImporter
> types to import.c, along with a factory function on sys.path_hooks to
> create them.  (This would've been the PEP-compliant way to implement the
> need-for-speed patch.)
>
> So, "fix" by documentation, fix by fixing, or fix by reverting?  Which
> should it be?
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix

[Neal Norwitz]

On 7/27/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
>
> Personally, I would prefer to see it properly fixed in 2.5 rather than
> having to rip it out.  It's more work for me to create the proper fix than
> it is to just work around it in my code, but it seems a more righteous
> labor, if you know what I mean.  It also means that already-shipped and
> distributed versions of my code would work with the 2.5 release.

Based on this comment, is it really acceptable to just document a
behaviour change?  ISTM there should really only be 2 choices:  fix
2.5 properly or revert the change.  This seemed to be Armin's
position.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] uuid test suite failing

[Neal Norwitz]

On 7/27/06, A.M. Kuchling <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote:
> > The UUID test suite, which wasn't run by regrtest.py until now, is
> > now failing on some buildbots (and my machine). This should be fixed
> > before releasing something.
>
> Looking at the failures, there seem to be two problems on Unix variants:
>  1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig -a'
> to print information about all interfaces.
>  2) on Solaris 9 (the only version in the SF compile farm), I can't
> figure out how to make ifconfig print MAC addresses at all.
> Searching online finds the incantation 'arp ' to print the
> MAC.

This is such a mess.  There are so many different ways of determining
the MAC addr on each flavour of Unix it seems hopeless to try.  I
fixed _ifconfig_getnode so it should work on at least:  Linux, Tru64,
Solaris, and HP-UX.  Who knows how many more variations there are.

This only fixes 1 of the 2 failures in test_uuid.  The other one is
due to _unixdll_getnode() failing.  This is because
_uuid_generate_time is None because we couldn't find it in the uuid
library.  This is just broken, not sure if it's the code or the test
though.  We should handle the case if _uuid_generate_time and the
others are None better.  I don't know what to do in this case.

Since getnode ignores exceptions, maybe it's the test that is broken?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another uuid problem

[Neal Norwitz]

It checks for ifconfig, /sbin/ifconfig, and /usr/sbin/ifconfig (same
for arp).  The problem is the os.pipe command doesn't hide these
issues.  It doesn't cause the test to fail, but is still broken.  The
test is presumably failing for the other reason I mentioned
(unixdll_getnode).  Let me know if you see that when running with -v.

n
--

On 7/28/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> I just tried building and testing 2.5 on a Solaris 10 box at work.  The uuid
> test failed with
>
> sh: ifconfig: not found
> sh: ifconfig: not found
> sh: ifconfig: not found
> sh: arp: not found
> sh: ifconfig: not found
> sh: ifconfig: not found
> sh: ifconfig: not found
> sh: arp: not found
>
> In our environment at least it's uncommon for /usr/sbin to be in the PATH of
> non-privileged users:
>
> piggy:% type -a ifconfig
> -bash: type: ifconfig: not found
> piggy:% PATH=$PATH:/usr/sbin type -a ifconfig
> ifconfig is /usr/sbin/ifconfig
>
> Perhaps test_uuid needs to do a little investigation to find ifconfig and
> arp.
>
> Skip
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py2.5 release schedule

[Neal Norwitz]

On 7/28/06, Michael Hudson <[EMAIL PROTECTED]> wrote:
> Anthony Baxter <[EMAIL PROTECTED]> writes:
>
> > Does anyone disagree with making the next release beta3?
>
> It seems like a good idea to me.  I guess this will mean the final
> release will be pushed back a bit?

Anthony and I talked about still having b3 on Aug 1.  rc1 around Aug
17-18 (just before the Google sprint which Martin, Jeremy and I will
be attending).  Final around 24-29.  We didn't discuss with Martin
yet, so these dates are quite tentative.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] test_uuid

[Neal Norwitz]

Ping,

I just checked in a change to disable testing 2 uuid functions
(_ifconfig_get_node and unixdll_getnode) that fail on many platforms.
Here's the message:

"""
Disable these tests until they are reliable across platforms. These
problems may mask more important, real problems.

One or both methods are known to fail on: Solaris, OpenBSD, Debian, Ubuntu.
They pass on Windows and some Linux boxes.
"""

Can you fix these issues or at least provide guidance how they should be fixed?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] internal weakref API should be Py_ssize_t?

[Neal Norwitz]

I'm wondering if the following change should be made to Include/weakrefobject.h:

-PyAPI_FUNC(long) _PyWeakref_GetWeakrefCount(PyWeakReference *head);
+PyAPI_FUNC(Py_ssize_t) _PyWeakref_GetWeakrefCount(PyWeakReference *head);

And the 2 other files which use this (weakref obj and module).  Should
this be changed now, wait for 2.6, or not worry about it?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TRUNK FREEZE 2006-07-03, 00:00 UTC for 2.5b3

[Neal Norwitz]

This looks like it needs to be fixed as it's a regression.  As Skip
said, bug fixes are allowed.  There will still be at least one release
candidate.  It's looking like c1 will be around Aug 18 and final
around Sept 12.  I'll update the PEP when I get a chance and confirm
all the dates.

n
--

On 8/2/06, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
> On Wed, 2 Aug 2006 22:29:44 +1000, Anthony Baxter <[EMAIL PROTECTED]> wrote:
> >The trunk is FROZEN from 00:00 UTC/GMT on 03 July 2006. That's in about 12
> >hours from now. This is for the third (and final, hopefully!) beta of 2.5.
> >
> >Please help the release team out by not making checkins on the trunk after
> >that time until I send out an email that the release is done.
>
> Does this mean that there is no further opportunity to resolve regressions
> such as 
> ,
>  or will changes be allowed between b3 and final?
>
> Jean-Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.5 status

[Neal Norwitz]

Things are getting better, but we still have some really important
outstanding issues.  PLEASE CONTINUE TESTING AS MUCH AS POSSIBLE.
Also, it would be great to use as many tools as possible to find bugs
and improve quality.  It would be especially nice to run Purify on
Windows.

I've updated PEP 356 with the current status.  The current schedule is:

rc 1:August 18, 2006 [planned]
final:   September 12, 2006 [planned]

This somewhat strange schedule is to accommodate availability of
people cutting the release.  A branch will be made when the release
candidate is done.  Don't try to rush bug fixes in.  Let's try to keep
things stable.  Keep the doc fixes coming.

I believe all 3 outstanding issues (and solutions!) could use some
more discussion.  All bugs/patches blocking release are set to
priority 9.

http://python.org/sf/1530559 - struct rejecting floats (patch pending)

http://mail.python.org/pipermail/python-dev/2006-July/067774.html
Problem with __index__ (patch pending)

http://mail.python.org/pipermail/python-dev/2006-August/067926.html
str/unicode dict keys can raise an exception

All the AST issues that I know about have been addressed.

There are various other issues that would be nice to fix for 2.5.
These are priority 8.  All exist in 2.4 and possibly earlier:

http://python.org/sf/1526585 - SystemError concat long strings
http://python.org/sf/1523610 - PyArg_ParseTupleAndKeywords
potential core dump
http://python.org/sf/1517042 - Fix crashers/gc_inspection.py
http://python.org/sf/1475523 - gettext.py bug
http://python.org/sf/1467929 - %-formatting and dicts

I will continue to use priorities.  Please review bugs and patches and
let me know if anything else should be re-prioritized.

I will continue to stress test Python and use various tools to keep
quality up.  The current suite includes in no particular order:
valgrind, Coverity, Klocwork, failmalloc, pychecker, not to mention
the buildbot testing on commits and refleak test runs every 12 hours.

Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5b3 and AIX 4.3 - It Works

[Neal Norwitz]

There is at least one outstanding bug report for test_mmap failing on
AIX IIRC.  Possibly another for test_resource.  Please review bug
reports and file new ones/update old ones with the current status.
Unless if you provide patches, they probably won't be fixed though.
No one has access to AIX AFAIK.

n
--

On 8/4/06, Michael Kent <[EMAIL PROTECTED]> wrote:
> Because of a requirement to remain compatible with AIX 4.3, I have been forced
> to stay with Python 2.3, because while 2.4 would compile under AIX 4.3, it 
> would
> segfault immediately when run.
>
> I'm happy to report that Python 2.5b3 compiles and runs fine under AIX 4.3, 
> and
> passes most of its test suite.  However, here are a few test failures.  I
> realize AIX 4.3 is long obsolete, so is there any interest on the list for me 
> to
> report which tests failed and how?
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is this a bug?

[Neal Norwitz]

On 8/9/06, Josiah Carlson <[EMAIL PROTECTED]> wrote:
> 2.4 performed these imports silently, while 2.5 complains "SystemError:
> Parent module 'x' not loaded", which is actually a useful message, and
> helped me fix it.

Can you make a small, self-contained test case?  The SystemError
should be a normal exception and might indicate a deeper problem.

Thanks,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fwd: [Python-checkins] r51236 - in python/trunk: Doc/api/abstract.tex Include/abstract.h Include/object.h Lib/test/test_index.py Misc/NEWS Modules/arraymodule.c Modules/mmapmodule.c Modul

[Neal Norwitz]

I checked in this fix for the __index__ clipping issue that's been
discussed.  This patch is an improvement, but still needs some work.
Please pull out any parts you have an issue with and suggest a patch
to address your concern.

n

-- Forwarded message --
From: neal.norwitz <[EMAIL PROTECTED]>
Date: Aug 12, 2006 10:03 AM
Subject: [Python-checkins] r51236 - in python/trunk:
Doc/api/abstract.tex Include/abstract.h Include/object.h
Lib/test/test_index.py Misc/NEWS Modules/arraymodule.c
Modules/mmapmodule.c Modules/operator.c Objects/abstract.c
Objects/classobject.c Objects/intobject.c Objects/listobject.c
Objects/longobject.c Objects/sliceobject.c Objects/stringobject.c
Objects/tupleobject.c Objects/typeobject.c Objects/unicodeobject.c
Python/ceval.c
To: [EMAIL PROTECTED]


Author: neal.norwitz
Date: Sat Aug 12 19:03:09 2006
New Revision: 51236

Modified:
   python/trunk/Doc/api/abstract.tex
   python/trunk/Include/abstract.h
   python/trunk/Include/object.h
   python/trunk/Lib/test/test_index.py
   python/trunk/Misc/NEWS
   python/trunk/Modules/arraymodule.c
   python/trunk/Modules/mmapmodule.c
   python/trunk/Modules/operator.c
   python/trunk/Objects/abstract.c
   python/trunk/Objects/classobject.c
   python/trunk/Objects/intobject.c
   python/trunk/Objects/listobject.c
   python/trunk/Objects/longobject.c
   python/trunk/Objects/sliceobject.c
   python/trunk/Objects/stringobject.c
   python/trunk/Objects/tupleobject.c
   python/trunk/Objects/typeobject.c
   python/trunk/Objects/unicodeobject.c
   python/trunk/Python/ceval.c
Log:
Patch #1538606, Patch to fix __index__() clipping.

I modified this patch some by fixing style, some error checking, and adding
XXX comments.  This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward.  I don't want this to hold up release if possible.



Modified: python/trunk/Doc/api/abstract.tex
==
--- python/trunk/Doc/api/abstract.tex   (original)
+++ python/trunk/Doc/api/abstract.tex   Sat Aug 12 19:03:09 2006
@@ -693,12 +693,31 @@
   \samp{float(\var{o})}.\bifuncindex{float}
 \end{cfuncdesc}

-\begin{cfuncdesc}{Py_ssize_t}{PyNumber_Index}{PyObject *o}
-  Returns the \var{o} converted to a Py_ssize_t integer on success, or
-  -1 with an exception raised on failure.
+\begin{cfuncdesc}{PyObject*}{PyNumber_Index}{PyObject *o}
+  Returns the \var{o} converted to a Python int or long on success or \NULL{}
+  with a TypeError exception raised on failure.
   \versionadded{2.5}
 \end{cfuncdesc}

+\begin{cfuncdesc}{Py_ssize_t}{PyNumber_AsSsize_t}{PyObject *o, PyObject *exc}
+  Returns \var{o} converted to a Py_ssize_t value if \var{o}
+  can be interpreted as an integer. If \var{o} can be converted to a Python
+  int or long but the attempt to convert to a Py_ssize_t value
+  would raise an \exception{OverflowError}, then the \var{exc} argument
+  is the type of exception that will be raised (usually \exception{IndexError}
+  or \exception{OverflowError}).  If \var{exc} is \NULL{}, then the exception
+  is cleared and the value is clipped to \var{PY_SSIZE_T_MIN}
+  for a negative integer or \var{PY_SSIZE_T_MAX} for a positive integer.
+  \versionadded{2.5}
+\end{cfuncdesc}
+
+\begin{cfuncdesc}{int}{PyIndex_Check}{PyObject *o}
+  Returns True if \var{o} is an index integer (has the nb_index slot of
+  the tp_as_number structure filled in).
+  \versionadded{2.5}
+\end{cfuncdesc}
+
+
 \section{Sequence Protocol \label{sequence}}

 \begin{cfuncdesc}{int}{PySequence_Check}{PyObject *o}

Modified: python/trunk/Include/abstract.h
==
--- python/trunk/Include/abstract.h (original)
+++ python/trunk/Include/abstract.h Sat Aug 12 19:03:09 2006
@@ -758,13 +758,27 @@

*/

- PyAPI_FUNC(Py_ssize_t) PyNumber_Index(PyObject *);
+#define PyIndex_Check(obj) \
+   ((obj)->ob_type->tp_as_number != NULL && \
+PyType_HasFeature((obj)->ob_type, Py_TPFLAGS_HAVE_INDEX) && \
+(obj)->ob_type->tp_as_number->nb_index != NULL)
+
+ PyAPI_FUNC(PyObject *) PyNumber_Index(PyObject *o);

/*
-Returns the object converted to Py_ssize_t on success
-or -1 with an error raised on failure.
+Returns the object converted to a Python long or int
+or NULL with an error raised on failure.
*/

+ PyAPI_FUNC(Py_ssize_t) PyNumber_AsSsize_t(PyObject *o, PyObject *exc);
+
+   /*
+Returns the object converted to Py_ssize_t by going through
+PyNumber_Index first.  If an overflow error occurs while
+converting the int-or-long to Py_ssize_t, then the second argument
+is the error-type to return.  If it is NULL, then the overflow error
+is cleared and the value is clipped.
+   */

  PyAPI_FUNC(PyObject *) PyNumber_Int(PyObject *o);


Modi

[Python-Dev] test_socketserver failure on cygwin

[Neal Norwitz]

I'm not sure if this is a race condition in the test, the test using
SocketServer inappropriately, or if the failure is exposing a problem
in SocketServer.

The issue is that when the second (UDP) server is setup, there is
still a child process from the first (TCP) server.  So when the UDP
server calls collect_children(), it gets back the pid from the TCP
server, which is not in the UDP server's list and an exception is
raised.  In general, this is a problem, if program is using a
ForkingServer and spawns it's own children, this problem can occur.
This is because os.waitpid() is called with a pid of 0, rather than
the individual pids in the list.  I suppose that would fix the
problem.  Is this worth fixing?  (It's a pain to handle properly
because there is also use of a blocking wait.)

http://www.python.org/dev/buildbot/all/x86%20cygwin%20trunk/builds/1107/step-test/0

Add this patch corrects the problem (see below for more details):

Index: Lib/test/test_socketserver.py
===
--- Lib/test/test_socketserver.py   (revision 51260)
+++ Lib/test/test_socketserver.py   (working copy)
@@ -183,7 +183,11 @@

 def testall():
 testloop(socket.AF_INET, tcpservers, MyStreamHandler, teststream)
+time.sleep(.1)
+reap_children()
 testloop(socket.AF_INET, udpservers, MyDatagramHandler, testdgram)
+time.sleep(.1)
+reap_children()
 if hasattr(socket, 'AF_UNIX'):
 testloop(socket.AF_UNIX, streamservers, MyStreamHandler, teststream)
 # Alas, on Linux (at least) recvfrom() doesn't return a meaningful

So modifying SocketServer like this:
-bash-3.1$ svn diff Lib/SocketServer.py Lib/test/test_socketserver.py
Index: Lib/SocketServer.py
===
--- Lib/SocketServer.py (revision 51260)
+++ Lib/SocketServer.py (working copy)
@@ -420,6 +420,7 @@
 except os.error:
 pid = None
 if not pid: break
+print >> sys.__stdout__, 'pid', pid, 'active',
self.active_children self.active_children.remove(pid)

 def process_request(self, request, client_address):
@@ -428,6 +429,7 @@
 pid = os.fork()
 if pid:
 # Parent process
+print >> sys.__stdout__, 'in parent, adding', pid,
self.active_children
 if self.active_children is None:
 self.active_children = []
 self.active_children.append(pid)

We see this on error (prior to the test patch above):

test_socketserver
in parent, adding 3084 None
pid 3084 active [3084]
in parent, adding 2408 []
pid 2408 active [2408]
in parent, adding 3908 []
in parent, adding 532 None
pid 3908 active [532]
Exception in thread Thread-12:
Traceback (most recent call last):
 ...
 File "/home/neal/build/python/trunk/Lib/SocketServer.py", line 424,
in collect_children
self.active_children.remove(pid)
ValueError: list.remove(x): x not in list
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] no remaining issues blocking 2.5 release

[Neal Norwitz]

I just updated the PEP to remove all references to issues blocking
release of 2.5.
I don't know of any.  I haven't heard of any issues with the fixes
that have been checked in.

If you have issues, respond ASAP!  The release candidate is planned to
be cut this Thursday/Friday.  There are only a few more days before
code freeze.  A branch will be made when the release candidate is cut.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type of range object members

[Neal Norwitz]

On 8/15/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:


That penalty is already paid today. Much code dealing with
ints has a type test whether it's an int or a long. If
int and long become subtypes of each other or of some abstract
type, performance will decrease even more because a subtype
test is quite expensive if the object is neither int nor
long (it has to traverse the entire base type hierarchy to
find out its not inherited from int).


I was playing around with a little patch to avoid that penalty.  It
doesn't take any additional memory, just a handful of bits we aren't
using. :-)

For the more common builtin types, it stores whether it's a subclass
in tp_flags, so there's no function call necessary and it's a constant
time operation.  It was faster when doing simple stuff.  Haven't
thought much whether this is really worthwhile or not.

n
Index: Include/stringobject.h
===
--- Include/stringobject.h	(revision 51237)
+++ Include/stringobject.h	(working copy)
@@ -55,8 +55,9 @@
 PyAPI_DATA(PyTypeObject) PyBaseString_Type;
 PyAPI_DATA(PyTypeObject) PyString_Type;
 
-#define PyString_Check(op) PyObject_TypeCheck(op, &PyString_Type)
 #define PyString_CheckExact(op) ((op)->ob_type == &PyString_Type)
+#define PyString_Check(op) (PyString_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_STRING_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyString_FromStringAndSize(const char *, Py_ssize_t);
 PyAPI_FUNC(PyObject *) PyString_FromString(const char *);
Index: Include/dictobject.h
===
--- Include/dictobject.h	(revision 51237)
+++ Include/dictobject.h	(working copy)
@@ -90,8 +90,9 @@
 
 PyAPI_DATA(PyTypeObject) PyDict_Type;
 
-#define PyDict_Check(op) PyObject_TypeCheck(op, &PyDict_Type)
 #define PyDict_CheckExact(op) ((op)->ob_type == &PyDict_Type)
+#define PyDict_Check(op) (PyDict_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_DICT_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyDict_New(void);
 PyAPI_FUNC(PyObject *) PyDict_GetItem(PyObject *mp, PyObject *key);
Index: Include/unicodeobject.h
===
--- Include/unicodeobject.h	(revision 51237)
+++ Include/unicodeobject.h	(working copy)
@@ -390,8 +390,9 @@
 
 PyAPI_DATA(PyTypeObject) PyUnicode_Type;
 
-#define PyUnicode_Check(op) PyObject_TypeCheck(op, &PyUnicode_Type)
 #define PyUnicode_CheckExact(op) ((op)->ob_type == &PyUnicode_Type)
+#define PyUnicode_Check(op) (PyUnicode_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_UNICODE_SUBCLASS))
 
 /* Fast access macros */
 #define PyUnicode_GET_SIZE(op) \
Index: Include/intobject.h
===
--- Include/intobject.h	(revision 51237)
+++ Include/intobject.h	(working copy)
@@ -27,8 +27,9 @@
 
 PyAPI_DATA(PyTypeObject) PyInt_Type;
 
-#define PyInt_Check(op) PyObject_TypeCheck(op, &PyInt_Type)
 #define PyInt_CheckExact(op) ((op)->ob_type == &PyInt_Type)
+#define PyInt_Check(op) (PyInt_CheckExact(op) || \
+		 PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_INT_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyInt_FromString(char*, char**, int);
 #ifdef Py_USING_UNICODE
Index: Include/listobject.h
===
--- Include/listobject.h	(revision 51237)
+++ Include/listobject.h	(working copy)
@@ -40,8 +40,9 @@
 
 PyAPI_DATA(PyTypeObject) PyList_Type;
 
-#define PyList_Check(op) PyObject_TypeCheck(op, &PyList_Type)
 #define PyList_CheckExact(op) ((op)->ob_type == &PyList_Type)
+#define PyList_Check(op) (PyList_CheckExact(op) || \
+		PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_LIST_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyList_New(Py_ssize_t size);
 PyAPI_FUNC(Py_ssize_t) PyList_Size(PyObject *);
Index: Include/object.h
===
--- Include/object.h	(revision 51237)
+++ Include/object.h	(working copy)
@@ -517,6 +517,18 @@
 /* Objects support nb_index in PyNumberMethods */
 #define Py_TPFLAGS_HAVE_INDEX (1L<<17)
 
+/* These flags are used to determine if a type is a subclass. */
+/* Uses bits 30-27. */
+#define Py_TPFLAGS_FAST_SUBCLASS_MASK (0x7800)
+#define Py_TPFLAGS_FAST_SUBCLASS (1L<<27)
+#define Py_TPFLAGS_INT_SUBCLASS		(Py_TPFLAGS_FAST_SUBCLASS | 0x7000)
+#define Py_TPFLAGS_LONG_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x6000)
+#define Py_TPFLAGS_LIST_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x5000)
+#define Py_TPFLAGS_TUPLE_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x4000)
+#define Py_TPFLAGS_STRING_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x3000)
+#define Py_TPFLAGS_UNICODE_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x2000)
+#define Py_TPFLAGS_DICT_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x1000)
+
 #define Py_TPFLAGS_DEFAULT  ( \
  Py_TPFLAGS_HAVE_GE

Re: [Python-Dev] no remaining issues blocking 2.5 release

[Neal Norwitz]

On 8/15/06, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
>
> It would be nice if someone could bytecompile Lib using
> Tools/compiler/compile.py and then run the test suite.  I'd do it
> myself but can't spare the time at the moment (I started but ran
> into what seems to be a gcc bug along the way).

Has this been done before?

# This code causes python to segfault
def foo(S):
  all(x > 42 for x in S)

# around Python/ceval.c 2167:  x = (*v->ob_type->tp_iternext)(v);
# tp_iternext is NULL

I added the changes below to Lib/compiler/pycodegen.py which are
clearly wrong.  It just crashes in a diff place.  I think the changes
to genxpr inner may be close to correct.  The changes to _makeClosure
and visitGenExpr are clearly wrong.  I was just wondering how far it
would go.  There are a bunch of differences. Some are the bytecode
optimizations or different ordering, but others are things dealing
with co_names, co_varnames.

Hopefully someone has time to look into this.  Otherwise, it will have
to wait for 2.5.1

n
--
Index: Lib/compiler/pycodegen.py
===
--- Lib/compiler/pycodegen.py   (revision 51305)
+++ Lib/compiler/pycodegen.py   (working copy)
@@ -628,9 +628,9 @@
 self.newBlock()
 self.emit('POP_TOP')

-def _makeClosure(self, gen, args):
+def _makeClosure(self, gen, args, gen_outer=False):
 frees = gen.scope.get_free_vars()
-if frees:
+if frees and not gen_outer:
 for name in frees:
 self.emit('LOAD_CLOSURE', name)
 self.emit('BUILD_TUPLE', len(frees))
@@ -646,7 +646,7 @@
 walk(node.code, gen)
 gen.finish()
 self.set_lineno(node)
-self._makeClosure(gen, 0)
+self._makeClosure(gen, 0, True)
 # precomputation of outmost iterable
 self.visit(node.code.quals[0].iter)
 self.emit('GET_ITER')
@@ -655,6 +655,11 @@
 def visitGenExprInner(self, node):
 self.set_lineno(node)
 # setup list
+after = self.newBlock()
+start = self.newBlock()
+self.setups.push((LOOP, start))
+self.emit('SETUP_LOOP', after)
+self.nextBlock(start)

 stack = []
 for i, for_ in zip(range(len(node.quals)), node.quals):
@@ -676,8 +681,12 @@
 self.startBlock(cont)
 self.emit('POP_TOP')
 self.nextBlock(skip_one)
+self.emit('POP_TOP')
 self.emit('JUMP_ABSOLUTE', start)
 self.startBlock(anchor)
+self.emit('POP_BLOCK')
+self.setups.pop()
+self.nextBlock(after)
 self.emit('LOAD_CONST', None)

 def visitGenExprFor(self, node):
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.4 & 2.5 beta 3 crash

[Neal Norwitz]

Thanks Dino.

The attached patch should fix the problem.  Once RC1 is cut, I'll
check this in unless someone beats me to it.  Since the compiler
changed, I can't backport this.  If someone wants to make a similar
fix for 2.4 go for it.

n
--

On 8/16/06, Dino Viehland <[EMAIL PROTECTED]> wrote:

We've been working on fixing some exception handling bugs in IronPython where 
we differ from CPython.  Along the way we ran into this issue which causes 
CPython to crash when the code below is run.  It crashes on both 2.4 and 2.5 
beta 3.  The code's technically illegal, but it probably shouldn't crash either 
:)

def test():
for abc in range(10):
try: pass
finally:
try:
continue
except:
pass


test()

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com

Index: Python/compile.c
===
--- Python/compile.c	(revision 51334)
+++ Python/compile.c	(working copy)
@@ -143,7 +143,8 @@
 	PyFutureFeatures *c_future; /* pointer to module's __future__ */
 	PyCompilerFlags *c_flags;
 
-	int c_interactive;	 /* true if in interactive mode */
+	unsigned int c_in_finally : 1;	 /* true if in finally clause */
+	unsigned int c_interactive : 1;	 /* true if in interactive mode */
 	int c_nestlevel;
 
 	struct compiler_unit *u; /* compiler state for current block */
@@ -2288,10 +2289,16 @@
 compiler_continue(struct compiler *c)
 {
 	static const char LOOP_ERROR_MSG[] = "'continue' not properly in loop";
+	static const char IN_FINALLY_ERROR_MSG[] = 
+			"'continue' not supported inside 'finally' clause";
 	int i;
 
 	if (!c->u->u_nfblocks)
 		return compiler_error(c, LOOP_ERROR_MSG);
+
+	if (c->c_in_finally)
+		return compiler_error(c, IN_FINALLY_ERROR_MSG);
+
 	i = c->u->u_nfblocks - 1;
 	switch (c->u->u_fblock[i].fb_type) {
 	case LOOP:
@@ -2306,8 +2313,8 @@
 		ADDOP_JABS(c, CONTINUE_LOOP, c->u->u_fblock[i].fb_block);
 		break;
 	case FINALLY_END:
-		return compiler_error(c,
-			"'continue' not supported inside 'finally' clause");
+		/* XXX(nnorwitz): should have already been handled above. */
+		return compiler_error(c, IN_FINALLY_ERROR_MSG);
 	}
 
 	return 1;
@@ -2367,7 +2374,9 @@
 	compiler_use_next_block(c, end);
 	if (!compiler_push_fblock(c, FINALLY_END, end))
 		return 0;
+	c->c_in_finally = 1;
 	VISIT_SEQ(c, stmt, s->v.TryFinally.finalbody);
+	c->c_in_finally = 0;
 	ADDOP(c, END_FINALLY);
 	compiler_pop_fblock(c, FINALLY_END, end);
 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type of range object members

[Neal Norwitz]

On 8/16/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
>
> It seems to me that you could drop the FAST_SUBCLASS bit, since none of the
> other bits will be set if it is not a subclass of a builtin.  That would
> free up one flag bit -- perhaps usable for that BaseException flag Guido
> wants.  :)

:-)  Right, I'm not using the bit currently.  I was thinking that it
would be interesting to change the CheckExact versions to also use
this.  It's a little more work, but you lose the second comparison for
Check.  I expect that it would be slower, but I was just curious.

So with the patch we currently have:

#define PyInt_CheckExact(op) ((op)->ob_type == &PyInt_Type)
#define PyInt_Check(op) (PyInt_CheckExact(op) || \
 PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_INT_SUBCLASS))

But we could have something like:

#define PyInt_CheckExact(op) (PyType_FastClass(op,Py_TPFLAGS_INT_CLASS))
#define PyInt_Check(op) (PyType_FastSubclass(op,Py_TPFLAGS_INT_SUBCLASS))

It would change the CheckExact()s from: op->ob_type ==
global-variable, to: op->ob_type & CONSTANT == CONSTANT.  Check would
be the same as the CheckExact, just with different constants.  The
Check version would then drop the || condition.

I might play with this at the sprint next week.  It does seem to make
sense to do BaseException too.  It will take 4 or 5 bits to handle the
current ones plus BaseException, which we can easily spare in
tp_flags.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing 2.5 windows buildbots

[Neal Norwitz]

On 8/17/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> I'd like to fix the two build failures that the Windows buildbots
> show for the 2.5 trunk. I'm not quite sure what the OpenSSL failure
> is, yet, but the sqlite error should be fixable with the patch
> below. Ok to work on this?

Please do.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Sprints next week at Google

[Neal Norwitz]

This is a reminder (I don't think anyone else sent one, I sure hope not).

We are holding 4 days of sprints next week at Google offices in NY
city and Mt View, CA.  These are open if you'd like to attend.  It
would be very helpful to pre-register on the wiki as we can notify
security and generally make things easier.  Of the full-time Googlers,
Guido and Alex will be sprinting in CA and Jeremy and I will be
sprinting in NY.

http://wiki.python.org/moin/GoogleSprint

Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.4 & 2.5 beta 3 crash

[Neal Norwitz]

I did something similar to what Andrew suggested.

http://python.org/sf/1542451

Could everyone interested take a look and offer more test cases?

n
--
On 8/17/06, A.M. Kuchling <[EMAIL PROTECTED]> wrote:
> On Fri, Aug 18, 2006 at 12:26:33AM +0200, Armin Rigo wrote:
> > Without more inspection, I'd say that this looks like it won't do the
> > right thing about nested finally's, as in:
>
> I guess it'll have to loop back up through the block stack:
>
> for (j=i- 1; j>=0; j--) {
>switch (c->u->u_fblock[j].fb_type) {
> case FINALLY_TRY:
>raise error;
> case LOOP:
>j=0;  /* Exit the loop */
>break;
> default:
>break;
>   }
> }
>
> --amk
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed building 2.5rc1 pythoncore on VC8

[Neal Norwitz]

VC8 is not a supported compiler at this point.  However, patches are
greatly accepted.

The _types module was added late and probably VC6 and VC8 project
files did not get updated.  You can search for the necessary mods to
the VC7 proj file(s) on python-checkins.

n
--
On 8/18/06, christopher baus <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I downloaded RC1 to give it a go.
>
> I failed to build pythoncore with VC8.
>
> The error is:
>
> Linking...
>Creating library .\./python25_d.lib and object .\./python25_d.exp
> config.obj : error LNK2001: unresolved external symbol _init_types
> ./python25_d.dll : fatal error LNK1120: 1 unresolved externals
>
> Thanks,
>
> Chris
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] xrange accepting non-ints

[Neal Norwitz]

I've been working on enhancing xrange and there are a bunch of issues
to consider.  I've got pretty much complete implementations in both C
and Python.  Currently xrange is 2 objects:  range and the iter.
These only work on C longs.  Here's what I propose:

2.6:
 * Add deprecation warning if a float is passed to xrange (currently
silently truncates)
 * Allow int, long, float, or obj.__index__
 * Implement xrange in python
 * Implement iterator over C longs (or Py_ssize_t) in C
 * Implement iterator over Python longs in Python (* may lose __length_hint__)
 * Capture the values on construction, so mutating objects wouldn't
affect xrange

The idea is to expand xrange's capabilities so that it can replace range in 3k.

I've profiled various combinations.  Here are the various results
normalized doing xrange(0, 1e6, 1):

Run on all integer (32-bit) values for start, step, end:
C xrange and iter:  1
Py xrange w/C iter: 1
Py xrange w/Py iter (gen): 5-8
Py xrange w/Py iter (class): ~30

So having xrange in python is the same speed as if xrange is written
in C.  The important part is that the iterator needs to be written in
C for speed.  If we use a generator, something like:

while value < end:
yield value
value += step

The result is ~5 times slower in a release build and 8 times slower in
a debug build than with an iterator implemented in C.  Using a
generator means that there is no __length_hint__.  If we go with a
full class that has a __length_hint__ the result was ~32 times slower
in a debug build.

The Python impl is about 1/10th the size of the C impl, though is
lacking some comments.

Run on Python longs the result is somewhat interesting.  The Python
based iterator is faster.  There's probably a bug in the C version,
but given that there is a lot of object allocation, I wouldn't expect
the C version to ever be much faster than a similar Python version.
Plus the Python version is trivial (same as above) for ints or longs.
The C version for longs is quite a bit of code.

Run on all Python longs (still 0..1e6, but sys.maxint..(sys.maxint +
1e6) is the same):
C xrange and iter:  1.4
Py xrange w/C iter: not tested
Py xrange w/Py iter (gen): 1
Py xrange w/Py iter (class): 4

Caveats:
 * The generator version above doesn't support floats.  We could
easily support floats with a  different calculation that would be
slightly more expensive, but not have accumulated error.
 * By using the generator version, __length_hint__ gets dropped.  This
means that converting the iterator into a sequence could be slightly
more costly as we have to increase the allocation.  This would only
happen if any of start, step, end weren't an int.
 * With a python implementation there is a little bit of bootstraping
that is necessary to get the iter implemented in C into the xrange
object implemented in Python
 * Since xrange is implemented in Python, it can be changed.
 * The Python code is much easier to understand than the C code (there
is at least one bug in the current C version where -sys.maxint -1
isn't always displayed properly).

Hopefully this is all understandable.  If I left anything out, Thomas
will remind me.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] for 2.5 issues

[Neal Norwitz]

I don't want to make any more changes to 2.5 unless we absolutely have
to.  I also don't want to lose fixes.  How about for anything that
should be resolved in 2.5, but wait for 2.5.1 we set the tracker item
to:  Group 2.5, Resolution: Later, Priority 7.

Then it should be easy to find these things.

Thanks,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] TRUNK IS UNFROZEN, available for 2.6 work if you are so inclined

[Neal Norwitz]

On 8/18/06, Georg Brandl <[EMAIL PROTECTED]> wrote:
>
> I'd like to commit this. It fixes bug 1542051.
>
> Index: Objects/exceptions.c

...

Georg,

Did you still want to fix this?  I don't remember anything happening
with it.  I don't see where _PyObject_GC_TRACK is called, so I'm not
sure why _PyObject_GC_UNTRACK is necessary.  You should probably add
the patch to the bug report and we can discuss there.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] no remaining issues blocking 2.5 release

[Neal Norwitz]

Gustavo,

Did you still want this addressed?  Anthony and I made some comments
on the bug/patch, but nothing has been updated.

n
--

On 8/15/06, Gustavo Niemeyer <[EMAIL PROTECTED]> wrote:
> > If you have issues, respond ASAP!  The release candidate is planned to
> > be cut this Thursday/Friday.  There are only a few more days before
> > code freeze.  A branch will be made when the release candidate is cut.
>
> I'd like to see problem #1531862 fixed.  The bug is clear and the
> fix should be trivial.  I can commit a fix tonight, if the subprocess
> module author/maintainer is unavailable to check it out.
>
> --
> Gustavo Niemeyer
> http://niemeyer.net
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.5 status

[Neal Norwitz]

There are 3 bugs currently listed in PEP 356 as blocking:
http://python.org/sf/1551432 - __unicode__ breaks on exception classes
http://python.org/sf/1550938 - improper exception w/relative import
http://python.org/sf/1541697 - sgmllib regexp bug causes hang

Does anyone want to fix the sgmlib issue?  If not, we should revert
this week before c2 is cut.  I'm hoping that we will have *no changes*
in 2.5 final from c2.  Should there be any bugs/patches added to or
removed from the list?

The buildbots are currently humming along, but I believe all 3
versions (2.4, 2.5, and 2.6) are fine.

Test out 2.5c1+ and report all bugs!

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5 status

[Neal Norwitz]

On 9/5/06, Brett Cannon <[EMAIL PROTECTED]> wrote:
>
> > [MAL]
> > The proper fix would be to introduce a tp_unicode slot and let
> > this decide what to do, ie. call .__unicode__() methods on instances
> > and use the .__name__ on classes.
>
> That was my bug reaction  and what I said on the bug report.  Kind of
> surprised one doesn't already exist.
>
> > I think this would be the right way to go for Python 2.6. For
> > Python 2.5, just dropping this .__unicode__ method on exceptions
> > is probably the right thing to do.
>
> Neal, do you want to rip it out or should I?

Is removing __unicode__ backwards compatible with 2.4 for both
instances and exception classes?

Does everyone agree this is the proper approach?  I'm not familiar
with this code.  Brett, if everyone agrees (ie, remains silent),
please fix this and add tests and a NEWS entry.

Everyone should be looking for incompatibilities with previous
versions.  Exceptions are new and deserve special attention.  Lots of
the internals of strings (8-bit and unicode) and the struct module
changed and should be tested thoroughly.  I'm sure there are a bunch
of other things I'm not remembering.  The compiler is also an obvious
target to verify your code still works.

We're stuck with anything that makes it into 2.5, so now is the time
to fix these problems.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5 status

[Neal Norwitz]

Doc patches are fine, please fix.

n
--

On 9/7/06, Ronald Oussoren <[EMAIL PROTECTED]> wrote:
>
> On 5-sep-2006, at 6:24, Neal Norwitz wrote:
>
> > There are 3 bugs currently listed in PEP 356 as blocking:
> > http://python.org/sf/1551432 - __unicode__ breaks on
> > exception classes
> > http://python.org/sf/1550938 - improper exception w/
> > relative import
> > http://python.org/sf/1541697 - sgmllib regexp bug causes hang
> >
> > Does anyone want to fix the sgmlib issue?  If not, we should revert
> > this week before c2 is cut.  I'm hoping that we will have *no changes*
> > in 2.5 final from c2.  Should there be any bugs/patches added to or
> > removed from the list?
> >
> > The buildbots are currently humming along, but I believe all 3
> > versions (2.4, 2.5, and 2.6) are fine.
> >
> > Test out 2.5c1+ and report all bugs!
>
> I have another bug that I'd like to fix: Mac/ReadMe contains an
> error: it claims that you can build the frameworkinstall into a
> temporary directory and then move it into place, but that isn't
> actually true. The erroneous paragraph is this:
>
> Note that there are no references to the actual locations in the
> code or
> resource files, so you are free to move things around afterwards.
> For example,
> you could use --enable-framework=/tmp/newversion/Library/
> Frameworks and use
> /tmp/newversion as the basis for an installer or something.
>
> My proposed fix is to drop this paragraph. There is no bugreport for
> this yet, I got notified of this issue in a private e-mail.
>
> Ronald
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.5c2

[Neal Norwitz]

PEP 356

  http://www.python.org/dev/peps/pep-0356/

has 2.5c2 scheduled for Sept 12.  I checked in a fix for the last
blocking 2.5 issue (revert sgml infinite loop bug).  There are no
blocking issues that I know of (the PEP is up to date).

I expect Anthony will call for a freeze real soon now.  It would be
awesome if there were no more changes from now until for 2.5 final!
(Changing the trunk or 2.4 branches are fine, updating doc for 2.5 is
also fine).

I will be running valgrind over 2.5, but don't expect anything to show
up since the last run was pretty recent.  Coverity has no outstanding
issues and Klocwork results are pretty clean.  It's not clear if the
remaining warnings from Klocwork are real issues or not.

Keep doing a bunch of testing so we don't have any surprises in 2.5.

n

PS Scary as it sounds, I hope to have an HP-UX buildbot up and running
real soon now.  After 2.5 is out, I will fix the issues with the
cygwin bot (ie, upgrade cygwin) and get the HP-UX bot running.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] _PyGILState_NoteThreadState should be static or not?

[Neal Norwitz]

Michael,

In Python/pystate.c, you made this checkin:

"""
r39044 | mwh | 2005-06-20 12:52:57 -0400 (Mon, 20 Jun 2005) | 8 lines

Fix bug:  [ 1163563 ] Sub threads execute in restricted mode
basically by fixing bug 1010677 in a non-broken way.
"""

_PyGILState_NoteThreadState is declared as static on line 54, but the
definition on line 508 is not static. (HP's cc is complaining.)  I
don't see this referenced in any header file, it seems like this
should be static?

$ grep _PyGILState_NoteThreadState */*.ch]
Python/pystate.c:static void _PyGILState_NoteThreadState(PyThreadState* tstate);
Python/pystate.c:   _PyGILState_NoteThreadState(tstate);
Python/pystate.c:   _PyGILState_NoteThreadState(t);
Python/pystate.c:_PyGILState_NoteThreadState(PyThreadState* tstate)

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] fun threading problem

[Neal Norwitz]

On everyones favorite platform (HP-UX), the following code consistently fails:

###
from thread import start_new_thread, allocate_lock
from time import sleep

def bootstrap():
from os import fork ; fork()
allocate_lock().acquire()

start_new_thread(bootstrap, ())
sleep(.1)
###

The error is:
Fatal Python error: Invalid thread state for this thread

This code was whittled down from test_socketserver which fails in the
same way.  It doesn't matter what value is passed to sleep as long as
it's greater than 0.  I also tried changing the sleep to a while 1:
pass and the same problem occurred.  So there isn't a huge interaction
of APIs, only:  fork, allocate_lock.acquire and start_new_thread.

HP-UX seems to be more sensitive to various threading issues.  In
Modules/_test_capimodule.c, I had to make this modification:

Index: Modules/_testcapimodule.c
===
--- Modules/_testcapimodule.c   (revision 51875)
+++ Modules/_testcapimodule.c   (working copy)
@@ -665,6 +665,9 @@
PyThread_acquire_lock(thread_done, 1);  /* wait for thread to finish */
Py_END_ALLOW_THREADS

+   /* Release lock we acquired above.  This is required on HP-UX. */
+   PyThread_release_lock(thread_done);
+
PyThread_free_lock(thread_done);
Py_RETURN_NONE;
 }

Without that patch, there would be this error:

sem_destroy: Device busy
sem_init: Device busy
Fatal Python error: UNREF invalid object
ABORT instruction (core dumped)

Anyone have any ideas?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] fun threading problem

[Neal Norwitz]

On 9/14/06, Aahz <[EMAIL PROTECTED]> wrote:
> On Thu, Sep 14, 2006, Neal Norwitz wrote:
> >
> > On everyones favorite platform (HP-UX), the following code
> > consistently fails:
>
> Which exact HP-UX?  I remember from my ancient days that each HP-UX
> version completely changes the way threading works -- dunno whether
> that's still true.

 HP-UX 11i v2 on PA-RISC

td191 on http://www.testdrive.hp.com/current.shtml
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Before 2.5 - More signed integer overflows

[Neal Norwitz]

I also tested the fix (see patch below) for the abs() issue and it
seemed to work for 4.1.1 on 64-bit.  I'll apply the patch to head and
2.5 and a test after 2.5 is out.

I have no idea how to search for these problems.  I know that xrange
can't display -sys.maxint-1 properly, but I think it works properly.

n
--

Index: Objects/intobject.c
===
--- Objects/intobject.c (revision 51886)
+++ Objects/intobject.c (working copy)
@@ -763,7 +763,7 @@
register long a, x;
a = v->ob_ival;
x = -a;
-   if (a < 0 && x < 0) {
+   if (a < 0 && (unsigned long)x == 0-(unsigned long)x) {
PyObject *o = PyLong_FromLong(a);
if (o != NULL) {
PyObject *result = PyNumber_Negative(o);


On 9/17/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > BTW, did anyone try compiling Python with -fwrapv on a box where it
> > matters?  I doubt that Python's speed is affected one way or the
> > other, and if adding wrapv makes the problems go away, that would be
> > an easy last-second workaround for all possible such problems (which
> > of course could get fixed "for real" for 2.5.1, provided someone cares
> > enough to dig into it).
>
> It's not so easy to add this option: configure needs to be taught to
> check whether the option is supported first; to test it, you ideally
> need an installation where it is supported, and one where it isn't.
>
> I've added a note to README indicating that GCC 4.2 shouldn't be
> used to compile Python. I don't consider this a terrible limitation,
> especially since GCC 4.2 isn't released, yet.
>
> OTOH, I get the same problem that Armin gets (abs(-sys.maxint-1)
> is negative) also on a 32-bit system, with Debian's gcc 4.1.2
> (which also isn't released, yet), so it appears that the problem
> is already with gcc 4.1.
>
> On my system, adding -fwrapv indeed solves the problem
> (tested for abs()). So I added this to the README also.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Before 2.5 - More signed integer overflows

[Neal Norwitz]

On 9/17/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz schrieb:
> > I also tested the fix (see patch below) for the abs() issue and it
> > seemed to work for 4.1.1 on 64-bit.  I'll apply the patch to head and
> > 2.5 and a test after 2.5 is out.
>
> Please also add it to 2.4.

Yes

>
> > Index: Objects/intobject.c
> > ===
> > --- Objects/intobject.c (revision 51886)
> > +++ Objects/intobject.c (working copy)
> > @@ -763,7 +763,7 @@
> >register long a, x;
> >a = v->ob_ival;
> >x = -a;
> > -   if (a < 0 && x < 0) {
> > +   if (a < 0 && (unsigned long)x == 0-(unsigned long)x) {
>
> Hmm. Shouldn't this drop 'x' and use 'a' instead? If a is
> -sys.maxint-1, -a is already undefined.

Yes, probably.  I didn't review carefully.

> P.S. As for finding these problems, I would have hoped that
> -ftrapv could help - unfortunately, gcc breaks with this
> option (consumes incredible amounts of memory).

I'm getting a crash when running test_builtin and test_calendar (at
least) with gcc 4.1.1 on amd64.  It's happening in pymalloc, though I
don't know what the cause is.  I thought I tested with gcc 4.1 before,
but probably would have been in debug mode.

n
--
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 16384 (LWP 22020)]
PyObject_Malloc (nbytes=40) at obmalloc.c:746
746 if ((pool->freeblock = *(block **)bp) != NULL) {
(gdb) p bp
$1 = (block *) 0x2a9558d41800 
(gdb) l
741  * Pick up the head block of its free list.
742  */
743 ++pool->ref.count;
744 bp = pool->freeblock;
745 assert(bp != NULL);
746 if ((pool->freeblock = *(block **)bp) != NULL) {
747 UNLOCK();
748 return (void *)bp;
749 }
750 /*
(gdb) p *pool
$2 = {ref = {_padding = 0x1a , count = 26},
  freeblock = 0x2a9558d41800 ,
  nextpool = 0x2a95eac000, prevpool = 0x620210, arenaindex = 0, szidx = 4,
  nextoffset = 4088, maxnextoffset = 4056}
(gdb) p size
$3 = 4
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Before 2.5 - More signed integer overflows

[Neal Norwitz]

On 9/18/06, Tim Peters <[EMAIL PROTECTED]> wrote:
> [Neal Norwitz]
> >> I'm getting a crash when running test_builtin and test_calendar (at
> >> least) with gcc 4.1.1 on amd64.  It's happening in pymalloc, though I
> >> don't know what the cause is.  I thought I tested with gcc 4.1 before,
> >> but probably would have been in debug mode.
>
> Neil, in context it was unclear whether you were using trapv at the
> time.  Were you?

No trapv, just ./configure --without-pydebug IIRC.  I should have sent
a msg last night, but was too tired.  I got the same crash (I think)
with gcc 3.4.4, so it's almost definitely due to an outstanding
change, not python's or gcc's fault.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release25-maint is UNFROZEN

[Neal Norwitz]

On 9/21/06, Armin Rigo <[EMAIL PROTECTED]> wrote:
> Hi Anthony,
>
> On Thu, Sep 21, 2006 at 09:12:03PM +1000, Anthony Baxter wrote:
> > Thanks to everyone for helping make 2.5 happen. It's been a long slog there,
> > but I think we can all be proud of the result.
>
> Thanks for the hassle!  I've got another bit of it for you, though.  The
> freezed 2.5 documentation doesn't seem to be available on-line.  At
> least, the doc links from the release page point to the 'dev' 2.6a0
> version, and the URL following the common scheme -
> http://www.python.org/doc/2.5/ - doesn't work.

I got  http://docs.python.org/dev/2.5/ working last night.  So when
the 2.5 docs are updated these pages will reflect that.

http://docs.python.org/ should point to the 2.5 doc too.  I looked at
making these changes, but was confused by what needed to be done.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_itertools fails for trunk on x86 OS X machine

[Neal Norwitz]

On 9/21/06, Jack Diederich <[EMAIL PROTECTED]> wrote:
>
> I should leave the tounge-in-cheek bombast to Tim and Frederik, especially
> when dealing with what might be an OS & machine specific bug.  The next
> checkin and re-test will or won't highlight a failure and certainly someone
> with a g4 will try it out before 2.5.1 goes out so we'll know if it was a
> fluke soonish. The original error was mine, I typed "Size_t" instead of
> "Ssize_t" and while my one-char patch might also be wrong (I hope not, I'm
> red-faced enough as is) we should find out soon enough.

It looks like %zd of a negative number is treated as an unsigned
number on OS X, even though the man page says it should be signed.

"""
The z modifier, when applied to a d or i conversion, indicates that
the argument is of a signed type equivalent in size to a size_t.
"""

The program below returns -123 on Linux and 4294967173 on OS X.

n
--
#include 
int main()
{
char buffer[256];
  if(sprintf(buffer, "%zd", (size_t)-123) < 0)
return 1;
 printf("%s\n", buffer);
 return 0;
}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_itertools fails for trunk on x86 OS X machine

[Neal Norwitz]

On 9/21/06, Tim Peters <[EMAIL PROTECTED]> wrote:
>
> Well, to be strictly anal, while the result of
>
> (size_t)-123
>
> is defined, the result of casting /that/ back to a signed type of the
> same width is not defined.  Maybe your compiler was "doing you a
> favor" ;-)

I also tried with a cast to an ssize_t and replacing %zd with an %zi.
None of them make a difference; all return an unsigned value.  This is
with powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple
Computer, Inc. build 4061).  Although i would expect the issue is in
the std C library rather than the compiler.

Forcing PY_FORMAT_SIZE_T to be "l" instead of "z" fixes this problem.

BTW, this is the same issue on Mac OS X:

>>> struct.pack('=b', -59)
__main__:1: DeprecationWarning: 'b' format requires 4294967168 <= number <= 127
'A'

n
--
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Typo.pl scan of Python 2.5 source code

[Neal Norwitz]

On 9/22/06, Johnny Lee <[EMAIL PROTECTED]> wrote:
>
> Hello,
> My name is Johnny Lee. I have developed a *ahem* perl script which scans
> C/C++ source files for typos.

Hi Johnny.

Thanks for running your script, even if it is written in Perl and ran
on Windows. :-)

> The Python 2.5 typos can be classified into 7 types.
>
> 2) realloc overwrite src if NULL, i.e. p = realloc(p, new_size);
> If realloc() fails, it will return NULL. If you assign the return value to
> the same variable you passed into realloc,
> then you've overwritten the variable and possibly leaked the memory that the
> variable pointed to.

A bunch of these warnings were accurate and a bunch were not.  There
were 2 reasons for the false positives.  1) The pointer was aliased,
thus not lost, 2) On failure, we exited (Parser/*.c)

> 4) if ((X!=0) || (X!=1))

These 2 cases occurred in binascii.  I have no idea if the warning is
wright or the code is.

> 6) XX;;
> Just being anal here. Two semicolons in a row. Second one is extraneous.

I already checked in a fix for these on HEAD.  Hard for even me to
screw up those fixes. :-)

> 7) extraneous test for non-NULL ptr
> Several memory calls that free memory accept NULL ptrs.
> So testing for NULL before calling them is redundant and wastes code space.
> Now some codepaths may be time-critical, but probably not all, and smaller
> code usually helps.

I ignored these as I'm not certain all the platforms we run on accept
free(NULL).

Below is my categorization of the warnings except #7.  Hopefully
someone will fix all the real problems in the first batch.

Thanks again!

n
--

# Problems
Objects\fileobject.c (338): realloc overwrite src if NULL; 17:
file->f_setbuf=(char*)PyMem_Realloc(file->f_setbuf,bufsize)
Objects\fileobject.c (342): using PyMem_Realloc result w/no check
30: setvbuf(file->f_fp, file->f_setbuf, type, bufsize);
[file->f_setbuf]
Objects\listobject.c (2619):using PyMem_MALLOC result w/no check
30: garbage[i] = selfitems[cur]; [garbage]
Parser\myreadline.c (144):  realloc overwrite src if NULL; 17:
p=(char*)PyMem_REALLOC(p,n+incr)
Modules\_csv.c (564):   realloc overwrite src if NULL; 17:
self->field=PyMem_Realloc(self->field,self->field_size)
Modules\_localemodule.c (366):  realloc overwrite src if NULL; 17:
buf=PyMem_Realloc(buf,n2)
Modules\_randommodule.c (290):  realloc overwrite src if NULL; 17:
key=(unsigned#long*)PyMem_Realloc(key,bigger*sizeof(*key))
Modules\arraymodule.c (1675):   realloc overwrite src if NULL; 17:
self->ob_item=(char*)PyMem_REALLOC(self->ob_item,itemsize*self->ob_size)
Modules\cPickle.c (536):realloc overwrite src if NULL; 17:
self->buf=(char*)realloc(self->buf,n)
Modules\cPickle.c (592):realloc overwrite src if NULL; 17:
self->buf=(char*)realloc(self->buf,bigger)
Modules\cPickle.c (4369):   realloc overwrite src if NULL; 17:
self->marks=(int*)realloc(self->marks,s*sizeof(int))
Modules\cStringIO.c (344):  realloc overwrite src if NULL; 17:
self->buf=(char*)realloc(self->buf,self->buf_size)
Modules\cStringIO.c (380):  realloc overwrite src if NULL; 17:
oself->buf=(char*)realloc(oself->buf,oself->buf_size)
Modules\_ctypes\_ctypes.c (2209):   using PyMem_Malloc result w/no
check 30: memset(obj->b_ptr, 0, dict->size); [obj->b_ptr]
Modules\_ctypes\callproc.c (1472):  using PyMem_Malloc result w/no
check 30: strcpy(conversion_mode_encoding, coding);
[conversion_mode_encoding]
Modules\_ctypes\callproc.c (1478):  using PyMem_Malloc result w/no
check 30: strcpy(conversion_mode_errors, mode);
[conversion_mode_errors]
Modules\_ctypes\stgdict.c (362):using PyMem_Malloc result w/no
check 30: memset(stgdict->ffi_type_pointer.elements, 0,
[stgdict->ffi_type_pointer.elements]
Modules\_ctypes\stgdict.c (376):using PyMem_Malloc result w/no
check 30: memset(stgdict->ffi_type_pointer.elements, 0,
[stgdict->ffi_type_pointer.elements]

# No idea if the code or tool is right.
Modules\binascii.c (1161)
Modules\binascii.c (1231)

# Platform specific files.  I didn't review and won't fix without testing.
Python\thread_lwp.h (107):  using malloc result w/no check 30:
lock->lock_locked = 0; [lock]
Python\thread_os2.h (141):  using malloc result w/no check 30:
(long)sem)); [sem]
Python\thread_os2.h (155):  using malloc result w/no check 30:
lock->is_set = 0; [lock]
Python\thread_pth.h (133):  using malloc result w/no check 30:
memset((void *)lock, '\0', sizeof(pth_lock)); [lock]
Python\thread_solaris.h (48):   using malloc result w/no check 30:
funcarg->func = func; [funcarg]
Python\thread_solaris.h (133):  using malloc result w/no check 30:
if(mutex_init(lock,USYNC_THREAD,0)) [lock]

# Who cares about these modules.
Modules\almodule.c:182
Modules\svmodule.c:547

# Not a problem.
Parser\firstsets.c (76)
Parser\grammar.c (40)
Parser\grammar.c (59)
Parser\grammar.c (83)
Parser\grammar.c (102)
Parser\node.c (95)
Parser\pgen.c (52)
Parser\pgen.c (69)
Parser\pgen.c (126)
Parser\pgen.c (438)
Pars

Re: [Python-Dev] what's really new in python 2.5 ?

[Neal Norwitz]

On 10/3/06, Fred L. Drake, Jr. <[EMAIL PROTECTED]> wrote:
> On Tuesday 03 October 2006 14:08, A.M. Kuchling wrote:
>  > That doesn't explain it, though; the contents of whatsnew26.html
>  > contain references to pep-308.html.  It's not simply a matter of new
>  > files being untarred on top of old.
>
> Ah; I missed that the new HTML file was referring to an old heading.  That
> does sound like a .aux file got left around.
>
> I don't know what the build process is for the material in
> docs.python.org/dev/; I think the right thing would be to start each build
> with a fresh checkout/export.

I probably did not do that to begin with.  I did rm -rf Doc && svn up
Doc && cd Doc && make.  Let me know if there's anything else I should
do.  I did this for both the 2.5 and 2.6 versions.

Let me know if you see anything screwed up after an hour or so.  The
new versions should be up by then.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)

[Neal Norwitz]

On 10/2/06, Brett Cannon <[EMAIL PROTECTED]> wrote:
>
> This is why I asked for input from people on which would take less time.
> Almost all the answers I got was that the the C code was delicate but that
> it was workable.  Several people said they wished for a Python
> implementation, but hardly anyone said flat-out, "don't waste your time, the
> Python version will be faster to do".

I didn't respond mostly because I pushed this direction to begin with.
 That and I'm lazy. :-)

There is a lot of string manipulation and some list manipulation that
is a royal pain in C and trivial in python.  Caching will be much
easier to experiement with in Python too.  The Python version will be
much smaller.  It will take far less time to code it in Python and
recode in C, than to try to get it right in C the first time.  If the
code is fast enough, there's no reason to rewrite in C.  It will
probably be easier to subclass a Python based version that a C based
version.

> As for the bootstrapping, I am sure it is resolvable as well.  There are
> several ways to go about it that are all tractable.

Right, I had bootstrapping with implementing xrange in Python, but it
was pretty easy to resolve in the end.  You might even want to use
part of that patch (from pythonrun.c?).  There was some re-org to make
bootstrapping easier/possible (I don't remember exactly right now).

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r51862 - python/branches/release25-maint/Tools/msi/msi.py

[Neal Norwitz]

On 9/12/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
>
> If you wonder how this all happened: Neal added sgml_input.html after
> c1, but didn't edit msi.py to make it included on Windows. I found out
> after running the test suite on the installed version, edited msi.py,
> and rebuilt the installer.

Is there an easy to fix this sort of problem so it doesn't happen in
the future (other than revoke my checkin privileges :-) ?

There are already so many things to remember for changes.  If we can
automate finding these sorts of problems (installation, fixing
something for one platform, but not another, etc), the submitter can
fix these things with a little prodding from the buildbots.  Or is
this too minor to worry about?

It would also be great if we could automate complaint emails about
missing NEWS entries, doc, and tests so I wouldn't have to do it. :-)
Unless anyone has better ideas how to improve Python.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Security Advisory for unicode repr() bug?

[Neal Norwitz]

On 10/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> Georg> [ Bug http://python.org/sf/1541585 ]
>
> Georg> This seems to be handled like a security issue by linux
> Georg> distributors, it's also a news item on security related pages.
>
> Georg> Should a security advisory be written and official patches be
> Georg> provided?
>
> I asked about this a few weeks ago.  I got no direct response.  Secunia sent
> mail to webmaster and the SF project admins asking about how this could be
> exploited.  (Isn't figuring that stuff out their job?)

FWIW, I responded to the original mail from Secunia with what little I
know about the problem.  Everyone on the original mail was copied.
However,  I got ~30 bounces for all the Source Forge addresses due to
some issue between SF and Google mail.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_codecs failures

[Neal Norwitz]

On 10/29/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I recently began running a Pybots buildslave for SQLAlchemy.  I am still
> struggling to get that working correctly.  Today, Python's test_codecs test
> began failing:

I checked in a fix for this that hasn't quite completed yet.  (Only
finished successfully on one box so far.)  So this should be taken
care of.  I *think* the fix was correct, but I'm not entirely
positive.

Also the refleak problem is fixed AFAIK.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] valgrind

[Neal Norwitz]

On 11/6/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Herman Geza schrieb:
> > Here python reads from an already-freed memory area, right?
>
> It looks like it, yes. Of course, it could be a flaw in valgrind, too.
> To find out, one would have to understand what the memory block is,
> and what part of PyObject_Free accesses it.

I'm a bit confused.  I ran with valgrind ./python -c pass which
returns 23 invalid read problems (some are the same chunk of memory).
This is with 2.5 (more or less).  Valgrind 3.2.1 on amd64.  Every
address ended with 0x5...020.  That seems odd.  I looked through the
valgrind bug reports and didn't see anything.  The first problem
reported was:

Invalid read of size 4
   at 0x44FA06: Py_ADDRESS_IN_RANGE (obmalloc.c:1741)
   by 0x44E225: PyObject_Free (obmalloc.c:920)
   by 0x44EB90: _PyObject_DebugFree (obmalloc.c:1361)
   by 0x444A28: dictresize (dictobject.c:546)
   by 0x444D5B: PyDict_SetItem (dictobject.c:655)
   by 0x462533: PyString_InternInPlace (stringobject.c:4920)
   by 0x448450: PyDict_SetItemString (dictobject.c:2120)
   by 0x4C240A: PyModule_AddObject (modsupport.c:615)
   by 0x428B00: _PyExc_Init (exceptions.c:2117)
   by 0x4C449A: Py_InitializeEx (pythonrun.c:225)
   by 0x4C4827: Py_Initialize (pythonrun.c:315)
   by 0x41270A: Py_Main (main.c:449)
 Address 0x52AE020 is 4,392 bytes inside a block of size 5,544 free'd
   at 0x4A1A828: free (vg_replace_malloc.c:233)
   by 0x5071635: qsort (in /lib/libc-2.3.5.so)
   by 0x474E4B: init_slotdefs (typeobject.c:5368)
   by 0x47522E: add_operators (typeobject.c:5511)
   by 0x46E3A1: PyType_Ready (typeobject.c:3209)
   by 0x46E2D4: PyType_Ready (typeobject.c:3173)
   by 0x44D13E: _Py_ReadyTypes (object.c:1864)
   by 0x4C4362: Py_InitializeEx (pythonrun.c:183)
   by 0x4C4827: Py_Initialize (pythonrun.c:315)
   by 0x41270A: Py_Main (main.c:449)
   by 0x411CD2: main (python.c:23)

Note that the free is inside qsort.  The memory freed under qsort
should definitely not be the bases which we allocated under
PyType_Ready.  I'll file a bug report with valgrind to help determine
if this is a problem in Python or valgrind.
http://bugs.kde.org/show_bug.cgi?id=136989

One other thing that is weird is that the complaint is about 4 bytes
which should not be possible.  All pointers should be 8 bytes AFAIK
since this is amd64.

I also ran this on x86.  There were 32 errors and all of their
addresses were 0x4...010.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] valgrind

[Neal Norwitz]

On 11/7/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz schrieb:
> >   at 0x44FA06: Py_ADDRESS_IN_RANGE (obmalloc.c:1741)
> >
> > Note that the free is inside qsort.  The memory freed under qsort
> > should definitely not be the bases which we allocated under
> > PyType_Ready.  I'll file a bug report with valgrind to help determine
> > if this is a problem in Python or valgrind.
> > http://bugs.kde.org/show_bug.cgi?id=136989
>
> As Tim explains, a read from Py_ADDRESS_IN_RANGE is fine, and by design.
> If p is the pointer, we do

Yeah, thanks for going over it again.  I was tired and only half
paying attention last night.  Tonight isn't going much better. :-(  I
wonder if we can capture any of these exchanges and put into
README.valgrind.  I'm not about to do it tonight though.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summer of Code: zipfile?

[Neal Norwitz]

You probably need to contact the authors for more info:

https://svn.sourceforge.net/svnroot/ziparchive/ziparchive/trunk/
http://wiki.python.org/moin/SummerOfCode

n
--
On 11/12/06, Giovanni Bajo <[EMAIL PROTECTED]> wrote:
> Hello,
>
> wasn't there a project about the zipfile module in the Summer of Code? How did
> it go?
>
> Giovanni Bajo
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LSB: Selection of a Python version

[Neal Norwitz]

What, if any, impact do you think the LSB should have wrt maintaining 2.4?

n

On 12/4/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> At the LSB meeting, there was a brief discussion of what Python
> version should be incorporated into LSB. This is more an issue
> if ABI compatibility for the C ABI is desired, but even if only
> script portability is a goal, application developers will need
> to know what APIs they have available in LSB 3.2 and which they
> don't.
>
> LSB codifies existing practice, rather than defining "anticipating"
> standards, so things get only into LSB if they are already
> commonplace in distributions. Currently, Python 2.4 is widely
> available in Linux distributions; Python 2.5 is not and (apparently)
> will not be included in the next RHEL release (which apparently
> is coming soon).
>
> So it looks like LSB will standardize on Python 2.4 (which will also
> be the default version in the next Debian release).
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LSB: Binary compatibility

[Neal Norwitz]

On 12/4/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> At the LSB meeting, Jeff Licquia asked whether Python could provide
> binary compatibility with previous versions by use of ELF symbol
> versioning. In ELF symbol versioning, you can have multiple
> definitions for the same symbol; clients (e.g. extension modules)
> would refer to a specific version. During static linking, the most
> recent (?) version of the symbol is coded into the client object.
>
> With symbol versioning, you can change the implementation of a
> function and even its interface, and it will be compatible as
> long as you keep the original version around.

[in a separate message that Python 2.4 is the first targetted version]

> All in all, I think providing binary compatibility would
> be feasible, and should be attempted. What do you think?

Let's assume that 2.4 is the first LSB version.  The ABI is different
for 2.4 and 2.5.  We can't change the ABI for 2.5 since it's already
released and our policy is to keep it constant.

Where does that leave us for moving forward?  Would we have to modify
2.5 to make it entirely compatible with 2.4?  Can we skip 2.5
altogether and worry about making 2.6 compatible with 2.4?  Even if
so, that means no matter what we still need to make 2.4 compatible
APIs going forward, right?  That will require some archeaology to
determine preciesly what we changed from 2.4.

Is our current practice of the following change acceptable?

Original:
  PyAPI_FUNC(xxx) Py_Foo(xxx);

New version:
  PyAPI_FUNC(xxx) Py_FooEx(xxx, yyy);
  #define Py_Foo(x) Py_FooEx(x, NULL)

And similarly in the C file, but keeping the original PyFoo with something like:

  xxx Py_FooEx(...) { ... }
  #undef Py_Foo
   xxx Py_Foo(xxx) { return Py_FooEx(xxx, NULL); }

I'm not arguing against the LSB at all.  I think it's a good thing,
but I'm trying to understand what work needs to be done and how we can
acheive it in the face of apparent competing requirements.  I also
realize this is a one time cost and we don't need to address the
details right now.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LSB: Selection of a Python version

[Neal Norwitz]

On 12/5/06, Anthony Baxter <[EMAIL PROTECTED]> wrote:
>
> > So I think a public statement that we will support 2.4 with
> > security patches for a while longer (and perhaps with security
> > patches *only*) would be a good thing - independent of the LSB,
> > actually.
>
> Well, I don't know what sort of public statement you want to issue,
> but will this do? (Wearing my release manager hat)

I think we should document our intentions in the release PEP (or some
other PEP possibly).  Should we state that we will support version x.y
for N years after initial release or based on version numbers such as
x.(y-N) where y is the current version and N is a small number?

I'd prefer the time based approach because it seems much more common.
For reference here are some release dates:

2.2: 2001-12-21
2.3: 2003-07-29
2.4: 2004-11-30

If we stopped supporting 2.3, that would mean we support for 2-3 years.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for developer privileges.

[Neal Norwitz]

Hi Lars.

Thanks for all your work on tarfile!

Please send your ssh2 key to pydotorg at python.org as an attachment.
One of us will add your key.  Hopefully I can remember how to do it.
Here's some other info:  http://www.python.org/dev/faq/#subversion-svn

I can't add you to SF to be assigned bugs since I'm not an admin.  So
hopefully one of the admins can either add me as an admin or you as a
developer.

I already updated Misc/developers.txt.

n
--

On 12/20/06, Lars Gustäbel <[EMAIL PROTECTED]> wrote:
> Hello,
>
> my name is Lars Gustäbel (SF gustaebel). I contributed
> tarfile.py to the Python standard library in January 2003 and
> have been the maintainer since then. I have provided about 25
> patches over the years, most of them fixes, some of them new
> features and improvements. As a result, I am pretty familiar
> with the Python development process.
>
> If possible I would like to get developer privileges to be able
> to work more actively on tarfile.py for a certain time.
>
> I am currently implementing read-write POSIX.1-2001 pax format
> support. Development is still in progress, but it is already
> clear at this point, that it will be a complex change, which
> will definitely require some maintenance once it is finished and
> in day-to-day use. I would like to clean up the tarfile test
> suite during this process as well. The introduction of the pax
> format is important because it is the first tar specification
> that puts an end to those annoying limitations of the "original"
> tar format. It will become the default format for GNU tar some
> day.
>
> Thank you,
> Lars.
>
> --
> Lars Gustäbel
> [EMAIL PROTECTED]
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.5.1 plans

[Neal Norwitz]

I don't have a schedule in mind for 2.5.1 yet, however we should start
preparing for it.  The release will probably happen sometime in
January if everyone is available.  The branch has been pretty quiet,
so I'm not expecting too many problems.

Once we figure out a date for release I'll follow up here.

If you have any bugs or patches that should be addressed prior to
release, please assign them to me and bump the priority to 9.  I'm not
sure that there are any bugs which absolutely must be fixed before
release, though there are a few that would be nice.

Any other discussion for 2.5.1 necessary?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] smarter temporary file object (SF #415692)

[Neal Norwitz]

On 1/2/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I came across a complaint that PEP 0042 had become a graveyard of
> neglected ideas, and decided to have a look through and implement
> something.  Creating a smarter temporary file object seemed simple
> enough.
>
> Oddly, even after GvR re-opened it, I can't post an attachment to that
> tracker item

Hi Dustin.

Thanks for your patch!

Unfortunately, SF doesn't allow anyone except developers and the
tracker item owner to submit attachments.  The best thing to do would
be to create a new patch.  Then you can make a comment in the old RFE
that references your new patch.  Something like this is sufficient:

  Patch available here:  http://python.org/sf/NEW_PATCH_NUMBER

Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Renaming Include/object.h

[Neal Norwitz]

On 1/3/07, Thomas Wouters <[EMAIL PROTECTED]> wrote:
>
>
> On 1/3/07, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> > On 1/3/07, Fred L. Drake, Jr. <[EMAIL PROTECTED]> wrote:
> > > On Wednesday 03 January 2007 11:06, Martin v. Löwis wrote:
> > >  > In #1626545, Anton Tropashko requests that object.h should be
> > >  > renamed, because it causes conflicts with other software.
> > >  >
> > >  > I would like to comply with this requests for 2.6, assuming there
> > >  > shouldn't be many problems with existing software as object.h
> > >  > shouldn't be included directly, anyway.
> > >
> > > +1
> >
> > Maybe this should be done in a more systematic fashion? E.g. by giving
> > all "internal" header files a "py_" prefix?
>
> I was thinking the same, and I'm sure Neal Norwitz is/was too (he suggested
> this a few times in the past, at least outside of python-dev.)

Wow, I didn't realize I was that much of a broken record. :-)
I don't even remember talking to Thomas about it, only Guido.  I
definitely would like to see all private header files clearly denoted
by their name or directory.

I saw Jack's comment about Apple's naming scheme, but I'm ignoring
that for the moment.
I have bad memories from the Motif days of including everything with
one file.  I prefer to see includes with the directory.  This provides
a sort of namespace:

#include "python/foo.h"

Though, if the user only has to include a single Python.h like
currently, this wouldn't make as much sense.  I don't recall the same
problems in Python that existed when using Motif.

Here are some options (python/ can be omitted too):

  #include "python/public.h"
  #include "python/internal/foo.h"
  #include "python/private/foo.h"
  #include "python/_private.h"

I don't really like prefixing with py_ because from a user's
perspective I interpret py_ to be a public header that gives me a
namespace.  I prefer a directory that indicates its intent because it
can't be misunderstood.  IIRC Guido didn't want to introduce a new
directory.  In that case my second choice is to prefix the filename
with an underscore as a leading underscore often is interpreted as
private.

Going along with this change I would like to see all identifiers in
public header files prefixed with [_]Py.  And public header files
shouldn't be able to include private header files (by convention).  Or
we should have some other way of denoting which files must prefix all
their identifiers and which can use anything because they are only
used in Python code.

For example, right now node (struct _node) is exported in node.h in
2.4.  I think it moved in 2.5, but it's still exported IIRC.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5.1 plans

[Neal Norwitz]

The current schedule looks like it's shaping up to be:

Wed, Jan 24 for 2.5.1c1
Wed Jan 31 for 2.5.1

It would be great if you could comment on some of the bug reports
below.  I think several already have patches/suggested fixes.

These looks like they would be nice to fix::

http://python.org/sf/1579370 Segfault provoked by generators and exceptions
http://python.org/sf/1377858 segfaults when using __del__ and weakrefs

There is one outstanding issue I was notified of in private mail::

http://python.org/sf/1568240 Tix is not included in 2.5 for Windows

It's not clear to me if this should be fixed, but it's got a high priority::

http://python.org/sf/1467929 %-formatting and dicts

n
--
On 12/25/06, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> I don't have a schedule in mind for 2.5.1 yet, however we should start
> preparing for it.  The release will probably happen sometime in
> January if everyone is available.  The branch has been pretty quiet,
> so I'm not expecting too many problems.
>
> Once we figure out a date for release I'll follow up here.
>
> If you have any bugs or patches that should be addressed prior to
> release, please assign them to me and bump the priority to 9.  I'm not
> sure that there are any bugs which absolutely must be fixed before
> release, though there are a few that would be nice.
>
> Any other discussion for 2.5.1 necessary?
>
> n
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Renaming Include/object.h

[Neal Norwitz]

On 1/3/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz schrieb:
> > Wow, I didn't realize I was that much of a broken record. :-)
> > I don't even remember talking to Thomas about it, only Guido.  I
> > definitely would like to see all private header files clearly denoted
> > by their name or directory.
>
> What is a private header file, and does Python have any?
>
> I can see why Modules/sre.h is "private": it won't get installed at
> all, so users can't include them. For everything in Include, I think
> users can, and will, include them directly, unless they get them
> through Python.h.

By private, I mean internal only to python and don't need to prefix
their identifiers with Py and are subject to change without backwards
compatibility.  Include/graminit.h is one example of what I mean.
Some others are:  bitset.h, grammar.h, opcode.h, metagrammar.h,
errcode.h

Others are kinda questionable (they have some things that are
definitely public, others I'm not so sure about):  code.h, parsetok.h,
pyarena.h, longintrepr.h, osdefs.h, pgen.h, node.h

These are just some examples of what I mean.  These may be in include
because they are used between two top level directories, but not meant
to be exported.  There could be other reasons too.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Private header files (Was: Renaming Include/object.h)

[Neal Norwitz]

On 1/3/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz schrieb:
> > By private, I mean internal only to python and don't need to prefix
> > their identifiers with Py and are subject to change without backwards
> > compatibility.  Include/graminit.h is one example of what I mean.
> > Some others are:  bitset.h, grammar.h, opcode.h, metagrammar.h,
> > errcode.h
>
> Ah. This seems to be a requirement completely different from the
> one I'm talking about. By this definition, object.h is *not* an
> internal header file, yet I want it to be renamed.

Agreed.  I was mixing two things that aren't necessarily related
because I see the same possible solution.  I'm also using this one
example as a good opportunity to clean up more things.  Let me try to
explain a bit more below.

> As for this issue: how about moving all such private header files
> out of Include entirely? The parser-related ones should go into
> Parser, for example (grammar.h, bitset.h, graminit.h, metagrammar.h,
> errcode.h). This would leave us with opcode.h only.
>
> > Others are kinda questionable (they have some things that are
> > definitely public, others I'm not so sure about):  code.h, parsetok.h,
> > pyarena.h, longintrepr.h, osdefs.h, pgen.h, node.h
>
> Thomas said that at least code.h must stay where it is.
>
> What is the reason that you want them to be renamed?

Sorry, I wasn't trying to imply that these should necessarily be
renamed, only that the internal portions be moved elsewhere.  I guess
I should explain my mental model first which might make things
clearer.  Then again, I'm tired, so who knows if it will explain
anything. :-)

I'm a Python embedder and I want to know what's available to me.  I
look in Include and see a ton of header files.  Do I need all these?
What do I *need* and what can I *use*?  I only want to see the public
stuff that is available to me.  Thus I want anything that has
internal/implementation details/etc out of my sight to reduce my
learning curve.  I don't ever want to learn about something I won't
need nor include anything I won't need.

That's one part.

Another part of my mental model is that I'm a Python developer and I'm
modifying a header file that is implementation specific.  I need to
share it among different subdiretories (say Python and Objects).  So I
really need to stick the header file in a common place, Include/ is
it.

I don't want to export anything, but I don't know if other third party
developers will use the header or not.  Or maybe I need to include
some implementation details in another public header.  I'll probably
be lazy and just make a single header which has some internal and some
public stuff.

I want clear rules on when identifiers need to be prefixed.  If it's
internal (e.g., in an internal directory or prefixed with _), it can
have any name and can't be included from any non-internal header.  I
can also change it in a point release.  If anyone uses anything from
here, they are on their own.  If I see any identifier in a
non-internal header, it must be public and therefore prefixed with Py
or _Py.

The Python headers are pretty good about prefixing most things.  But
they could be better.  I think it gets harder to maintain without the
rules though.

Finally, by putting everything in directories and always including a
directory in the header file, like:

  #include "python/python.h"
  #include "python/internal/foo.h"

There can never be an include file name collision as what started this
thread.  It also provides a simple way of demonstrating what's public
and what is not.  It addresses all my complaints.  There are only a
few rules and they are simple.  But I am addressing several points
that are only loosely related which I what I think generated some
confusion.

Adding the directory also makes clear were the header file comes from.
 If you see:

  #include "node.h"

you don't know if that's a python node.h, from some other part of the
code or a third party library.

Not to try to confuse things even more, but I will point out something
Google does that is only indirectly related.  Google requires
importing modules.  You aren't supposed to import classes.  You
wouldn't do:

  from foo.bar import Message
  # ...
  msg = Message(...)

You would do:

  from foo import bar
  # ...
  msg = bar.Message(...)

This makes it clear where Message comes from, just like adding a
python prefix to all header file names makes it clear where the header
file lives.  Both are good for traceability, though in different ways.
 This technique makes it easier to manage larger code bases,
particularly when there are multiple libraries used.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.5.1 plans

[Neal Norwitz]

On 1/4/07, A.M. Kuchling <[EMAIL PROTECTED]> wrote:
> On Thu, Jan 04, 2007 at 10:22:54AM -0800, Mike Klaas wrote:
> > >  [ 1598181 ] subprocess.py: O(N**2) bottleneck
> > >
> > >I submitted the trivial fix almost two months ago, but apparently nobody
> > >feels responsible...
>
> Is Peter Astrand still actively maintaining the module?  I've been
> assigning subprocess bugs to him on the theory that he'll fix them.
> If he's not around, we can apply patches to subprocess.  (The problem
> with subprocess in particular would be the difficulty of testing
> changes that affect both Windows and POSIX code; I think relatively
> few developers use both Windows and Unix.)

We have the buildbots to help with this.

> We should also try to make an effort to go through the tracker and
> look for bug fixes and patches suitable for 2.5.1.

Definitely!  I only did a really quick review.  If you want someone to
think about any of these, assign them to me with a priority of 9.
I'll discuss with others and see if the fixes should go in to 2.5.1 or
not.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] crashing on shutdown due to warning at shutdown

[Neal Norwitz]

I fixed the crash that was due to raising a warning on shutdown.  I
have heard about crashes at shutdown and wonder if this was the cause.
 There might be similar bugs lurking that assume PyModule_GetDict()
always returns a valid pointer.  It doesn't, it can return NULL.

I'm not sure if the original problem existed or not, but after this
fix on shutdown, I see:

warning: DBTxn aborted in destructor.  No prior commit() or abort().

I don't know if this is a real problem, a test problem, or what.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] updating Misc/NEWS

[Neal Norwitz]

I've noticed a bunch of changes recently without corresponding items
added to Misc/NEWS.  Can everyone update NEWS especially when fixing
bugs or adding new features?If you have made recent checkins, it
would be great if you could go back and update Misc/NEWS if you missed
it the first time.

Thanks,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buglet in long("123\0", 10)

[Neal Norwitz]

SVN rev 52305 resolved Bug #1545497: when given an explicit base,
int() did ignore NULs embedded in the string to convert.

However, the same fix wasn't applied for long().

n

On 1/13/07, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> What's wrong with this session? :-)
>
> Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17)
> [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> int('123\0')
> Traceback (most recent call last):
>   File "", line 1, in 
> ValueError: null byte in argument for int()
> >>> int('123\0', 10)
> Traceback (most recent call last):
>   File "", line 1, in 
> ValueError: invalid literal for int() with base 10: '123\x00'
> >>> long('123\0')
> Traceback (most recent call last):
>   File "", line 1, in 
> ValueError: null byte in argument for long()
> >>> long('123\0', 10)
> 123L
> >>>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] adding _Py prefix to names in 2.5.1?

[Neal Norwitz]

http://python.org/sf/1637022 points out a problem caused by the lack
of a _Py prefix on Ellipsis.  Most (all?) of the new AST names are not
prefixed.  These are all meant to be internal names.  Are there any
issues with changing this?  If we do so, it means that any module
built with 2.5 that is using these names will fail to work in 2.5.1.
No code outside the core *should* be using these names.

Assuming there is no dissent, does some budding Python developer want
to take on a patch?  This should be pretty straightforward.  Feel free
to mail me off list if you prefer.

Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pythn 2.5 buildbots do not build anymore?

[Neal Norwitz]

On 1/17/07, Thomas Heller <[EMAIL PROTECTED]> wrote:
> Why don't the Python 2.5 buildbots build anymore?

It looks like there were no checkins that caused a build.  Changing
doc or Misc/NEWS is excluded.  I'm guessing that there is a bug in
buildbot that causes it to miss any changes from svnmerge.  Those
should have been caught ISTM.

I just checked in a change and the build looks like it's going.

http://www.python.org/dev/buildbot/all/changes/521

Thomas, you might want to look at that, cause it's something I don't
understand in cytpes. :-)

In other buildbot news, I fixed the failures on windows through some
lucky guessing.  I'm not sure the fixes are optimal, so someone with
more Windows knowledge should take a look at them.

I also got the alpha tru64 buildbot running again.  We still have a
few down, but they generally look in decent shape.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Object creation hook

[Neal Norwitz]

Short of using a memory debugger such as Valgrind or Purify, have you
considered looking for reference leaks?  These may be the cause and
can be checked with pure python code.  See how Lib/test/regrtest.py
handles the -R option.

n
--

On 1/24/07, Kristján V. Jónsson <[EMAIL PROTECTED]> wrote:
> Thanks, but the question is really, "how do I build a better debug hook than 
> sys.getobjects?" so I argue this is a valid python-dev question.
>
> We have been using gc.get_objects() but it has several problems:
> 1) It returns all objects in the system.  This results in a list so long that 
> it often kills the system.  Our system is of a scale that makes this very 
> risky.
> 2) There is no way to frame certain operations and get just those objects 
> that were created during their execution.  In our case, we would like to get 
> the server cluster running, then frame a certain operation to get a callback 
> for all created objects, so that we could track that they were later 
> destroyed correctly.  I have done this previously by storing the id()s of all 
> objects returned from gc.get_objects() and comparing them "before" and 
> "after" but this suffers from 1) above, and the ambiguity of id() in the face 
> of objects being created and destroyed.
>
>
> Working with the metaclasses sounds reasonable if one has the luxury of 
> changing the entire codebase to use a different metaclass.  It also doesn't 
> work with old style classes (plenty of those), c extensions, and builtins.
>
> (I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from 
> a python script, I'm fairly sure that is possible in Ruby :)  but I 
> digress...)
>
> My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding 
> this final line:
> if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\
>
> The function then looks like this:
>
> void PyCCP_CreationHookFunc(PyObject * obj)
> {
> if (PyCCP_CreationHook) {
> PyObject *result, *tmp = PyCCP_CreationHook;
> PyCCP_CreationHook = 0; //guard against recursion
> result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook, 
> obj, 0);
> Py_XDECREF(result);
> if (!result)
> PyErr_Clear();
> PyCCP_CreationHook = tmp;
> }
> }
>
> Doing this, and setting a no-op function as as the PyCCP_CreationHook, does 
> seem to work for a while in the interactive python shell, but it then crashes 
> and I haven't been able to work out the crash.  In any case, doing stuff at 
> the point of GC list insertion is very hairy, especially in the face of 
> __del__ methods and such (of which we don't have many)
>
> I am waiting to get Rational Purify set up on my machine and then I'll maybe 
> be able to understand the crash case better.
>
> Cheers,
> Kristján
>
>
> -Original Message-
> From: "Martin v. Löwis" [mailto:[EMAIL PROTECTED]
> Sent: 23. janúar 2007 23:32
> To: Kristján V. Jónsson
> Cc: 'python-dev@python.org'
> Subject: Re: [Python-Dev] Object creation hook
>
> Kristján V. Jónsson schrieb:
> > I am trying to insert a hook into python enabling a callback for all
> > just-created objects.  The intention is to debug and find memory leaks,
> > e.g. by having the hook function insert the object into a WeakKeyDictionary.
>
> I'd like to point out that this isn't a python-dev question, but more
> appropriate for comp.lang.python (as it is of the "how do I x with
> Python?" kind).
>
> I would use a debug build, and use sys.getobjects to determine all
> objects and find leaks.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.5.1 release postponed

[Neal Norwitz]

As you may have noticed we missed the schedule for getting 2.5.1 out.
Unfortunately several of us are having scheduling problems with making
this release.  Since there don't seem to be major problems with the current
2.5 release, our current plan is to defer release until April when
everyone should have more time to do it right.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3000 upgrade path

[Neal Norwitz]

On 2/25/07, Thomas Wouters <[EMAIL PROTECTED]> wrote:
>
> It's about how we get Python 2.x to 3.0, and howmuch of 3.0 we put into 2.6 
> and later.

I've also talked to a bunch of people at PyCon, including Thomas.
There seems to be much concern (rightfully so!) about the upgrade path
from 2.x to 3.x.  I don't think it will be easy to navigate, but I
have confidence we can do a great job.

My goal is to rip out as much cruft from the code base for 3k to make
it easier to maintain.  In order for users to adopt 3k, it must
provide real benefit.  In addition, it as painless as possible for
third party developers to support both 2.x and 3.x.  All the
developers I talked to expressed relief that we weren't forgetting
about them.  There's still unease and will be until we demonstrate
what we plan to do.  They were satisfied with our general approach.

The time schedules in PEP 361 (2.6 release schedule) and what Guido
has said for 3k (from what I remember) are roughly:

  April 2007 - 3.0 PEPs and features accepted/decided
 June 2007 - 3.0a1 - basic (most) features implemented
 Dec 2007 - 2.6a1
 Apr 2008 - 2.6 final
 July 2008 - 3.0 final

Even if we slip these schedules, as long as we keep them in relative
order, we can keep 2.6 robust, while also offering many of the 3k
features.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3000 upgrade path

[Neal Norwitz]

On 2/25/07, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
> Neal Norwitz <[EMAIL PROTECTED]> wrote:
> > The time schedules in PEP 361 (2.6 release schedule) and what Guido
> > has said for 3k (from what I remember) are roughly:
> >
> >   April 2007 - 3.0 PEPs and features accepted/decided
> >  June 2007 - 3.0a1 - basic (most) features implemented
>
> Any talk at PyCon regarding the new IO system?  That looks like the
> biggest piece of unfinished Py3k work.

AFAIK, there hasn't been any work on the new IO system or str/unicode
unification.  Guido asked for help on these, but I don't know if there
were any takers.  Sprints will be held over the next several days, so
hopefully there will be some work in these areas.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Renaming Include/object.h

[Neal Norwitz]

On 2/25/07, Jeremy Hylton <[EMAIL PROTECTED]> wrote:
> On 1/3/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > In #1626545, Anton Tropashko requests that object.h should be
> > renamed, because it causes conflicts with other software.
> >
> > I would like to comply with this requests for 2.6, assuming there
> > shouldn't be many problems with existing software as object.h
> > shouldn't be included directly, anyway.
>
> I like the idea of renaming the header files, but I expect there is a
> lot more opportunity for breaking code that you estimate.  I did a
> search on google.com/codesearch and found seven external packages that
> include Python.h and object.h (before I gave up).

So rather than a simple rename, we should probably rename, change all
references in the core to use the new name, and make a simple object.h
that only does:

#include "new_object.h"

> I assume we expect people won't include it, because it is also
> included in object.h.  But they do it.  Is there documentation that
> says you shouldn't import it?

I thought somewhere (embedding/extending doc?) it mentions that only
Python.h should be included, but if we have a naming
convention/directory structure, this becomes more obvious.

Doc/ext/extending.tex:

To support extensions, the Python API (Application Programmers
Interface) defines a set of functions, macros and variables that
provide access to most aspects of the Python run-time system.  The
Python API is incorporated in a C source file by including the header
\code{"Python.h"}.

(But it may not say nothing else should be included.)

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encouraging developers

[Neal Norwitz]

On 3/6/07, Phil Thompson <[EMAIL PROTECTED]> wrote:
> On Tuesday 06 March 2007 5:49 am, Martin v. Löwis wrote:
> > Phil Thompson schrieb:
> > > I'm not sure what your point is. My point is that, if you want to
> > > encourage people to become core developers, they have to have a method of
> > > graduating through the ranks - learning (and being taught) as they go. To
> > > place a very high obstacle in their way right at the start is completely
> > > counter-productive.
> >
> > And please be assured that no such obstacle is in the submitters way.
> > Most patches are accepted without the submitter actually reviewing any
> > other patches.
>
> I'm glad to hear it - but I'm talking about the perception, not the fact. When
> occasionally submitters ask if their patch is going to be included, they will
> usually get a response suggesting they review other patches. That will only
> strengthen the perception.
>
> This discussion started because the feeling was expressed that it was
> difficult to get patches accepted and that new core developers were not being
> found. I would love to contribute more to the development of Python - I owe
> it a lot - but from where I stand (which is most definitely not where you
> stand) I can't see how to do that in a productive and rewarding way.

I recognize there is a big problem here.  Each of us as individuals
don't scale.  So in order to get stuff done we need to be more
distributed.  This means distributing the workload (partially so we
don't burn out).  In order to do that we need to distribute the
knowledge.  That probably involves some changes.

I understand it's really hard to get involved.  It can be frustrating
at times.  But I think the best way is to find a mentor.  Find an
existing core developer that you relate to and/or have similar
interests with.  I've been trying to do this more.

So here's my offer.  Anyone who is serious about becoming a Python
core developer, but doesn't know how to get involved can mail me.
Privately, publicly, it doesn't matter to me.  I will try to mentor
you.

Be prepared!  I will demand submissions are high quality which at a
minimum means:

 - it adds a new test, enhances an existing test or fixes a broken test
 - it has *all* the required documentation, including
versionadded/versionchanged
 - most people think the feature is desirable
 - the code is maintainable and formatted according to the prevailing style
 - we follow the process (which can include improving the process if
others agree)
ie, there's no free lunch, unless you work at Google of course :-)

It also means you are willing to hold other people up to equally high standards.

Your contributions don't have to be code though.  They can be doc,
they can be PEPs, they can be the python.org website.  It could even
be helping out with Google's Summer of Code.  The choice is yours.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com