Re: [Python-Dev] Capturing PyRun_String stdout
Hi, 2015-06-30 21:40 GMT+02:00 Chris Moller : > I expect this has been asked before, but I can't find out much about it... > Please ask this kind of questions on the python-users mailing list <https://www.python.org/community/lists/> (or comp.lang.python). There you will find helpful people who will tell you how to modify sys.stdout, and the differences between exec() and eval(). Best of luck using Python! > I'm trying to embed Python as a scripting language and I need to capture > the output of PyRun_String(), PyEval_EvalCode(), or whatever as a char * > (or wchar_t * or whatever) rather than have it go to stdout. > > Python 3.3.2 under plain C, not C++ > > And, while I'm interrupting everyone's afternoon, another question: if I > pass Py_single_input to PyRun_String() or > Py_CompileString()/PyEval_EvalCode(), it accepts statements like "a=10" and > can then properly do stuff like "print(a)". If I use Py_eval_input > instead, I get error messages. In both cases, I'm using the same > global_dict and local_dict, if that makes any difference. What am I doing > wrong? > > Thanks, > Chris Moller > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/amauryfa%40gmail.com > > -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling string interning for null and single-char causes segfaults
2013/3/4 Serhiy Storchaka
> On 01.03.13 17:24, Stefan Bucur wrote:
>
>> Before digging deeper into the issue, I wanted to ask here if there are
>> any implicit assumptions about string identity and interning throughout
>> the interpreter implementation. For instance, are two single-char
>> strings having the same content supposed to be identical objects?
>>
>
> I think this is not a bug if the code relies on the fact that an empty
> string is a singleton. This obviously is an immutable object and there is
> no public method to create different empty string.
Really?
>>> x = u'\xe9'.encode('ascii', 'ignore')
>>> x == '', x is ''
(True, False)
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling string interning for null and single-char causes segfaults
2013/3/4 Guido van Rossum
> >>>> x = u'\xe9'.encode('ascii', 'ignore')
> >>>> x == '', x is ''
> > (True, False)
>
> Code that relies on this is incorrect (the language doesn't guarantee
> interning) but nevertheless given the intention of the implementation,
> that behavior of encode() is also a bug.
>
The example above is obviously from python2.7; there is a similar example
with python3.2:
>>> x = b'\xe9\xe9'.decode('ascii', 'ignore')
>>> x == '', x is ''
(True, False)
...but this bug has been fixed in 3.3: PyUnicode_Resize() always returns
the unicode_empty singleton.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Difference in RE between 3.2 and 3.3 (or Aaron Swartz memorial)
Hi, 2013/3/6 Matěj Cepl > > On 2013-02-26, 16:25 GMT, Terry Reedy wrote: > > On 2/21/2013 4:22 PM, Matej Cepl wrote: > >> as my method to commemorate Aaron Swartz, I have decided to port his > >> html2text to work fully with the latest python 3.3. After some time > >> dealing with various bugs, I have now in my repo > >> https://github.com/mcepl/html2text (branch python3) working solution > >> which works all the way to python 3.2 (inclusive; > >> https://travis-ci.org/mcepl/html2text). However, the last problem > >> remains. This > >> > >> Run this command: > >> ls -l *.html > >> ? > >> > >> should lead to > >> > >>* Run this command: > >> > >> ls -l *.html > >> > >>* ? > >> > >> but it doesn’t. It leads to this (with python 3.3 only) > >> > >> * Run this command: > >>ls -l *.html > >> > >> * ? > >> > >> Does anybody know about something which changed in modules re or > >> http://docs.python.org/3.3/whatsnew/changelog.html between 3.2 and > >> 3.3, which could influence this script? > > > > Search the changelob or 3.3 misc/News for items affecting those two > > modules. There are at least 4. > > http://docs.python.org/3.3/whatsnew/changelog.html > > > > It is faintly possible that the switch from narrow/wide builds to > > unified builds somehow affected that. Have you tested with 2.7/3.2 on > > both narrow and wide unicode builds? > > So, in the end, I have went the long way and bisected cpython to > find the commit which broke my tests, and it seems that the > culprit is http://hg.python.org/cpython/rev/123f2dc08b3e so it is > clearly something Unicode related. > > Unfortunately, it really doesn't tell me what exactly is broken > (is it a known regression) and if there is known workaround. > Could anybody suggest a way how to find bugs on > http://bugs.python.org related to some particular commit (plain > search for 123f2dc0 didn’t find anything). > I strongly suspect an incorrect usage of the "is" operator: https://github.com/mcepl/html2text/blob/master/html2text.py#L95 Identity of strings is not guaranteed... Does it change something if you use "==" instead? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
2013/3/27 Andrew Svetlov > No. _decimal is new functionality that will never be backported. > > On Wed, Mar 27, 2013 at 9:40 AM, Andriy Kornatskyy > wrote: > > Any plans backport decimal C implementation from 3.3? > No. 2.7 does not accept new features anymore, but you can install this backport from PyPI: https://pypi.python.org/pypi/cdecimal/2.3 -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
2013/3/27 Vinay Sajip
> When you mount a wheel, its absolute path name is added to
> sys.path, allowing the Python code in it to be imported.
>
Better: just put the wheel path to sys.path
sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
and let a sys.path_hook entry do the job.
Such a WheelImporter could even inherit from zipimporter, plus the magic
required for C extensions.
It avoids the mount/nomount methods, only the usual sys.path operations are
necessary from the user.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyPy, Jython, & IronPython: Enum convenience function and pickleablity
2013/5/2 Guido van Rossum > On Thu, May 2, 2013 at 1:18 PM, fwierzbi...@gmail.com > wrote: > > On Thu, May 2, 2013 at 12:07 PM, Ethan Furman > wrote: > >> In order for the Enum convenience function to be pickleable, we have > this > >> line of code in the metaclass: > >> > >> enum_class.__module__ = sys._getframe(1).f_globals['__name__'] > >> > >> This works fine for Cpython, but what about the others? > > This should work for Jython, but I can't say I like it. I believe > > IronPython has a sort of speedup mode that disallows the use of > > _getframe, and I'd like to add this to Jython someday. > > This particular function is typically only called at module load time, > so speeding it up isn't worth it. It works fine on PyPy as well. It probably also kills any JIT optimization, but it's not an issue since classes are not usually created in tight loops. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Serhiy Storchaka > Now with enumerations in the stdlib the stat module constants are > candidates for flag enumerations. How easy will be implement it on C? Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] add new lambda syntax
2013/6/28 Pynix Wang
> I want use coffeescript function syntax to write python lambda expression
> so I modified the Grammar file.
>
> ```
> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' |
>'[' [testlist_comp] ']' |
>'{' [dictorsetmaker] '}' |
>NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text
> ```
>
> but when I write
> ```
> (x,y=1)->x+y
> ```
> the parser doesn't go into vararglist.
>
This grammar is not LL(1) anymore (it's probably LALR now)
when seeing "x", it has the choice between testlist_comp and vararglist,
and the first one is picked.
Python's parser generator only supports LL(1) grammars.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why are captured parameters also listed in co_varnames?
2013/7/2 Andrea Griffini > I'm trying to understand how CPython implements closure variable capture > and there is one minor point I can't understand. > > When a local is captured it gets allocated in co_cellvars and is accessed > with (LOAD|STORE)_DEREF, and this is clear. > However when a local is coming from a parameter it gets ALSO allocated in > co_varnames even if the local slot apparently is not accesible because > *_FAST opcodes are not generated. > > Is there a technical reason for this? It happens in CPython 2, 3 and even > in PyPy... > > co_varnames is also used in error messages, for example in the following code: >>> def f(): ... def g(): ... x ... print x ... x = 1 ... return g ... >>> f() UnboundLocalError: local variable 'x' referenced before assignment This is also needed when x is a parameter of f(), for inspect.signature of course, but also because in python3 you can "del x". -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] EINTR handling...
2013/8/30 Charles-François Natali > Hello, > > This has been bothering me for years: why don't we properly handle > EINTR, by running registered signal handlers and restarting the > interrupted syscall (or eventually returning early e.g. for sleep)? > > EINTR is really a nuisance, and exposing it to Python code is just > pointless. > I agree. Is there a way to see in C code where EINTR is not handled? Or a method to handle this systematically? > Now some people might argue that some code relies on EINTR to > interrupt a syscall on purpose, but I don't really buy it: it's highly > non-portable (depends on the syscall, SA_RESTART flag...) and subject > to race conditions (it it comes before the syscall or if you get a > partial read/write you'll deadlock). > > Furthermore, the stdlib code base is not consistent: some code paths > handle EINTR, e.g. subprocess, multiprocessing, sock_sendall() does > but not sock_send()... > Just grep for EINTR and InterruptedError and you'll be amazed. > > GHC, the JVM and probably other platforms handle EINTR, maybe it's > time for us too? > > Just for reference, here are some issues due to EINTR popping up: > http://bugs.python.org/issue17097 > http://bugs.python.org/issue12268 > http://bugs.python.org/issue9867 > http://bugs.python.org/issue7978 > http://bugs.python.org/issue12493 > http://bugs.pythoto see n.org/issue3771 <http://bugs.python.org/issue3771> > > > cf > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/amauryfa%40gmail.com > -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_io fails on test_1686475
Hello, On Sun, Mar 1, 2009 at 23:04, Cesare Di Mauro wrote: > Running the test suite with Python 2.6.1 32 bit (compiled in DEBUG mode > with Visual Studio Express Edition 2008) on Vista x64, I've got an assert > error: > > test_1686475 (__main__.StatAttributeTests) ... Assertion failed: > (__int64)(int)((in / 1000) - secs_between_epochs) == ((in / 1000) > - secs_between_epochs), file ..\Modules\posixmodule.c, line 790 > > I have no idea about this failure. Any hint? The failing assertion comes from this code in posixmodule.c: /* XXX Win32 supports time stamps past 2038; we currently don't */ *time_out = Py_SAFE_DOWNCAST((in / 1000) - secs_between_epochs, __int64, int); the test (btw, it's in test_os.py) is trying os.stat(r"c:\pagefile.sys") Can you please check the three time stamps of this file (creation, update, access)? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_io fails on test_1686475
On Mon, Mar 2, 2009 at 13:25, Nick Coghlan wrote: > Cesare Di Mauro wrote: >> However, they are correct timestamps for Windows files, so I think that at >> least >> the API on posixmodule.c should not fail when working with them. I don't >> know if >> there's a way to handle them correctly. > > Use 64-bit time values (which is highly unlikely to ever be the case for > CPython on a 32-bit OS). 64bit time_t is the default since VS2005. See the patch at http://bugs.python.org/issue4379 -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interpreter crash in test_strftime?
On Tue, Mar 3, 2009 at 00:19, Antoine Pitrou wrote: > Hello everybody, > > I'm trying the current py3k under a Windows virtual machine (with VS Express > 2008), and the interpreter crashes in test_strftime in test_datetime.py. > Has anyone been getting something like that? > > (not being a Windows user, it's a bit hard for me to investigate what's wrong) Let me guess: it a RuntimeError, "Invalid format directive", when calling strftime. CRT assertions where enabled a few weeks ago (issue4804) It may be a bad merge from trunk to the py3k branch. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interpreter crash in test_strftime?
On Tue, Mar 3, 2009 at 00:47, Antoine Pitrou wrote: > > Hi Amaury, > > Le mardi 03 mars 2009 à 00:39 +0100, Amaury Forgeot d'Arc a écrit : >> > >> > I'm trying the current py3k under a Windows virtual machine (with VS >> > Express >> > 2008), and the interpreter crashes in test_strftime in test_datetime.py. >> > Has anyone been getting something like that? >> > >> > (not being a Windows user, it's a bit hard for me to investigate what's >> > wrong) >> >> Let me guess: it a RuntimeError, "Invalid format directive", when >> calling strftime. > > I don't know. I've built in release mode, and I only get a window > proposing me to send the crash information to Microsoft... > (I must mention that I recently upgraded that VM to XP SP3) This should be fixed with r70114. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate lxml into the stdlib?
On Fri, Mar 6, 2009 at 22:10, "Martin v. Löwis" wrote: >>> libs/_sqlite3.lib 2K >> >> I think this is a summary of the entry points into one of the above >> DLLs for the benefit of other code wanting to link against it, but I'm >> not sure. > > Correct. I don't know why I include them in the MSI - they are there > because they were also shipped with the Wise installer. I see no > use - nobody should be linking against an extension module. They even cause trouble. Just yesterday I (well, not me: the pypy translation process) was caught by the presence of the "bz2.lib" file, which pypy found there, just because the linker lists c:\python25\LIBs before other directories. Of course the real bz2.lib, which defines the compression routines, was installed somewhere else, and compilation failed. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OSError.errno => exception hierarchy?
Hello, On Thu, Apr 2, 2009 at 22:35, Jack diederich wrote: > On Thu, Apr 2, 2009 at 4:25 PM, Benjamin Peterson wrote: >> 2009/4/2 Gustavo Carneiro : >>> Apologies if this has already been discussed. >> >> I don't believe it has ever been discussed to be implemented. >> >>> Apparently no one has bothered yet to turn OSError + errno into a hierarchy >>> of OSError subclasses, as it should. What's the problem, no will to do it, >>> or no manpower? >> >> Python doesn't need any more builtin exceptions to clutter the >> namespace. Besides, what's wrong with just checking the errno? > > The problem is manpower (this has been no ones itch). In order to > have a hierarchy of OSError exceptions the underlying code would have > to raise them. That means diving into all the C code that raises > OSError and cleaning them up. > > I'm +1 on the idea but -1 on doing the work myself. > > -Jack The py library (http://codespeak.net/py/dist/) already has a py.error module that provide an exception class for each errno. See for example how they use py.error.ENOENT, py.error.EACCES... to implement some kind of FilePath object: http://codespeak.net/svn/py/dist/py/path/local/local.py But I'm not sure I would like this kind of code in core python. Too much magic... -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Thu, Apr 2, 2009 at 03:23, Christian Heimes wrote: > John Ehresman wrote: >> * To what extent should non-debugger code use the hook? At one end of >> the spectrum, the hook could be made readily available for non-debug use >> and at the other end, it could be documented as being debug only, >> disabled in python -O, & not exposed in the stdlib to python code. > > To explain Collin's mail: > Python's dict implementation is crucial to the performance of any Python > program. Modules, types, instances all rely on the speed of Python's > dict type because most of them use a dict to store their name space. > Even the smallest change to the C code may lead to a severe performance > penalty. This is especially true for set and get operations. A change that would have no performance impact could be to set mp->ma_lookup to another function, that calls all the hooks it wants before calling the "super()" method (lookdict). This ma_lookup is already an attribute of every dict, so a debugger could trace only the namespaces it monitors. The only problem here is that ma_lookup is called with the key and its hash, but not with the value, and you cannot know whether you are reading or setting the dict. It is easy to add an argument and call ma_lookup with the value (or NULL, or -1 depending on the action: set, get or del), but this may have a slight impact (benchmark needed!) even if this argument is not used by the standard function. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting values stored inside sets
Hi, On Fri, Apr 3, 2009 at 17:45, Sebastian Rittau wrote: > I am missing a simple way to retrieve the "first" element of any > iterable in python that matches a certain condition anyway. Something > like this: > > def first(iter, cb): > for el in iter: > if cb(el): > return el > raise IndexError() > > Or (shorter, but potentially slower): > > def first(iter, cb): > return [el for el in iter if cb(el)][0] > > To be used like this: > > my_el = first(my_set, lambda el: el == "foobar") > > This is something I need from time to time and this also seems to solve > your problem. def first(iter, cb): return itertools.ifilter(cb, iter).next() -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does read() return bytes instead of bytearray?
Hello, On Wed, Apr 15, 2009 at 03:01, Dan Eloff wrote: > Hi, > > Can someone please explain why read() should return an immutable bytes > type instead of a mutable bytearray? It's not like read() from a file > and use buffer as a key in a dict is common. Certainly read() from > file or stream, modify, write is very common. I don't understand why > the common case pays the price in performance and simplicity. It > seemed to me that the immutable bytes was described as being useful in > niche situations, but it actually seems to have been favored over > bytearray in Python 3. > > Was there was a good reason for this decision? Or was this just an > artifact in the change to two bytes types? No, the read() method did not change from the 2.x series. It returns a new object on each call. > The reason I ask is I have a server application that is mostly stream > reading/writing on the hot path and in Python 2.5 the redundant copies > add up to a significant overhead, (I estimate as much as 25% from my > measurements) I was looking at Python 3 as a way to solve that > problem, but unfortunately it doesn't look like it will help. Files opened in binary mode have a readinto() method, which fills the given bytearray. Is this what you are looking for? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue5434: datetime.monthdelta
On Thu, Apr 16, 2009 at 10:45, wrote:
> >>> date(2008, 1, 30) + monthdelta(1)
> datetime.date(2008, 2, 29)
>
> What would this loop would print?
>
> for d in range(1, 32):
> print date(2008, 1, d) + monthdelta(1)
>
> I have this funny feeling that arithmetic using monthdelta wouldn't always
> be intuitive.
FWIW, the Oracle database has two methods for adding months:
1- the add_months() function
add_months(to_date('31-jan-2005'), 1)
2- the ANSI interval:
to_date('31-jan-2005') + interval '1' month
"add_months" is calendar sensitive, "interval" is not.
"interval" raises an exception if the day is not valid for the target
month (which is the case in my example)
"add_months" is similar to the proposed monthdelta(),
except that it has a special case for the last day of the month:
"""
If date is the last day of the month or if the resulting month has
fewer days than the day
component of date, then the result is the last day of the resulting month.
Otherwise, the result has the same day component as date.
"""
indeed:
add_months(to_date('28-feb-2005'), 1) == to_date('31-mar-2005')
In my opinion:
arithmetic with months is a mess. There is no such "month interval" or
"year interval" with a precise definition.
If we adopt some kind of month manipulation, it should be a function
or a method, like you would do for features like last_day_of_month(d),
or following_weekday(d, 'monday').
date(2008, 1, 30).add_months(1) == date(2008, 2, 29)
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] string to float containing whitespace
Hi,
2009/4/29 :
> Someone please tell me I'm not going mad. I could have sworn that once upon
> a time attempting to convert numeric strings to ints or floats if they
> contained whitespace raised an exception. As far back as 1.5.2 it appears
> that float(), string.atof() and string.atoi() allow whitespace. Maybe I'm
> thinking of trailing non-numeric, non-whitespace characters.
You are maybe referring to the Decimal constructor:
decimal.Decimal(" 123")
fails with 2.5, but works with 2.6. (issue 1780)
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed: drop unnecessary "context" pointer from PyGetSetDef
Hi, Larry Hastings wrote: > > Mark Dickinson wrote: >> >> Still, binary compatibility seems like a fairly strong reason not to >> remove the closure field. > > My understanding is that there a) 2.x extension modules are not binary > compatible with 3.x, and b) there are essentially no 3.x extension modules > in the field. Is that accurate? If we don't have an installed base (yet) > to worry about, now's the time to make this change. cx_Oracle at least uses this closure field, and has already been ported to 3.x: http://www.google.com/codesearch?q=Connection_SetOCIAttr+trunk -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Serious regression in doctest in Py3.1rc1
Hello,
2009/6/3 Stefan Behnel :
> Hi,
>
> I can't currently file a bug report on this, but I was told by Lisandro
> Dalcín that there is a serious problem with the doctest module in Py3.1rc1.
> In Cython, we use doctests to test the compiler in that we compile a
> Python/Cython module with doctests into a C module and then run doctest on
> the imported extension module.
>
> >From the error report it seems to me that doctest is now trying to read the
> module itself through linecache for some reason, which horribly fails for a
> binary module.
>
> Could someone please look into this? I'll open up a bug report tomorrow
> unless someone beats me to it.
I don't have the time either, but the problem looks very similar to
http://bugs.python.org/issue4050
The fix was to replace:
file = inspect.getsourcefile(object) or inspect.getfile(object)
was replaced with
file = inspect.getsourcefile(object)
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cannot set PYTHONPATH with big paths with Python 3.0 and 3.1
Hi, 2009/6/9 Antoine Pitrou : > Martin v. Löwis v.loewis.de> writes: >> In absence of a patch, it can't be a release blocker, IMO. > > I think we've had lots of issues filed as release blockers while they still > didn't have a patch. > As for whether this particular bug deserves to be a blocker, I don't know. It > is > certainly annoying, however. FYI, I just submitted a patch. It is simple enough to be considered for inclusion for rc2. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Define metatype and a type that uses it
Hello, 2009/7/8 Erik Groeneveld : > I am working on an extension that allows one to open Java .jar files > and use the objects defined in it (via CNI or JNI) in Python. It > works like a charm, but I am stuck with the definition of a metatype > for the wrappers. I have to do this in Python now, with a helper > class. > > I did find examples of how to define metatypes, but I can't find > anything on how to declare one type using this metatype. > > Could anyone point me to examples or documentation or help me fix my code > below? [...] There are several changes to do in your code: - Don't define a JObjectMeta struct, use JObjectType directly instead. An instance of the metatype is the type itself! - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base type (the correct value would be sizeof(PyHeapTypeObject), this is important in order to create derived classes in python) - Don't set JObjectMetaType.tp_new, let it inherit from the base type (PyType_GenericNew just allocates memory; types are more complex...) - JObjectType.tp_base = &JObjectMetaType" is wrong, it should be "JObjectType.ob_type = &JObjectMetaType;", or better to be compatible with python3: "Py_TYPE(JObjectType) = &JObjectMetaType;" In summary, I made your example work with: JObjectMetaType.tp_name = "metaclass.JObjectMeta"; JObjectMetaType.tp_flags= Py_TPFLAGS_DEFAULT; JObjectMetaType.tp_doc = "JObjectMeta objects"; JObjectMetaType.tp_init = JObjectMeta_init; JObjectMetaType.tp_base = &PyType_Type; JObjectMetaType.tp_getattro = JObjectMeta_getattro; Py_TYPE(&JObjectType) = &JObjectMetaType; JObjectType.tp_name = "metaclass.JObject"; JObjectType.tp_basicsize= sizeof(JObject); JObjectType.tp_flags= Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE; JObjectType.tp_doc = "JObject objects"; JObjectType.tp_init = JObject_init; -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Define metatype and a type that uses it
Hi,
2009/7/13 Erik Groeneveld :
> Amaury,
>
> Thank you very much for your detailed explanation. It helps to
> understand it better, and it mostly works now. There is one thing
> however:
>
> On Wed, Jul 8, 2009 at 17:35, Amaury Forgeot d'Arc wrote:
>> - Don't define a JObjectMeta struct, use JObjectType directly instead.
>> An instance of the metatype is the type itself!
>> - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base
>> type (the correct value would be sizeof(PyHeapTypeObject), this is
>> important in order to create derived classes in python)
>
> I'd like to add a C pointer field to the metatype instance (the type).
> So, contrary to your advice, I have defined:
>
> typedef struct {
> PyHeapTypeObject x;
> void* p;
> } JObjectMeta;
>
> This seems the way to do it for objects, but for types, it doesn't
> seem right, as the p member turns out to be overwritten unexpectedly
> at runtime.
>
> Reading Python's object.h file it turns out that PyHeapTypeObject
> 'extends' PyTypeObject, which in turn has a PyObject_VAR_HEAD init
> macro. So mustn't:
>
> PyTypeObject JObjectMetaType = {
> PyObject_HEAD_INIT(NULL)
> };
>
> actually be:
>
> PyTypeObject JObjectMetaType = {
> PyVarObject_HEAD_INIT(NULL, 1)
> };
>
> with:
>
> JObjectMetaType.tp_basic_size = sizeof(JObjectMeta);
> JObjectMetaType.tp_itemsize = sizeof(void*);
>
> I tried this, but it doesn't keep my app from dumping core on an
> overwritten 'p'.
>
> My question basically is: how can I define a pointer for each type
> created with this metatype, such as is intended by the JObjectMeta
> struct?
The best is probably to store it in the class dictionary:
PyObject_SetAttrString(self, "_javatype_", PyLong_FromVoidPtr(p));
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Toolchain
2009/7/13 Nick Coghlan : > Nick Coghlan wrote: >> For the record, we only have SVN set to force Windows line endings for >> the old VC6 project files (.dsp and .dsw). The newer versions of Visual >> Studio can handle Unix line endings just fine so the .sln/.vcprops/etc >> files are left with native line endings and hence end up with Unix line >> endings in the source tarball. > > It would appear that Amaury just changed that. I'm not sure that was > actually necessary though (since nobody else has complained and the > original poster of this thread discovered the line endings weren't the > problem after all). Well, I was caught several times by this line ending problem. And the first time, I did not even understand what was going on... -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile
Hi, 2009/7/22 Eric Pruitt : > Hello, > > I have written replacements for a few of Mark Hammond's PyWin32 functions > using ctypes to call upon the Windows kernel API. Code can be found here; > http://pastebin.com/m46de01f . When calling ReadFile, it appears that the > kernel API converts '/n' newlines '/r/n' newlines. I have not been able to > find any information about this but as far as I can tell, there is nothing > in my code that is causing the problem. Initially I suspected it related to > newline translation but the function in subprocess.Popen for translating > newlines only converts to '/n' newlines. When I replaced my ReadFile and > WriteFile functions with the PyWin32 functions I was imitating, the problem > still arose. I was hoping someone could confirm this problem for me from > experience or by testing out my code. If you would like the see the > functions used in full context, I have a Mercurial Google Code project that > can be checked out at http://code.google.com/p/subprocdev/source/list > (branch "python3k"). My work entails modifications to subprocess.py so when > running the code, please update your PYTHONPATH variable so that my > subprocess.py will be imported. These questions should be redirected to comp.lang.python. But as a quick response, the subprocess stdout is likely to be opened in text mode. So reading \r\n is not a surprise to me. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile
Hi, 2009/7/23 Christian Heimes : > Lisandro Dalcin wrote: >> Eric seems to be working on a GSoC for PFS related to improving >> subprocess. Even in such case this list is not the right place to make >> these posts?? > > Eric didn't say that he is working on a GSoC project for the PSF. Anyway > the Python general mailing list might still be a better place. IMHO he > can reach many more competent Python developers there who can help him > with this particular problem. Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful. > By the way I don't think that ctypes is the right way to go here. ctypes > is very handy if you need a quick solution. However I wouldn't use it as > a permanent solution for the subprocess module. It's tricky to get > ctypes based solutions right on multiple platforms (32 vs 64bit). It's > harder to debug a ctypes solutions rather than a C extension, too. I > assume that calling overhead is greater than a pure C extension. And some distributions may choose to not distribute the ctypes module. No stdlib module should require it. OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away. > Can ctypes release the GIL for a function call? Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mingw32 and gc-header weirdness
2009/7/23 David Cournapeau : > On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore wrote: >> 2009/7/22 Christian Tismer : >>> Maybe the simple solution is to prevent building extensions >>> with mingw, if the python executable was not also built with it? >>> Then, all would be fine I guess. >> >> I have never had problems in practice with extensions built with mingw >> rather than MSVC - so while I'm not saying that the issue doesn't >> exist, it certainly doesn't affect all extensions, so disabling mingw >> support seems a bit of an extreme measure. > > I am strongly against this as well. We build numpy with mingw on > windows, and disabling it would make my life even more miserable on > windows. One constant source of pain with MS compilers is when > supporting different versions of python - 2.4, 2.5 and 2.6 require a > different VS version (and free versions are available only for the > last version of VS usually). > > I am far from a windows specialist, but I understand that quite a few > problems with mingw-built extensions with python are caused by some > Python decisions as well (the C API with runtime-dependent structures > like FILE, etc...). So mingw is not the only to blame :) In this case, the OP tries to use an API that is explicitly documented as dangerous for extension modules. The recommended function is not sensitive to compiler differences. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: pruning/reorganizing branches
2009/8/3 Dirkjan Ochtman : > So PEP 385 proposes to clean up the old branches we still have lying > around in SVN. > > io-c: keep-clone? strip - it was merged into py3k some months ago. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 389: argparse - new command line parsing module
2009/10/8 Paul Moore : > 2009/10/7 Antoine Pitrou : >> Python 3 complains at startup and is a bit more explicit: >> >> $ ./python -c '1' >&- >> Fatal Python error: Py_Initialize: can't initialize sys standard streams >> OSError: [Errno 9] Bad file descriptor >> Abandon >> >> Of course, if it is stderr that you explicitly close, you lose all reporting: >> >> $ ./python -c '1' 2>&- >> Abandon > > Hmm, how does Python 3's pythonw work on Windows? (I don't have a > Windows installation of Python 3 to hand at the moment) When running pythonw, fileno(stdout) is negative, so sys.stdout is set to None, and print() silently returns. But the situation is not perfect, see http://bugs.python.org/issue6501 where Apache provides a stdout, but python cannot determine the encoding because there is no console. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Better module shutdown procedure
2009/10/16 Neil Schemenauer : > After some experimentation I realize this idea is not ready yet. > The main problem comes from references to Python objects that > modules keep but don't expose to the garbage collector. For example, > gcmodule.c has a static pointer "tmod" that is a reference to the > "time" module. This reference prevents the "time" module from being > freed during interpreter shutdown. > > Ideally, I suppose modules should be treated like any other object > and have tp_traverse and tp_clear methods that deal with these sorts > of pointers. They would have to delegated to the instance since each > module would have its own implementation. Note since python 3.0 (and PEP 3121), the PyModuleDef structure has some members like m_traverse, m_clear and m_free for this very purpose. So far, nobody cared to implement these methods for any module. Maybe one should start at least for static PyObject* that contain references to modules. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] No buildbot to test wide unicode?
Hello, It seems that there is no buildbot to test a wide unicode build. On http://www.python.org/dev/buildbot/3.x/ all outputs of the "configure" steps show this message:: checking what type to use for str... unsigned short which looks like a ucs2 build to me. Since wide unicode is the standard chosen by some Linux distributions, it would make sense to have at least one buildbot running with --with-wide-unicode (3.x) or --enable-unicode=ucs4 (2.x). -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Data descriptor doc/implementation inconsistency
Hi,
2010/1/11 Benjamin Peterson :
> Consider this program:
>
> class Descr(object):
> def __init__(self, name):
> self.name = name
> def __set__(self, instance, what):
> instance.__dict__[self.name] = what
>
> class X(object):
> attr = Descr("attr")
>
> x = X()
> print(x.attr)
> x.attr = 42
> print(x.attr)
>
> It gives in output:
>
> <__main__.Descr object at 0x7fe1c9b28150>
> 42
>
> The documentation [1] says that Descr is a data descriptor because it
> defines the __set__ method. It also states that data descriptors
> always override the value in the instance dictionary. So, the second
> line should also be the descriptor object according to the
> documentation.
>
> My question is: Is this a doc bug or a implementation bug? If the
> former, it will be the description of a data descriptor much less
> consistent, since it will require that a __get__ method be present,
> too. If the latter, the fix may break some programs relying on the
> ability to "cache" a value in the instance dictionary.
>
> [1] http://docs.python.org/reference/datamodel#invoking-descriptors
Quoting the documentation:
"""Normally, data descriptors define both __get__() and __set__(),
while non-data descriptors have just the __get__() method.
"""
Your example is neither a data descriptor nor a non-data descriptor...
The thing that worries me a bit is the "x.attr" returning the Descr
object. Descriptors should remain at the class level, and instance
should only see values. I'd prefer an AttributeError in this case.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.1.2
Hi, 2010/2/15 Benjamin Peterson > 2010/2/15 "Martin v. Löwis" : > > Stefan Behnel wrote: > >> Benjamin Peterson, 13.02.2010 03:52: > >>> It's about time for another 3.1 bug fix release. I propose this > schedule: > >>> > >>> March 6: Release Candidate (same day as 2.7a4) > >>> March 20: 3.1.2 Final release > >> > >> Does a crash like #7173 qualify as a blocker for 3.1.2? > > > > I'm not the release manager, but my feeling is that, because there is no > > proposed resolution of the issue, it can't possibly be a blocker. Only > > if a patch is available, waiting for application of that patch may block > > the release. Waiting for a patch may cause indefinite delay, which would > > be bad. > > I agree with Martin here. I would be more inclined to make #7173 a > release blocker if it had a more specific test than "run cython and > maybe it'll crash". > I just updated #7173 with a short crasher. In short, I think that next() in an exception handler messes with the exception state. This doesn't play well with the cyclic garbage collector which can call tp_clear() on a resurrected object, if the reference is *moved* out of the cycle by some tp_dealloc. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] embedding Python interpreter in non-console windows application
Hi,
2010/2/17 stephen
> Hello,
>
> THE PROBLEM:
> I am having a problem that I have seen asked quite a bit on the web, with
> little to no follow up.
> The problem is essentially this. When embedding (LoadLibraryA()) the python
> interpreter dll
> in a non-windows application the developer must first create a console for
> python to do output/input with.
> I properly initialize the CRT and AllocConsole() to do this. I then
> GetSTDHandle() for stdin and stdout accordingly
> and open those handles with the requisite flags "read" for STDIN and
> "write" for stdout. This all works great
> and is then verified and tested to work by printf() and fgets(). This issue
> however happens when attempting
> to PyRun_InteractiveLoop() and PyRun_SimpleString(). A
> PyRun_SimpleString("print 'test'") displays nothing in my
> freshly allocated console window. Similarly a PyRun_InteractiveLoop(stdin,
> NULL); yields nothing either even though
> the line printf("testing"); directly ahead of it works just fine. Does
> anyone have insight on how I can make this work
> with the freshly allocated console's stdin/stdout/stderr?
>
> SPECULATION:
> That is the question, so now on to the speculation. I suspect that
> something in the python runtime doesn't "get handles"
> correctly for STDIN and STDOUT upon initialization. I have perused the
> source code to find out exactly how this is done
> and I suspect that it starts in PyInitializeEx with calls to
> PySys_GetObject("stdin") and "stdout" accordingly. However I
> don't actually see where this translates into the Python runtime checking
> with the C-runtime for the "real" handles to STDIN and STDOUT. I dont ever
> see the Python runtime "ask the system" where his handles to STDIN and
> STDOUT are.
>
Are you using the same compiler as the one used to compile Python? It's
important that your program and python use the same C runtime library
(MSVCR90.dll for python 2.6), otherwise "stdout" refers to different things.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] argparse.py is licensed under the Apache License
Hello, I noticed that the newly added argparse module has an unusual licence header, included below. This is the only file in the Python tree that contains an explicit reference to the Apache License, and this leads me to some questions: - Is the Apache license compatible with the Python license? Will this cause problem for some organizations that redistribute Python, possibly with proprietary software? Are there additional constraints? - Does this addition require a paragraph in the python documentation? http://docs.python.org/license.html#licenses-and-acknowledgements-for-incorporated-software - The Apache License states that:: You must cause any modified files to carry prominent notices stating that You changed the files but r78749 already modified the file (to remove a py3k warning) didn't we break the License? - Did the contributor sign a Contributor agreement? In this case, shouldn't the code be marked as "Licensed to PSF under a Contributor Agreement", as mentioned in the contribution form? http://www.python.org/psf/contrib/contrib-form And then, could this Apache License be removed? The first lines of Lib/argparse.py are: # Copyright 2006-2009 Steven J. Bethard . # # Licensed under the Apache License, Version 2.0 (the "License"); you may not # use this file except in compliance with the License. You may obtain a copy # of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Odd lines in unicodedata_db.h
2010/4/4 MRAB :
> I've just downloaded the daily snapshot at
> http://svn.python.org/snapshots/python.tar.bz2
>
> In the header file /python/Modules/unicodedata_db.h, there are the
> following lines in the change_records_3_2_0 struct:
>
> { 255, 255, 255, 255, 1.0 },
> { 255, 255, 255, 255, 2.0 },
> { 255, 255, 255, 255, 3.0 },
> { 255, 255, 255, 255, 4.0 },
> ...
> { 255, 255, 255, 255, 1e+16 },
> { 255, 255, 255, 255, 1e+20 },
>
> Looks like a bug to me.
I don't think so. Unicode 3.2 did contain two entries with large numeric values.
The file Unihan-3.2.0.txt contains these two lines:
U+4EAC kPrimaryNumeric 10,000,000,000,000,000 ten quadrillion (American)
U+5793 kPrimaryNumeric 100,000,000,000,000,000,000 hundred quintillion
(American)
For some reason newer versions of the unicode standard removed these values.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 : > Sorry, I wrote tp_alloc in last post, it should be always tp_dealloc: > > My tp_dealloc method (of non-subtypable type) calls the freeMem-method > of a memory manager (this manager was also used for the corresponding > allocation). > This freeMem-method deallocates and modifies the memory, > which is a valid action, because after free, the memory-manager > has ownership of the freed memory. > Several memory managers do this (for example the Memory Manager in > Delphi during debug mode, in order to track invalid memory access after free). > > The python31.dll calls tp_dealloc and later (after return of tp_dealloc) > the python31.dll is still awaiting valid content in the deallocated memory. > I don't know where this happens, I'm not a developer of CPython, > but at this point the python31.dll causes an access violation. > IMO the python31.dll assumes that freeMem never modifies the memory > (pyobject header), this is valid for many memory managers, but not for all. > And from my perspective, this assumption a bug, which can cause access > violations in many applications (for example, applications which use the > PythonForDelphi-package; PyScripter is one of them, but also many others) > > Please, could some CPython-developer take a look, thank you! CPython does not access memory after the call to tp_dealloc. There is even a mode (--without-pymalloc) where tp_dealloc calls free() at the end, and would cause crashes if the memory was read afterwards. This said, there may be a bug somewhere, but what do you want us to look at? Do you have a case that we could reproduce and investigate? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 : >> This said, there may be a bug somewhere, but what do you want us to look >> at? >> Do you have a case that we could reproduce and investigate? >> >> -- >> Amaury Forgeot d'Arc > > Thank you, I'm not a C-Developer, > but still I have one more detail: > > I call py_decRef( pyObj) of dll (version 3.1.1), > ( which calls tp_dealloc, which calls my freeMem() method)) > No problem is reported here. > Now, the freed memory should not be accessed anymore by python31.dll. > You may fill the freed pyObjectHead with invalid values, > in my case it's: ob_refcnt= 7851148, ob_type = $80808080 > > But later, when I call Py_Finalize, > there inside is some access to the same freed memory; > this causes an AV, more precisely, > when the value $80808080 is checked. > > My Delphi-Debugger shows the following byte-sequence inside python31.dll: > 5EC3568B7424088B4604F7405400407504 > > 5E - pop esi > C3 - ret > 56 - push esi > 8B742408 - mov esi, [esp+$08] > 8B4604 - mov eax, [esi+$04] > // eax = $80808080 // > > F740540040 - test [eax+$54], $4000 > // AV exception by read of address $808080D4 // > > 7504 - jnz $1e03681b > > > Maybe this can help someone, thank you! I'm sorry but this kind of issue is difficult to investigate without the source code. Normally I would compile everything (python & your program) in debug mode, and try to see why the object is used after tp_dealloc. For example, it's possible that your code does not handle reference counts correctly A call to Py_INCREF() may be missing somewhere, for example. This is a common error. tp_dealloc() is called when the reference count falls to zero, but if the object is still referenced elsewhere, memory will be accessed again! Without further information, I cannot consider this as a problem in Python. I know other extension modules that manage memory in their own way, and work. It's more probably an issue in the code of your type. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 :
>> Without further information, I cannot consider this as a problem in
>> Python.
>> I know other extension modules that manage memory in their own way, and
>> work.
>> It's more probably an issue in the code of your type.
>>
>> --
>> Amaury Forgeot d'Arc
>
> Ok, thank you, but I'm still hoping, someone could test this.
> I'm very sure, my app is not the cause;
> only the python31.dll (py_finalize) is accessing the freed memory.
> Inside py_finalize there is really no call to my hosting app (or reverse),
> I even tested this in my debugger.
To be clear:
- you did not provide anything for us to test.
- the fact that the crash is inside python31.dll does not indicates a
bug in python.
Consider this (bogus) code:
FILE *fp = fopen("c:/temp/t", "w");
free(fp);
This will lead to a crash at program exit (when fcloseall() is called
by the system)
but the issue is really in the code - it should not free(fp).
Without knowing what your code really do, we won't be able to help.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] debug and release python
2010/6/14 Kristján Valur Jónsson : > Hello there. > > I‘m sure this has come up before, but here it is again: > > > > Python exports a different api in debug mode, depending on whether > PYMALLOC_DEBUG and WITH_PYMALLOC are exported. This means that _d.pyd files > that are used must have been compiled with a version of python using the > same settings for these macros. It is unfortunate that the > _PyObject_DebugMalloc() api is exposed to external applications using macros > in objimpl.h > > > > I would suggest two things: > > 1) provide dummy or thunking versions of those in builds that don‘t > have PYMALLOC_DEBUG impolemented, that thunk to PyObject_Malloc et al. (This > is what we have done at CCP) > > 2) Remove the _PyObject_DebugMalloc() from the api. It really should > be an implementation of in the exposed PyObject_Malloc() functions whether > they use debug functionality at all. the _PyObject_DebugCheckAddress and > _PyObject_DebugDumpAddress() can be left in place. But exposing this > functionality in macros that external moduled compile in, is not good at > all. > > The reason why this is annoying: > > Some external software comes with proprietary .pyd bindings. When > developing my own application, with modified preprocessor definitions (e.g. > to turn off PYMALLOC_DEBUG) we find that those externally provided libraries > don‘t work. It takes a fair amount of detective work to find out why > exactly linkage fails. The external API really shouldn‘t change depending > on preprocessor definitions. I remember having the same issue years ago: http://mail.python.org/pipermail/python-list/2004-July/855844.html At the time, I solved the issue by compiling extension modules with pymalloc options turned on (which it fortunately the default, so this applies to the supplied proprietary .pyd), and I added a (plain) definition for functions like _PyObject_DebugMalloc, even when PYMALLOC_DEBUG is undefined. Since the python_d.dll is a custom build anyway, adding the code is not too much pain. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
Hello, 2010/7/15 Barry Warsaw : > The first draft of PEP 3149 is ready for review. I like it! I think it could mention the case where packages are not installed in the canonical directory, but placed elsewhere along the PYTHONPATH. This is how I deploy applications, for example, and the differences between python versions makes this a pain. The specific case of Windows should be mentioned: each foo.pyd contains the name of the python library (Python27.dll) it has been linked with; It must be rebuilt for a major version change. IMO the Windows installers provided by python.org should be built with a tag that contains this major number. > Thus for example, an initial implementation of PEP 384, compiled with > `--with-so-abi-tag=cpython-xy` would search for the following file > names when extension module `foo` is imported (in this order):: > > foo.abi3.so > foo.cpython-xy.so > foo.so Is this the correct order? IMO the so-abi-tag is more precise and the two first items should be swapped. PyPy would also benefit from this patch: it can now use extension modules, but the ABI is slightly different. By default, PyPy would load only modules containing the ABI tag, and refuse foo.so which is incompatible for sure. But the installations could still be shared between Python implementations. Cheers, -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible bug in randint when importing pylab?
Hi, 2010/8/19 Timothy Kinney : > I am getting some unexpected behavior in Python 2.6.4 on a WinXP SP3 box. This mailing list is for development *of* python, not about development *with* python. Your question should be directed to the comp.lang.python newsgroup, or the python-list mailing list. In any case, reading the documentation of both functions should answer your question. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 status
Hi, 2010/8/31 Antoine Pitrou : > David Cournapeau wrote: >> As far as IO is concerned, FILE* is just a special case of a more >> generic issue, though, so maybe this could be a bit reworded. For >> example, file descriptor cannot be shared between runtimes either. > > Er, really? Yes, on Windows, file descriptors returned by open() or dup() cannot be shared between runtimes. What can be safely shared is the underlying "handle", you can get it with the _get_osfhandle() function. > So it means that, for example, a FileIO object couldn't be shared > between runtimes either? How about a socket object? > Do you want to forbid FileIO and socket objects as part of the API? Python objects don't have this concern: all methods of FileIO are implemented in a single file (fileio.c), linked to a single C runtime. > Again, I propose that FILE* functions are allowed in the API, but their > use discouraged in the docs (with proper explanations from those who > know how to word them). IMO the warnings you'd write there would be similar to the motivations of PEP 384. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API doc question
2010/9/3 Nick Coghlan : > Due to the Unicode API discussion, I happened to be looking at the C > API docs at http://docs.python.org/dev/c-api/unicode.html#plain-py-unicode > and noticed that out of: > > PyUnicode_FromUnicode > PyUnicode_FromStringAndSize > PyUnicode_FromString > PyUnicode_FromFormat > PyUnicode_FromFormatV > PyUnicode_FromEncodedObject > PyUnicode_FromObject > > only the first and the last two are flagged in the online docs as > returning a new reference. The other 4 are not (but probably should > be). > > However, I can't see anything in the markup which is even causing > those "Return value: New reference" markings to appear in the first > place, nor any clues in the Documenting Python info. What am I > missing? This information is in the file: Doc/data/refcounts.dat There is a extension to sphinx that reads this file and generates the annotation in the documentation. This file is not very well known, even by core developers... -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
Hi, 2010/9/7 M.-A. Lemburg : >> Ok. I'm fine with excluding Py_UNICODE from the stable ABI. However, >> I think in the long run, I guess more support for wchar_t will then >> be needed in the API, e.g. more convenient argument parsing. > > Sure, we could add that. Just to be clear: does this mean that PyUnicode_FromUnicode() and PyUnicode_AsUnicode() won't belong to the stable ABI? PyUnicode_AsWideChar() is not as fast, because it needs to copy the data. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Garbage announcement printed on interpreter shutdown
2010/9/10 Fred Drake : > On Fri, Sep 10, 2010 at 4:32 PM, Georg Brandl wrote: >> IMO this runs contrary to the decision we made when DeprecationWarnings were >> made silent by default: it spews messages not only at developers, but also at >> users, who don't need it and probably are going to be quite confused by it, > > Agreed; this should be silent by default. +1. I suggest to enable it only when Py_DEBUG (or Py_TRACE_REFS or Py_REF_DEBUG?) is defined. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r84983 - in python/branches/py3k: Doc/library/os.rst Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c
2010/9/24 Antoine Pitrou : > > The getlogin test fails on many Unix buildbots, either with errno 2 > (ENOENT) or 22 (EINVAL) or "OSError: unable to determine login name": Do these buildbots run in a Windows service, i.e. with no user logged in? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pronouncement needed in issue9675
Hi, 2010/9/28 Jesus Cea : > http://bugs.python.org/issue9675 > > Long history sort: Python 2.7 backported Capsule support and > (incorrectly, in my opinion) marked CObject as deprecated. > > All C modules in the stdlib were updated to Capsule (with a CObject > compatibility layer), except BSDDB, because this change was done late in > the cycle, the proposed patch was buggy (solvable) and a pronouncement > was done that CObject was not actually deprecated. > > But in python 2.7 release, CObject is marked as deprecated (arg!), so > when executing python with -We (mark warnings as errors), bsddb fails. > > Since I think that adopting Capsule in BSDDB for 2.7.1 would break the > API compatibility (maybe the CObject proxy would solve this), and since > a previous pronouncement was done abour CObject not-deprecated in 2.7.x, > I would like comments. Is compatibility really broken? PyCObject_AsVoidPtr(), PyCObject_Import() accept Capsule objects as well. Or are there other usages of the api pointer? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pronouncement needed in issue9675
2010/9/28 Nick Coghlan : > Converting to a Py3k warning sounds like the best option. Can someone please explain why converting to a PyCapsule object is not an option? PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will work as before. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pronouncement needed in issue9675
2010/9/29 Guido van Rossum : >> Can someone please explain why converting to a PyCapsule object is not >> an option? >> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >> work as before. > > Because bsddb is an external module? Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pronouncement needed in issue9675
2010/9/29 Guido van Rossum : > On Tue, Sep 28, 2010 at 4:02 PM, Amaury Forgeot d'Arc > wrote: >> 2010/9/29 Guido van Rossum : >>>> Can someone please explain why converting to a PyCapsule object is not >>>> an option? >>>> PyCObject_AsVoidPtr() and PyCObject_Import() accept Capsules and will >>>> work as before. >>> >>> Because bsddb is an external module? >> >> Yes, bsddb is compiled in a separate .pyd or .so. But what does this change? > > Because it needs to support multiple Python versions from single source? But here we are talking about the bsddb module shipped with core python2.7 http://svn.python.org/view/python/branches/release27-maint/Modules/_bsddb.c This is a single source for a single version. And in any case a #ifdef is good enough. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is this a bug? no environment variables
Hi, 2010/11/23 Glenn Linderman : > File "C:\Python32\lib\random.py", line 108, in seed > a = int.from_bytes(_urandom(32), 'big') > WindowsError: [Error -2146893818] Invalid Signature In the subprocess documentation http://docs.python.org/library/subprocess.html """On Windows, in order to run a side-by-side assembly the specified env *must* include a valid SystemRoot.""" Can you keep this variable and start again? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] len(chr(i)) = 2?
2010/11/23 Alexander Belopolsky :
> This discussion motivated me to start looking into how well Python
> library itself is prepared to deal with len(chr(i)) = 2. I was not
> surprised to find that textwrap does not handle the issue that well:
>
>>>> len(wrap(' \U00010140' * 80, 20))
> 12
>>>> len(wrap(' \U0140' * 80, 20))
> 8
>
> That module should probably be rewritten to properly implement the
> Unicode line breaking algorithm
> <http://unicode.org/reports/tr14/tr14-22.html>.
>
> Yet finding a bug in a str object method after a 5 min review was a
> bit discouraging:
>
>>>> 'xyz'.center(20, '\U00010140')
> Traceback (most recent call last):
> File "", line 1, in
> TypeError: The fill character must be exactly one character long
>
> Given the apparent difficulty of writing even basic text processing
> algorithms in presence of surrogate pairs, I wonder how wise it is to
> expose Python users to them.
This was already discussed two years ago:
http://mail.python.org/pipermail/python-dev/2008-July/080900.html
So yes, wrap() and center() should be fixed.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 final review
2010/11/29 "Martin v. Löwis" > I have now completed > > http://www.python.org/dev/peps/pep-0384/ was structseq.h considered? IMO it could be made PEP384-compliant with two additions that would replace two non-compliant functions: - A new function to create types, since PyStructSequence_InitType is supposed to work on a unititialized static variable: PyTypeObject *PyStructSequence_NewType(PyStructSequence_Desc *desc); - PyStructSequence_SetItem(), similar to the macro PyStructSequence_SET_ITEM; the PyStructSequence structure should be hidden. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 384 accepted
Hi, 2010/12/3 Michael Foord > On 02/12/2010 23:01, "Martin v. Löwis" wrote: > >> [snip...] >>> >> I'm just getting tired having to talk to >> five projects just to make a single change to the build infrastructure >> available to the Python community. >> >> > The very best hope of resolving that particular problem is distutils2. :-) > > distutils2 is *already* available to the Python community, and whether or > not there is a fixed release date it will have betas and then a 1.0 release > in the foreseeable future. The team working on it has made an enormous > amount of progress. We're much better off as a development community putting > our support and energy into distutils2 rather than pining for evolution of > distutils. > Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; they change the paths and filenames used by python. Either we modify distutils to comply with the new names, or defer these PEPs until distutils2 is ready. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 11: Dropping support for ten year old systems
Hi, 2010/12/6 "Martin v. Löwis" > In my original posting, I proposed a clause where support could be > extended as long as an individual steps forward to provide that support. > So if XP remains popular by the time Microsoft stops providing patches > for it, some volunteer would have to step forward. > +1. Such a clause is already used to keep supporting the old VC6.0 compiler (a.k.a VC98!) -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Backport troubles with mercurial
Hello, The PyPy project recently switched from svn to mercurial. Since this day I have some difficulties to perform simple tasks, and my questions did not receive satisfying answers. I was sure the Python project would have the same issues; so I started from the repositories in http://code.python.org/hg/ and tried simple merges between Python versions. I would like several people to try the same example, and tell how they handle it. I'm new to mercurial, and I may have missed something important! Let's say you have to do the simple change shown in the diff below, and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. The diff was made against py3k. How would you do it with Mercurial? Please try to do it for real! "hg merge" is not the correct command: it merges whole branches, a change comes with all its ancestors. What we want is to "cherry-pick" a single change. "hg transplant" fails to apply the change to release31-maint because of a tiny conflict (in the diff context!) that leaves you with an ugly "hunks FAILED" and a .rej file you have to parse and apply by hand. "hg transplant" seems to succeed in release27-maint, but the test fails to run because "support" should be changed to "test_support". The change is already committed - to fix it another changeset is needed. This does not look clean to me: both changesets will be pushed to the repository, and diff review (on the python-checkins mailing list) is almost impossible. Furthermore, "hg transplant" does not keep track of integrations. There is a "transplants" file, but it stays in my local clone. Faced with a similar case in pypy, we finally manually copied the files between directories... and the fastest with our example is probably to do the change manually in all three directories. There is surely a better way to work with maintenance branches! PEP374 suggests to first modify the oldest release, but this does not seems right to me (I have three reasons in mind) At the moment, I feel that mercurial just cannot work for core Python development. At the very least before the migration we need precise use cases like this and recipes for common cases. Thanks for your help, -- Amaury Forgeot d'Arc diff -r 2777ae4d10d9 Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py Tue Dec 28 22:14:34 2010 +0100 +++ b/Lib/test/test_contextlib.py Wed Dec 29 00:08:18 2010 +0100 @@ -26,6 +26,7 @@ state.append(x) self.assertEqual(state, [1, 42, 999]) +@support.cpython_only def test_contextmanager_finally(self): state = [] @contextmanager @@ -36,10 +37,10 @@ finally: state.append(999) with self.assertRaises(ZeroDivisionError): -with woohoo() as x: +with woohoo() as y: self.assertEqual(state, [1]) -self.assertEqual(x, 42) -state.append(x) +self.assertEqual(y, 42) +state.append(y) raise ZeroDivisionError() self.assertEqual(state, [1, 42, 999]) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 David Cournapeau > The easiest way I found to emulate git cherry-pick (which does exactly > what you want) with hg is to use import/export commands: > http://mercurial.selenic.com/wiki/CommunicatingChanges > > It is indeed quite a pain to use in my experience, because you cannot > easily refer to the original commit the cherry pick is coming from > (i.e. no equivalent to git cherry-pick -x), and the conflict > resolution is quite dumb. > This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes? And finally the biased question: can you find one improvement over the current situation with svn? One alternative is to be careful about where > you apply your patch > ( > http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to-handle-hg-import-rejects > ), > but that's not very convenient either. > I've read all this, and this method does not work, for several reasons: - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. we usually consider py3k as a child of both 2.7 and 3.1, and there is no common parent. - even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge. - in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository. (btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?) -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl > > Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: > > > - even if there was one, there is the problem of changes specifically made > > for 2.7 > > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k > > which has the disadvantage of cluttering the py3k change log. And think of > > the > > horror > > if someone forgets this dummy merge. > > No horror at all: the next committer notices the extra changes in his merge > and > removes them. *Never* commit merges without reviewing the diff. (The > situtation is similar, by the way, to someone typing the wrong revision number > when using svnmerge, and not noticing it because he does not review the diff.) Except that it's easy to overlook a diff and not notice another small change mixed in your merge. Checking "hg merge -P" is probably better. But is it possible to "remove" the extra change? What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)? With subversion it's easy to work on trunk, and then have svnmerge tell you which chunk is in conflict and does not apply to the maintenance branch. > > > - in any case, the actual repositories in http://code.python.org/hg/ are > > not clones > > of each other, so "hg merge" won't work and "hg pull" will clone the whole > > remote repository. > > Note that the repos on code.python.org are not the results of our conversion > process. Those will be on hg.python.org. The former are repos contributed > by Antoine (I think) that he uses with hgsubversion to work on Python using > Mercurial right now; they will disappear after hg.python.org works. ok. I used them because other the repos I found were really old. And http://hg.python.org/cpython/ probably needs an initial "dummy merge" on every branch. > > > (btw, now that I have "hg pull" another repo, how can I remove it? is my > > clone > > broken forever?) > > No, you can "hg strip" away the root of the wrongly pulled revisions (which > also strips all descendants), or use "hg clone -r XXX" to create a new clone > with only a specified revision and all its ancestors. "hg strip" worked for me. The root revision to strip was simply the first one (=oldest) in "hg outgoing". -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >> What worries me more is the requirement to find the correct branch before I >> can >> even edit the file. How would you, Georg, deal with doc patches >> (typos, bad markup)? > > Finding the correct branch is not really hard. Bugfixes go to maintenance, > features to trunk. This works well indeed, provided there is only one maintenance branch. > You need to think about which category your change is > right now too, when deciding what to backport/svnmerge. No, today this decision can take place after the development, some tickets got reopened because a backport was needed. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >>> You need to think about which category your change is >>> right now too, when deciding what to backport/svnmerge. >> >> No, today this decision can take place after the development, >> some tickets got reopened because a backport was needed. > > A change can of course also be transplanted after the fact. It requires > another > merge, but usually a trivial one. No, it's not trivial with hg: this is the very subject of the thread! -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 David Cournapeau : > On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc > wrote: >> 2010/12/29 David Cournapeau >>> >>> The easiest way I found to emulate git cherry-pick (which does exactly >>> what you want) with hg is to use import/export commands: >>> http://mercurial.selenic.com/wiki/CommunicatingChanges >>> >>> It is indeed quite a pain to use in my experience, because you cannot >>> easily refer to the original commit the cherry pick is coming from >>> (i.e. no equivalent to git cherry-pick -x), and the conflict >>> resolution is quite dumb. >> >> This is precisely why I proposed a specific example. >> Which precise steps do you take? >> How much typing or manual copy/paste is required for this very simple case? >> Can you do the merge in less than 10 minutes? > > I don't know in this specific case. As I said, when I have to use hg, > that's the technique I use, and you get the issue you mention. That's > a hg limitation AFAICS. Yes, Georg identified three things that "hg transplant" should do better: - an option to not commit - a way to add conflict markers in the source instead of the .rej file (In this case, it may be just as easy to use the standard merge tools) - somehow share the "transplants" cache file between clones. > sometimes the people who become the most vocal proponents of the new > tool were the most sceptic ones before. I really was not sceptic before, and I certainly don't want to become one! But yesterday I was blocked the whole afternoon by something I still call an routine task with most other SCMs; and the only answer I get is "that's right, it's a pain" hg will certainly impose a change in the way we develop Python. I'm not sure everybody understands the consequences. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >>> A change can of course also be transplanted after the fact. It requires >>> another >>> merge, but usually a trivial one. >> >> No, it's not trivial with hg: this is the very subject of the thread! > > I don't understand: if you make the same change in two branches, but > in different changesets, and then merge these branches, Mercurial will > usually notice that and not trouble you with conflicts. Actually I never passed the first step: make the same change to two branches. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)
Hi, 2011/1/31 : > Reviewers: , > > Please review this at http://codereview.appspot.com/4080047/ [...] It looks good, but did you create an item in the issue tracker? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)
2011/2/1 anatoly techtonik : > we can definitely > find a lot of ways to improve Python development process for general > public Definitely. And the future migration to Mercurial will certainly help as well. But this thread started with a patch review, not with a proposal to change the development process! For the moment, we use svn and the issue tracker. If every patch comes with its own workflow, no coordination is possible. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failed issue tracker submission
56/256 bits)) > (No client certificate requested) > by mail.python.org (Postfix) with ESMTPS > for ; Tue, 29 Mar 2011 22:10:55 +0200 (CEST) > Received: from localhost > ([127.0.0.1] helo=dinsdale.python.org ident=hg) > by dinsdale.python.org with esmtp (Exim 4.72) > (envelope-from ) > id 1Q4fFf-00023G-4C > for rep...@bugs.python.org; Tue, 29 Mar 2011 22:10:55 +0200 > Date: Tue, 29 Mar 2011 22:10:55 +0200 > Message-Id: > Content-Type: text/plain; charset="utf8" > MIME-Version: 1.0 > Content-Transfer-Encoding: base64 > From: python-dev@python.org > To: rep...@bugs.python.org > Subject: [issue22663] > > TmV3IGNoYW5nZXNldCBkZDg1MmEwZjkyZDYgYnkgZ3VpZG8gaW4gYnJhbmNoICcyLjUnOgpJc3N1 > ZSAyMjY2MzogZml4IHJlZGlyZWN0IHZ1bG5lcmFiaWxpdHkgaW4gdXJsbGliL3VybGxpYjIuCmh0 > dHA6Ly9oZy5weXRob24ub3JnL2NweXRob24vcmV2L2RkODUyYTBmOTJkNgo= > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/amauryfa%40gmail.com > > -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Policy for versions of system python
2011/4/4 Eugene Toder : > Hello, > > CPython source code currently contains a number of python scripts (e.g > Python/makeopcodetargets.py, Objects/typeslots.py, Parser/asdl_c.py) > that are used during the build of the python interpreter itself. For > this reason they are run with system installed python. What is the > policy regarding > the range of python versions that they should support? > > I looked at some of the scripts and they seem to support both 2 and 3, > starting from at most 2.4. Python/makeopcodetargets.py says at the > top: > # This code should stay compatible with Python 2.3, at least while > # some of the buildbots have Python 2.3 as their system Python. > Is this the official minimal version or do we have this spelled out > more explicitly somewhere? Normally PEP291 lists the packages which should remain compatible with previous versions of Python: http://www.python.org/dev/peps/pep-0291/ makeopcodetargets.py is not mentioned there, though. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Borrowed and Stolen References in API
Hi, Le mercredi 4 mai 2011, Mark Shannon a écrit : > The online documentation specifies which API function borrow and/or steal > references (as opposed to the default behaviour). > Yet, I cannot find this information anywhere in the source. > > Any clues as to where I should look? It's in the file Doc/data/refcounts.dat in some custom format. -- Amaury -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Borrowed and Stolen References in API
Hi, Le jeudi 5 mai 2011, Greg Ewing a écrit : > Amaury Forgeot d'Arc wrote: > > > It's in the file Doc/data/refcounts.dat > in some custom format. > > > However, it doesn't seem to quite convey the same information. > It lists the "refcount effect" on each parameter, but translating > that into the notion of borrowed or stolen references seems > to require knowledge of what the function does. > > For example, PyDict_SetItem has: > > PyDict_SetItem:PyObject*:p:0: > PyDict_SetItem:PyObject*:key:+1: > PyDict_SetItem:PyObject*:val:+1: > > All of these parameters take borrowed references, but the > key and val get incremented because they're being stored > in the dict. This is not always true, for example when the item is already present in the dict. It's not important to know what the function does to the object, Only the action on the reference is relevant. > > So this file appears to be of limited usefulness. -- Amaury -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Borrowed and Stolen References in API
2011/5/5 Guido van Rossum : > Seems you're in agreement with this. IMO when references are borrowed > it is not very interesting. The interesting thing is when calling a > function *steals* a reference. The other important thing to know is > whether the caller ends up owning the return value (if it is an > object) or not. I *think* you can tell the latter from the +1 for the > return value; but the former (whether it steals a reference) is > unclear from the data given. There's even an XXX comment about this in > the file: > > # XXX NOTE: the 0/+1/-1 refcount information for arguments is > # confusing! Much more useful would be to indicate whether the > # function "steals" a reference to the argument or not. Take for > # example PyList_SetItem(list, i, item). This lists as a 0 change for > # both the list and the item arguments. However, in fact it steals a > # reference to the item argument! Should we change this file then? And only list functions that don't follow the usual conventions. But I'm sure that there are external tools which already use refcounts.dat in its present format. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Borrowed and Stolen References in API
Le vendredi 6 mai 2011, Mark Shannon a écrit : > What about #defining PY_STOLEN in some header? > > Then any stolen parameter can be prefixed with PY_STOLEN in signature. > > For return values, similarly #define PY_BORROWED. Header files are harder to parse, and I don't see how it would apply to macros. What about additional tags in the .rst files? -- Amaury -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how do you find out what version of Python a PEP landed in?
Hi, 2011/5/18 Chris Withers : > A friend of mine is coming over to Python and asked a question I thought > would have a better answer than it appears to: > > How do I know which version of Python a PEP lands in? > > I was expecting there to be a note at the bottom of the PEP, 342 in this > case, but that doesn't appear to be the case. > > What is the policy on this? Where should we be looking? Normally PEPs are important enough to be mentioned in the "whatsnew" document of each release. Googling for "what's new pep 342" suggests that it was released with Python 2.5. Now, an "official" way to get this information would probably be better... -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.2.1 rc 1
Hi, 2011/5/18 anatoly techtonik : > That's great, but where is the list if changes? All changes are always listed in the Misc/NEWS file. A "Change log" link on every download page displays this file. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how do you find out what version of Python a PEP landed in?
2011/5/18 "Martin v. Löwis" : >> How do I know which version of Python a PEP lands in? > > You should look at the Python-Version header of the PEP. But some PEPs don't have it: 341, 342, 343, 353... -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Don't set local variable in a list comprehension or generator
Hi,
2011/5/18 Terry Reedy :
> On 5/18/2011 10:19 AM, Nadeem Vawda wrote:
>
>> I'm not sure why you would encounter code like that in the first place.
>> Surely any code of the form:
>>
>> ''.join(c for c in my_string)
>>
>> would just return my_string? Or am I missing something?
>
> Good question. Anything useful like "'-'.join(c for c in 'abc')" is the same
> as "'-'.join('abc'). The same, as far as I can think of, for anything like
> list() or set() taking an iterable arg.
With a little imagination you can build something non trivial.
For example, a join_words function:
def join_words(words):
return ', '.join(w.strip() for w in words)
Like Victor says, the code of the generator object contains a
STORE_FAST followed by LOAD_FAST.
This pair of opcodes could be removed, and the value left on the stack.
>>> dis.dis(join_words.func_code.co_consts[2])
1 0 SETUP_LOOP 24 (to 27)
3 LOAD_FAST0 (.0)
>>6 FOR_ITER17 (to 26)
9 STORE_FAST 1 (w)
12 LOAD_FAST1 (w)
15 LOAD_ATTR0 (strip)
18 CALL_FUNCTION0
21 YIELD_VALUE
22 POP_TOP
23 JUMP_ABSOLUTE6
>> 26 POP_BLOCK
>> 27 LOAD_CONST 0 (None)
30 RETURN_VALUE
It's probably not easy to do though.
Think of expressions where the variable appears several times,
or even where the variable is not the first object, like str(ord(x)).
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AIX 5.3 - Build of Python 2.7.1
Hi, 2011/6/6 Anurag Chourasia : > ld: 0711-593 SEVERE ERROR: Symbol C_BSTAT (entry 559) in object > Parser/grammar1.o: A quick Google search reveals a probable issue with gcc on recent AIX systems. http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46072 A lot of users report this issue on various programs. Nothing Python can do about. You could try to disable the -g option. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Threaded asynchronous return from functions
Hello, 2011/7/4 Aigars Mahinovs > I have been doing some multithreaded work lately and have found that often what I find wanting to do is to call a function, have it check > it's arguments, possibly do some work and then return to the caller, > but still do some extra processing right after that. Currently to > accomplish such feat I need to separate the 'extra processing' bit > into a separate function and call that in a separate thread. A nice > convenience would be a function or statement that would allow to > return a value from the current function, but still keep running its > code (in a separate thread). Such approach could then be used in many > places where async processing is required, such as GUI programming, > XMLRPC, web applications, ... with less boilerplate and more obvious > code flow. > This kind of topic is not suitable to python-dev. Please ask this on the python-list mailing list, or eventually on python-ideas. (where someone will probably suggest you to use a nested function) Cheers, -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cannot build new documentation
Hello,
The new documentation system is really a major improvement: congratulations!
However, I run into a problem while trying to build the new python docs.
I initially used python2.5 on windows, but Linux seems to have the same problem.
Am I really the only one who gets this error?
The offending code is in Doc\tools\sphinx\builder.py, and looks like this:
>>> from __future__ import with_statement
>>> import codecs
>>> with codecs.open('foo.txt', 'w', 'utf-8') as fp:
... print type(fp), fp
... fp.write(u"\xb6")
...
Traceback (most recent call last):
File "", line 3, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb6' in
position 0: ordinal not in range(128)
Where does the 'ascii' codec come from? I propose the following explanation:
- codecs.open returns a wrapped stream (a codec.StreamReaderWriter).
This object implements some methods (read, write...) and delegates the
others to the underlying file object.
- 'with .. as ..' calls the __enter__ method, and assigns the result to fp.
- but StreamReaderWriter does not define __enter__, so file.__enter__
is called instead
and fp actually references the underlying file!
An obvious workaround is to add the __enter__ method to StreamReaderWriter:
def __enter__(self):
return self
This is not perfect though, because one may want to wrap say a
socket.makefile instead of a file.
It seems like the delegation pattern does not mix well with context managers...
Is there another solution?
Or did I miss something obvious?
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cannot build new documentation
Hello, Martin v. Löwis wrote: > I think the obvious thing you missed is that the problem got fixed > already. Whether the documentation system should be more defensive and > work with 2.5.0 also is a different question. You are right. python 2.5.1 corrected this. May I still dare propose the following patch: Index: Doc/tools/sphinx-build.py === --- Doc/tools/sphinx-build.py (revision 57158) +++ Doc/tools/sphinx-build.py (working copy) @@ -11,9 +11,9 @@ if __name__ == '__main__': -if sys.version_info[:3] < (2, 5, 0): +if sys.version_info[:3] < (2, 5, 1): print >>sys.stderr, """\ -Error: Sphinx needs to be executed with Python 2.5 or newer. +Error: Sphinx needs to be executed with Python 2.5.1 or newer. (If you run this from the Makefile, you can set the PYTHON variable to the path of an alternative interpreter executable.) """ -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] New python developer
Hello, As you may have seen, I have recently been granted developer privileges on python svn. Let me introduce myself. I am French, 35 years old, and father of 5 children. I work as a programmer in a fund manager company, and have been working with python for eight years now. What I mostly like in python are: - fun to work with - its robustness: not only against bugs, but the stability of its concepts, and the readability of its code base; - easy to interface with any other system. To start with python development, I think I will try to help with python3.0, or picking some bugs in the issue tracker. I have some knowledge of the python internals, with hours of debugging sessions of a C++ application mixed with the python interpreter. And I think I have read almost every svn commit for two years. I can also help on win32 specific development. As an example, I find that the distutils module don't work very well with the new compilers. I won't be of much help on Unix, though. I also like good and accurate documentation. Even if my english may be poor sometimes, I can help with writing some parts. Digging in the code to extract specifications is one of my favourite jobs. And of course I can help on other subjects if there is some interest. I am very excited to take part in this great project. So please forgive me if I do something wrong or too quickly. And your (kind) remarks will always be welcome. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should we do away with unbound methods in Py3k?
Guido van Rossum wrote: > I'm asking a Py3k question on python-dev because I'd like to have > opinions from people who haven't thought about Py3k much yet. Consider > the following example: > > class C: > def foo(self): pass > > C.foo(42) > > This currently fails with this error message: > > TypeError: unbound method foo() must be called with C instance as > first argument (got int instance instead) > > This message is called when isinstance(self, C) returns False, where > self is the first argument passed to the unbound method. > > That's nice, but there is a cost associated with this: the expression > "C.foo" can't just return the function object "foo", it has to wrap it > in an unbound method object. In Py3k the cost of calling an unbound > method object goes up, because the isinstance() check may be > overloaded. This typically happens when the class C uses the special > metaclass (abc.ABCMeta) used for virtual inheritance (see PEP 3119). > in Py3k the I/O stream classes are perhaps the most common use case. Could we check for "real" inheritance first, and call __instancecheck__ only when the previous is false? It would speed-up the common cases. Or is there really a use case for a derived class to appear as NOT being a subclass of its base class? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] why is mmap a builtin module on windows?
Hello, Ralf Schmitt: > > It's not an ugly hack, it's a well known feature. Add you don't have to > > change a lot of places, too. It's sufficient to add the alias at the > > entry point of your application (the script that starts your app). Once > > the alias sys.modules]'mmap'] = ralfmmap is set, every import mmap gets > > your ralfmmap. > > Well, I have multiple scripts using multiple libraries using the new mmap > objects. > So, I would have to change those scripts (if I don't change the libraries), > or change the libraries. And when I want to distribute my apps with bbfreeze > (similar to py2exe), I also need to handle that import hackery. This is > what I call ugly :) (at least in comparison to linux). You can also modify site.py and put the hack there. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX - in funcobject.c
Guido van Rossum wrote:
> I think we really *are* talking about the caller -- the caller owns
> the dict, if it managed to delete something from the dict before the
> callee can incref it, you'd have trouble. I don't immediately see how
> this could happen, which is probably why I left it as an XXX
> comment...
I found one way to call python code before the callee can incref the
args: the __eq__ between variable names and the dict entries. The
following snippet crashes the trunk version on win32:
class Name(str):
def __eq__(self, other):
del d[self]
return str.__eq__(self, other)
def __hash__(self):
return str.__hash__(self)
d = {Name("a"):1, Name("b"):2}
def f(a, b): print a,b
f(**d) # Segfault
There are several variants of this crasher; they all have more than
one keyword argument, and keywords strings must override __eq__ or
__hash__.
I could not find any other way to execute python code in this area.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX - in funcobject.c
Guido van Rossum wrote: > Thanks Amaury! Do you think it would be sufficient to change the > PyString_Check() call in PyEval_EvalCodeEx into a > PyString_CheckExact() call? This would prevent this "attack", but would remain fragile - future developments could allow execution of python code somewhere. > Or is the proper fix to incref the values > going into the kw array and decref them upon exit? Yet Another Kind Of Tuple... However this seems the correct thing to do. In addition, if we agree to restrict arguments names to str (and disallow subclasses), there are easy optimizations in PyEval_EvalCodeEx, somewhere around the "XXX slow" comment (!) -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can't decode \x876F character encoded in Shift JIS charset ?
Hello,
Nicolas Dumazet :
> >>> unicode('\x87\x6F', "shift jis")
> Traceback (most recent call last):
> File "", line 1, in
> UnicodeDecodeError: 'shift_jis' codec can't decode bytes in position 0-1:
> illegal multibyte sequence
>
> Still, \x87\x6F is a valid Shift-JIS character :
> http://demo.icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003&b=87&s=MIME#layout,
> it is "�L"...
It is possible that the encoding is actually "shift jis 2004" or
"cp932", which are both extensions to the original shift jis.
Please continue this discussion on comp.lang.python; or fill a bug request.
Cheers quand même,
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Py_CLEAR to avoid crashes
Hello,
I think I found a prolific vein of potential crashes throughout the
python interpreter.
The idea is inspired from the discussion Issue1020188 and is quite
simple: find a Py_DECREF(xxx->yyy), where xxx represents any kind of
Python object, and craft a __del__ method so that the same function is
recursively called with the same object.
I recently submitted corrections for 3 problems found this way. Here
are two more examples of this method:
#
class Special:
def __del__(self):
print a.args
class MyException(BaseException):
def __init__(self):
global a
a = self
BaseException.__init__(self, Special(), 0)
BaseException.__init__(self, "other", 1)
MyException() # segfault
#
import sys
class Special2(str):
def __del__(self):
f.__init__("@temp", "w")
f = file(Special2("@temp"), "w")
f.__init__("@temp", "w") # segfault
#
Issue1020188 (which is still open btw) deals with member reset to
NULL; but any modification of the pointer is potentially concerned.
And the correction is always the same: use Py_CLEAR, or a similar
construction that detach the value from the structure before
defcref'ing it.
I think it would help a lot of code if we had a safe standard way to
set struct members from C. A macro like the following:
Py_SETREF(lvalue, newobj)
(whatever its name) would perform the assignment and expand to code
equivalent to:
PyObject* oldobj = lvalue;
Py_INCREF(newobj);
lvalue = newobj;
Py_DECREF(oldobj);
I am currently searching for all places that could benefit of this,
but I am not sure to find a test case for every potential crash.
Most of the time, it is very unlikely that "normal" python code can
fall in these traps (who calls f.__init__ in a __del__ method?), with
the exception of the one corrected by r60871.
Should we however intensively search and correct all of them?
Is there a clever way to prevent these problems globally, for example
by delaying finalizers "just a little"?
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_CLEAR to avoid crashes
Hello,
Neil Schemenauer wrote:
> Nick Coghlan <[EMAIL PROTECTED]> wrote:
> > The problem is calls to Py_DECREF(self->attr) where some of the code
> > invoked by __del__ manages to find a way back around to reference
> > self->attr and gets access to a half-deleted object.
>
> Don't you mean "__del__ manages to find a way back around to self"?
> If so, how can that happen? If such a reference path exists, the
> reference count of self should not be zero. I don't understand why
> Py_CLEAR is necessary outside of tp_clear functions.
Of course we are speaking of different objects.
For example, in exception.c, BaseException_init() starts with the instruction:
Py_DECREF(self->args);
this may call __del__ on self->args, which can execute arbitrary
python code - including access to the now-invalid "args" member of the
exception.
class S:
def __del__(self):
print e.args
e = BaseException(1, S())
e.__init__("hello") # segfault
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Small RFEs and the Bug Tracker
Jeroen Ruigrok van der Werven wrote: > -On [20080218 13:38], Virgil Dupras ([EMAIL PROTECTED]) wrote: > >Personally, I think that a bug tracker is a good place to keep RFE, > >not a PEP. I think that the PEP would tend to be cluttered with RFE > >nobody cares about forever. So the clutter can never be cleaned unless > >someone takes the responsibility to mercilessly remove them. > > A bug tracker is a much better way of registering such information. It also > can be easier referenced in the future since even though when it is closed, > the debate and other stuff will remain in the tracker's tickets for > posterity. :) > > PEP: -1 > tracker: +1 I agree. Then we can set some status/keyword when the subject of a RFE is accepted by core developers, saying "if someone proposes a patch, it has a chance to be reviewed and applied". It may incite occasional contributors to work on some of these tasks, confident that their work will not be thrown away in two seconds. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r60919 - peps/trunk/pep-0008.txt
Skip Montanaro wrote: > Why not just skip the specifics except to say < 80 characters for all > lines? Don't mention 72, 79 or any other number than 80: I often run "svn diff" in a console limited to 80 characters. Because of the leading "+", lines longer than 78 wrap, and the output is more difficult to read. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nested classes leaking in compiler
On Fri, Mar 28, 2008 at 11:50 AM, Georg Brandl <[EMAIL PROTECTED]> wrote: > While preparing the Python-AST compilation patch, I noticed that each > class nested in a class leaks one reference (2.5 and trunk). > > It wasn't found by regrtest -R because it only happens on compiling, > and it seems that all snippets compiled during the tests as opposed to > on import didn't contain such a construct. > > The AST generation stage is fine, the leak happens somehwere > after that (I suspect the symtable code). It would be nice if someone > who understands more about that code than I do could fix this. The problem is actually in compile.c, in compiler_class(): Each compilation scope holds the name of the enclosing class (for __name mangling). In the case of nested classes, the inner scope is first initialized with the outer class name, then replaced with the inner class name. But a Py_XDECREF(c->u->u_private) is missing here... I will submit the correction in a few hours. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nested classes leaking in compiler
Christian Heimes wrote:
> Georg Brandl schrieb:
>
> > Somehow I knew you'd be the one to find the problem :)
>
> Agreed! :)
>
> Amaury is really good in finding loose ends in the parser and AST code.
> I still can't wrap my brain around the AST code but it can see the light
> at the end of the code.
Actually the ast compiler is the part I know least in CPython code.
The approach I used is a bit brute-force, but it may work for other
reference leaks:
- First write a big leaking script:
for x in xrange(10): leaking_function()
- Since memory usage does not increase dramatically, it's only a
refcount problem - no new object every time.
- Modify the Py_INCREF macro so that it generates a breakpoint in the
debugger when the refcount is large:
On Windows, it is something like:
if (ob->refcount > 1) { __asm int 3; }
(with gdb I think you may signal SIGUSR1)
- carefully inspect the code each time the debugger stops.
Fortunately, many functions can be skipped: PyDict_SetItem and other
bullet-proof parts of the code.
The 20th break was the good one: a lack of symmetry between
Py_INCREF and Py_DECREF.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] refleaks on trunk
On Tue, Apr 1, 2008 at 6:34 AM, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> test_io is the only leaky test on trunk that I know of. I narrowed
> down the leaks to the code below.
>
> It's possible there are other leaks in test_io.
>
> n
> --
> import sys, gc
> import _fileio, io
>
> class FileIO(_fileio._FileIO, io.RawIOBase):
> def close(self):
> io.RawIOBase.close(self)
>
> def main():
> class MyFileIO(FileIO): pass
> f = MyFileIO('tt-test', "w")
> f.close()
>
> for i in range(10):
> main()
> print(gc.collect())
> print(sys.gettotalrefcount())
The problem is that the MyFileIO class is registered in
io.RawIOBase._abc_cache, and never freed.
This is a problem with ABCs: _abc_cache should be changed to a
WeakSet, like python3.0 did.
I filed http://bugs.python.org/issue2521 about this problem.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracing at a more granular level (branch coverage)
Hello, On Tue, Apr 1, 2008 at 9:02 PM, m h wrote: > Howdy Folks- > > I hope this is not too off topic. Sadly the current sys.settrace only > allows tracing at the line level. This isn't sufficient to do branch > and path coverage. The main problem is that many boolean operations > can appear on a single line, and so you can't be sure which > conditional branch was taken using the current tracing method. > > After speaking about code complexity/testing at Pycon [0], with a few > people there, and seeing a few people have made some strides or shown > interested in metrics/code graphs/flow and branch coverage I think > that there is interest in this. > > One proposed method of getting branch coverage was to use Dalke's > Python4Ply [1] to translate code so that all branches occur on their > own line. Then using line coverage on that and converting it back to > branch coverage of the original code. > > The maintainer of coverage.py suggested that we look into patching > python instead to trace at a more granular level. His feeling was > that there would be too many corner cases and the translation would > get hairy quite quickly. > > Sadly in my 8 years of python experience I've yet to touch any c based > guts of python. I'm looking for advice on how to get finer grain > tracing with sys.settrace. > > Any advice or suggestions? There is a quorum of people (at least 5 > others) who would be very interested in this functionality because it > could lead to some cool tools built on top of it. (With the end goal > that python code be cleaner, simpler and better tested). You can start by looking at the file Python/ceval.c, which contains the interpreter loop. More precisely, the function maybe_call_line_trace() is responsible to call sys.settrace. I think you will have to change the logic there. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] the release gods are angry at python
On Wed, Apr 2, 2008 at 9:58 AM, Trent Nelson <[EMAIL PROTECTED]> wrote: > > > In the py3k branch I've assigned the audio resource to the winsound > > > tests. Only regrtest.py -uall or -uaudio runs the winsound test. > > Reason: > > > the test sound was freaking out my poor cat. :/ > > > > I feel with your cat ;-). > > This would not help on the buildbot since it runs 'rt.bat -d -q -uall - > > rw'. > > I feel for the poor NOC engineers at my colo that freak out when some random > server in a farm of thousands starts making bizarre sounds. > > I detest test_winsound. There are so many corner cases you need to account > for that makes the test pointless as you end up wrapping everything in > except: pass blocks. Does the system have a legacy beep driver? Is it > enabled? Is it disabled? Is there a sound card? Is it enabled or disabled? > Pah! > > +1 to removing audio out of -uall, if only for the sake of cats, erroneously > red buildbots, and poor ServerCentral NOC engineers. And I would not mind removing this module altogether, and provide a ctypes implementation. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
On Fri, Apr 4, 2008 at 12:57 PM, Trent Nelson <[EMAIL PROTECTED]> wrote: > > > I don't like the part where the solution kills the Python process during > > a rebuild. It's too surprising for the user. > > Hmmm. When you say rebuild, I assume you mean the change I made to the > pythoncore project's pre-link step to call kill_python.exe, and not to the > post-build step of kill_python that runs itself? Personally, when I'm doing > development, if I've got the pcbuild\python_d.exe console open, it's usually > to test one liners, I'm not using it to do anything important. If I forget > to close it before I kick off a build, it's annoying running into errors at > the link stage, I'd certainly prefer the build system to kill off anything > that'll inhibit a successful link before actually linking. > > What do others that do Windows development think? I don't have a problem > changing the build behaviour if the approach I've taken is generally disliked. I agree with Christian: in interactive sessions, the F7 key should not kill my running testsuite... I prefer the linker errors. Please do this only for buildbot builds! Or maybe have it controlled by an enviroment variable. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help in MAPI
Hello, Antony Joseph wrote: > Hi, > > My Code: > mapi.MAPIInitialize(None) > session = mapi.MAPILogonEx(0, MAPIProfile, None, > mapi.MAPI_EXTENDED | mapi.MAPI_USE_DEFAULT) > > I am trying to send a mail using the extended MAPI interface, I am > new to work with MAPI. > I am trying to execute your code,i getting the following exception, and a > popup message of Either there is no default mail client or the current mail > client cannot fullfill the messaging request,please run Microsoftoffice > outlookand set it as the defaukt mail client. > I am using thunderbird as my default mail client , then i set my outlook as > my default mail client.its running fine. > can u tell me is there possiblites to run the code with out changing the > default mail client to Ms Outlook. First, this mailing list is for the development of the python language, not for development with python. Please ask this kind of questions on the comp.lang.python newsgroup. Then, your problem is not related to python at all. The same call from any other program would return the same error. A quick google search gave: http://www.tech-archive.net/Archive/Development/microsoft.public.win32.programmer.messaging/2007-04/msg00036.html which describes the same problem (and a solution) Otherwise you are welcome ;-) -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
