Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Sat, Sep 13, 2014, 09:33 R. David Murray wrote: > On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan > wrote: > > On 13 Sep 2014 10:18, "Jeff Allen" wrote: > > > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, > it > > would have to do it the same way as CPython, as it is visible. It's not > > impossible (I think), but is messy. Some are strongly against. > > > > It may be worth trying *without* it (i.e. treat "surrogateescape" as > > equivalent to "strict" initially), and seeing how you go. The main > purpose > > of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data > > round trips work on POSIX" aspect of CPython 2, which doesn't apply in > > exactly the same way on Jython. > > > > Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2, > > "surrogateescape is equivalent to strict" seems like a relatively small > > discrepancy in behaviour. > > That would totally break the email package. > > It would of course be possible to rewrite email to not use surrogate > escape, but it is a seriously non-trivial undertaking. > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > tlesher%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyEval_Call* convenience functions
Is there a reason that the PyEval_CallFunction() and PyEval_CallMethod() convenience functions remain undocumented? (i.e., would a doc-and-test patch to correct this be rejected?) I didn't see any mention of this coming up in python-dev before. Also, despite its name, PyEval_CallMethod() is quite useful for calling module-level functions or classes (given that it's just a PyObject_GetAttrString plus the implementation of PyEval_CallFunction). Is there any reason (beyond its undocumented status) to believe this use case would ever be deprecated? Thanks. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Making the GIL faster & lighter on Windows
On Tue, May 26, 2009 at 16:07, "Martin v. Löwis" wrote: >> 3. ?? - I'm sure there are other issues that deserve a look. > > What about fairness? I don't know off-hand whether the GIL is > fair, or whether critical sections are fair, but it needs to be > considered. FWIW, Win32 CriticalSections are guaranteed to be fair, but they don't guarantee a defined order of wakeup among threads of equal priority. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Toolchain
On Mon, Jul 13, 2009 at 11:46, Eric Pruitt wrote: > Hello, > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft > Visual C++ Express Edition in order to test some modifications I made to the > _subprocess.c file as part of my Google Summer of Code proposal. I cannot > seem to figure out how to compile Python on Windows and could use some > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not > work when I attempt to open and no error message is given when clicked. When > opened from the Visual C++, it says "Solution folders are not supported in > this version of the application." What version of Visual C++ Express are you using? This sounds suspiciously like a version issue. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows make_buildinfo changes in 2.6 and cross-compilation
I ran across this while trying to upgrade our company's internal Windows CE build from 2.5.2 to either 2.6 or 3.1. In the 2.5 era, the Windows version of make_buildinfo used to fall back to just copying the getbuildinfo.c file from \Modules if subwcrev failed; since 2.6, make_buildinfo fails the build if the compilation of getbuildinfo.c fails when make_buildinfo is run, without falling back to copying. Any reason not to keep the fallback strategy in the process? The new process fails in cross-compilation scenarios (like building for Windows CE on a Windows host) because, while the VC++ solution can be told to build native (win32) binaries for make_buildinfo and make_versioninfo, when make_buildinfo tries to compile via its hardcoded call to cl.exe, it tries to use the cross-compiler without the required flags from the cross-compilation configs in the vcproj files. Thanks! -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Removing Windows 95-related code in posixmodule.c
Now that Windows 9x is no longer supported (per PEP 11), would it be
appropriate to replace the ANSI ("A") versions of Windows OS calls in
posixmodule.c with their Unicode ("W") equivalents and remove the
unicode_file_names() function that determines which version to call?
There are a number of places in that module where we try to decide whether
to call ANSI or UNICODE versions (because only the ANSI versions were fully
supported on Windows 9x). This is still fine on Win32, where the ANSI
versions are still supported for backwards compatibility, but it causes
issues with Windows CE, which doesn't have the ANSI versions at all.
I've been maintaining some fairly ugly patches to cope with this in my
company's Python 2.5.x for CE build, and I'd love to drop them as we move to
a post-2.6 version.
Thanks!
--
Tim Lesher
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can ftp url start with file:// ?
On Fri, Jul 9, 2010 at 12:41, Bill Janssen wrote:
> So, FTP is *not* the "default protocol". On the other hand, if
> actually begins with "ftp.", it's a pretty good guess that FTP will
> work.
Actually, FTP *is* the default protocol for most URLs with hostnames in
urllib.py.
urllib.open_file() delegates to open_ftp() if there's a any host other than
the exact string "localhost", and open_local_file() otherwise.
>>> import urllib
>>> f =urllib.urlopen('file:///foo.txt')
>>> f =urllib.urlopen('file://localhost/foo.txt')
>>> f = urllib.urlopen('file://www.google.com/')
Traceback (most recent call last):
File "", line 1, in
File "c:\python25\lib\urllib.py", line 82, in urlopen
return opener.open(url)
File "c:\python25\lib\urllib.py", line 190, in open
return getattr(self, name)(url)
File "c:\python25\lib\urllib.py", line 457, in open_file
return self.open_ftp(url)
File "c:\python25\lib\urllib.py", line 538, in open_ftp
ftpwrapper(user, passwd, host, port, dirs)
File "c:\python25\lib\urllib.py", line 844, in __init__
self.init()
File "c:\python25\lib\urllib.py", line 850, in init
self.ftp.connect(self.host, self.port)
File "c:\python25\lib\ftplib.py", line 129, in connect
raise socket.error, msg
IOError: [Errno ftp error] (10060, 'Operation timed out')
>>> f =urllib.urlopen('file://127.0.0.1/foo.txt')
Traceback (most recent call last):
File "", line 1, in
File "c:\python25\lib\urllib.py", line 82, in urlopen
return opener.open(url)
File "c:\python25\lib\urllib.py", line 190, in open
return getattr(self, name)(url)
File "c:\python25\lib\urllib.py", line 457, in open_file
return self.open_ftp(url)
File "c:\python25\lib\urllib.py", line 538, in open_ftp
ftpwrapper(user, passwd, host, port, dirs)
File "c:\python25\lib\urllib.py", line 844, in __init__
self.init()
File "c:\python25\lib\urllib.py", line 850, in init
self.ftp.connect(self.host, self.port)
File "c:\python25\lib\ftplib.py", line 129, in connect
raise socket.error, msg
IOError: [Errno ftp error] (10061, 'Connection refused')
--
Tim Lesher
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can ftp url start with file:// ?
To be clear, Python 2.x's urllib.urlopen() has this issue; 3.1's urllib.request.urlopen() rejects non-local hosts in a file URL. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is this a bug? no environment variables
On Mon, Nov 22, 2010 at 16:54, Glenn Linderman wrote: > I suppose it is possible that some environment variables are used by Python > directly (but I can't seem to find a documented list of them) although I > would expect that usage to be optional, with fall-back defaults when they > don't exist. I can verify that that's the case: Python (at least through 3.1.2) runs fine on Windows platforms when environment variables are completely unavailable. I know that from running our port for Windows CE (which has no environment variables at all), cross-compiled for Windows XP. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Module size
On Tue, Nov 30, 2010 at 09:41, Antoine Pitrou wrote: > That said, I don't think the size is very important. For any non-trivial > Python application, the size of unicodedata will be negligible compared > to the size of Python objects. That depends very much on the platform and the application. For our embedded use of Python, static data size (like the text segment of a shared object) is far dearer than the heap space used by Python objects, which is why we've had to excise both the UCD and the CJK codecs in our builds. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Iterating over marshal/pickle
Both marshal and pickle allow multiple objects to be serialized to the
same file-like object.
The pattern for deserializing an unknown number of serialized objects
looks like this:
objs = []
while True:
try:
objs.append(marshal.load(fobj)) # or objs.append(unpickler.load())
except EOFError:
break
This seems like a good use case for an generator:
def some_name(fobj):
while True:
try:
yield marshal.load(fobj) # or yield unpickler.load()
except EOFError:
raise StopIteration
1. Does this seem like a reasonable addition to the standard library?
2. Where should it go, and what should it be called?
>From an end-user point of view, this "feels" right:
import pickle
u = pickle.Unpickler(open('picklefile'))
for x in u:
print x
import marshal
for x in marshal.unmarshalled(open('marshalfile')):
print x
But I'm not hung up on the actual names or the use of sequence
semantics in the Unpickler case.
Incidentally, I know that pickle is preferred over marshal, but some
third-party tools (like the Perforce client) still use the marshal
library for serialization, so I've included it in the discussion
--
Tim Lesher <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitext('.cshrc')
On 3/6/07, Hans Meine <[EMAIL PROTECTED]> wrote: > The current behavior is clearly a bug, since a leading dot does not start an > extension, but marks a file as "hidden". The latter is on UNIX, and while > this is different on Windows, I cannot imagine that anyone would > a) have dotfiles under that OS, or even then > b) rely on the current behavior that their full filename counts as > "extension". FWIW, all of the "standard" Windows functions from the Microsoft CRT (_splitpath) to the Shell API (PathRemoveExtension) to the CLR (System.IO.Path.*) believe that ".cshrc" is the extension of the filename ".cshrc". I'm not sure if that's an argument for or against the patch, though. -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] public visibility of python-dev decisions "before it's too late" (was: PyCObject_AsVoidPtr removed from python 3.2 - is this documented?)
On Wed, Mar 9, 2011 at 01:15, Stefan Behnel wrote: > Actually, why not put up a web page of "upcoming changes" somewhere, that > lists major decisions with user impact that were taken on python-dev? > Including a link to the relevant discussion and decision. Often enough, > decisions are taken inside of huge mailing list threads that get off-topic > before someone has "the right idea" and everyone who's still there to listen > agrees. Even for people lurking around on python-dev, it's easy enough to > miss these moments. We used to do biweekly-ish Python-Dev summaries for this reason. The original links at python.org appear to be down, but I found an example mirrored at ftp://ftp.ntua.gr/mirror/python/dev/summary/2005-02-01_2005-02-14.html Would resuming these and putting them back on python.org address the issue? It's been on my back burner for about two years now, but I want to make sure I can keep up before diving in again. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pydoc for named tuples is missing methods
[I mentioned this to Raymond Hettinger after his PyCon talk, and I
promised a bug and hopefully a patch. I don't see an obvious solution,
though, so I'll ask here first.]
Because named tuple prefixes a single underscore to its added method
names (_asdict, _replace, and _make), those methods' docstrings are
omitted from pydoc:
>>> Point=collections.namedtuple('Point', 'x y')
>>> help(Point)
Help on class Point in module __main__:
[output omitted; it excludes _asdict, _replace, and _make]
pydoc's rules for name inclusion are in pydoc.visiblename():
* If the name is in the hidden list, omit it
* If the name looks like a __special_method__, include it
* If the there is an "all" specified, then include it if it appears in all
* Otherwise, include it if it doesn't begin with an underscore
There doesn't seem to be an obvious way to get around these rules for
named tuples... am I overlooking something?
Thanks.
--
Tim Lesher
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc for named tuples is missing methods
Addendum: this looks related to bug 1189811. http://bugs.python.org/issue1189811 That issue seems to hinge on the definition of "private". -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc for named tuples is missing methods
On Mon, Mar 14, 2011 at 05:45, Nick Coghlan wrote: > There are two relatively simple ways forward I can see: > > A. Add a __public__ attribute that pydoc (and import *) understand. > This would overrride the default "this is private" detection and add > affected names to the public list (without needing to respecify all > the "default public" names - this is important in order to support > subclassing correctly) I believe this was the direction the bug report was implying. I'll be sprinting for a few hours this morning; if there are no objections, I will try to turn this idea into a patch that makes pydoc.visiblename look for a __public__ function attribute as "step 0". Maybe there should also be a @public decorator to apply it, although that name may be an attractive nuisance, tempting C++ or Java programmers new to Python to ask for @protected and @private... -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc for named tuples is missing methods
On Mon, Mar 14, 2011 at 08:28, Tim Lesher wrote: > On Mon, Mar 14, 2011 at 05:45, Nick Coghlan wrote: >> There are two relatively simple ways forward I can see: >> >> A. Add a __public__ attribute that pydoc (and import *) understand. >> This would overrride the default "this is private" detection and add >> affected names to the public list (without needing to respecify all >> the "default public" names - this is important in order to support >> subclassing correctly) > > I believe this was the direction the bug report was implying. > > I'll be sprinting for a few hours this morning; if there are no > objections, I will try to turn this idea into a patch that makes > pydoc.visiblename look for a __public__ function attribute as "step > 0". > > Maybe there should also be a @public decorator to apply it, although > that name may be an attractive nuisance, tempting C++ or Java > programmers new to Python to ask for @protected and @private... I implemented this on Monday, and it worked great... for instance methods. Unfortunately, it doesn't do a thing for classmethods or data attributes, which are 2/4 of the original namedtuple use cases, so it feels like a bad idea. In the process of implementing it, though, I realized that the __all__ mechanism for modules could still be reused. Two implementations came to mind: 1. Allow an __all__ class attribute that affects pydoc for classes the same way it does for modules. Pro: it's familiar, easy to explain, and easy to implement (four lines of code) Con: the original use case (adding a small number of otherwise-excluded attributes to existing documentation) is cumbersome. Because __all__ means "don't document anything but __special_names__ and the contents of __all__, you'd have to manually add names that pydoc would normally pick up. Also, __all__ itself will show up in the documentation, which seems to me like useless clutter to a reader of help(). 2. Add a new class attribute that uses the same mechanism, with a different name (e.g., __api__). Pro: fairly easy to explain; because it doesn't share a name with __all__ its semantics can be tweaked without confusing people. Con: slight additional cognitive burden of a new feature I'm implementing both of these separately this week to see which works better in practice. So far I'm liking __api__ better, with these semantics: 1. Classes without __api__ are treated just as they are today. 2. __api__ on classes works just like __all__ on modules (only special names, plus its contents are documented) 3. Additionally, __api__ can take an Ellipsis. That makes the __api__ list additive--pydoc documents everything it normally would, *plus* the contents of __api__ 4. __api__ is added to pydoc's "hidden names" list; since its only purpose is to help generate docs, its value would be of little use in the generated docs themselves. Point 3 (Ellipsis) is unusual, but to me it makes the declaration read well and solves the namedtuple case succinctly and compatibly, without changing a published API. It also could be used to address the issue of stdlib classes that have non-underscored members that really shouldn't be considered part of the "public" API, but can't be renamed now for fear of breaking code. Usage example: class C1: __api__ = ['meth1', '_data2'] # read as "The API consists of meth1 and _data2." def __init__(self): """documented because it's a special name""" def meth1(self): """documented because it's in __api__""" def meth2(self): """Shouldn't really have been exposed, but it was released that way, and we don't want to break users' code. NOT documented because it's not in __api__ """ def _meth3(self): """NOT documented because it's not in __api__""" data1 = None # not documented--not in __api__ _data2 = None # documented--it's in __api__ class C2: __api__ = ['_data2', ...] # read as "The API includes _data2." def __init__(self): """This is documented because it's a special name""" def meth1(self): """documented because it follows the normal rules""" def meth2(self): """documented because it follows the normal rules""" def _meth3(self): """NOT documented because it's not in __api__ and starts with underscore""" data1 = # documented because it follows the normal rules _data2 = None # documented--it's in __api__ Comments and criticisms welcome. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc for named tuples is missing methods
On Tue, Mar 15, 2011 at 11:24, Antoine Pitrou wrote: > Wouldn't a decorator be adequate? > > @pydoc.public_api > def _asdict(self): > """some docstring""" > ... That was my first attempt. Unfortunately, it didn't work well with classmethods or immutable data members, and forcing the module to import pydoc had some bad side effects (particularly in the eval-ed code for namedtuple). -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc for named tuples is missing methods
On Tue, Mar 15, 2011 at 11:15, Nick Coghlan wrote: > The challenge here is how it would interact with inheritance. pydoc > couldn't use normal attribute lookup, it would have to walk the MRO > manually, only applying __api__ to the specific class that defined it. Great catch. I know pydoc already looks at this in the attrs-processing loop (to group attrs by their defining class), but my current implementation applies __api__ too early to deal with that. I'll fix it. Any test cases should definitely throw some diamond-pattern or even more degenerate cases at the implementation. What *is* the worst case for MRO complexity? Overall, this is becoming more interesting than I'd thought at first. Is this something that should require a PEP? -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.sleep(-1) behaviour
On Thu, Jun 30, 2011 at 15:13, Ulrich Eckhardt wrote: > Hi! > > This is a request for clarification for the thread "how to call a function for > evry 10 seconds" from the user mailinglist/newsgroup. > > > The gist is this: > 1. On Linux/Python 2.6, time.sleep(-1.0) raises an IOError. > 2. On MS Windows/Python 2.5 or 2.7 this sleeps forever. It seems that > converting this to a 32-bit integer for Sleep() causes an underflow. > 3. Is the behaviour under MS Windows acceptable or a bug? On the Windows side, Sleep(-1) as "infinite" is correct and documented: http://msdn.microsoft.com/en-us/library/ms686298(v=vs.85).aspx -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.sleep(-1) behaviour
On Fri, Jul 1, 2011 at 08:46, Nick Coghlan wrote: > For Sleep, yes, but the time.sleep() docs [1] are completely silent on > the behaviour of negative sleep values at the Python level. Question 3 > isn't "Is there something wrong with Sleep()?", it is "Is there > something wrong with the way time.sleep() *uses* Sleep()?" That makes sense. Better to be consistent within the time API--I know the different semantics of time.clock() have confused people around here. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OFF-TOPIC-BUT-IMPORTANT: End of free internet model by 2012 ?
On Wed, Jun 11, 2008 at 4:25 PM, Daniel Bonekeeper <[EMAIL PROTECTED]> wrote: > Hi everybody on the list ! > > Sorry about the off-topic message, but giving the nature of the > message and the nature of the list (after all, everybody here uses > Internet or develops for web in some way), I decided to post this here > and see what you guys think about it. > My gut feeling is that you are being taken by some fairly amateurish viral marketing, and that you've played into it by uncritically regurgitating the message onto these mailing lists. Googling "Dylan Pattyn" yields only references to this very video and a site for a performance art/activist group called IPower, which boasts such "projects" as "an suicide-awareness campaign disguised as a personal suicide blog", a self-described "online stunt" called "40,000 Blowjobs", and a "wacky online reality series". Really, do you honestly think that Time Magazine would use a twenty-something freelancer, who has never published anything in any major periodical, to break a news article that would at the least create a decade or more of anti-trust lawsuits? -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Tue, Jan 20, 2009 at 08:02, Luke Kenneth Casson Leighton wrote: > of course - if python for win32 ENTIRELY DROPPED msvc as a development > platform, and went for an entirely free software development > toolchain, then this problem goes away. That's a non-starter for anyone who incorporates Python in an existing MSVC-based development environment. When in Rome... -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in generator if the generator in created in a C thread
On Wed, Mar 28, 2012 at 06:45, Victor Stinner wrote:
> We have a crash in our product when tracing is enabled by
> sys.settrace() and threading.settrace(). If a Python generator is
> created in a C thread, calling the generator later in another thread
> may crash if Python tracing is enabled.
>
> - the C thread calls PyGILState_Ensure() which creates a temporary
> Python thread state
> - a generator is created, the generator has a reference to a Python
> frame which keeps a reference to the temporary Python thread state
> - the C thread calls PyGILState_Releases() which destroys the
> temporary Python thread state
> - when the generator is called later in another thread, call_trace()
> reads the Python thread state from the generator frame, which is the
> destroyed frame => it does crash on a pointer dereference if the
> memory was reallocated (by malloc()) and the data were erased
>
This is timely. We've seen several similar bugs in our 3.1.2-based
embedded product--in fact, I'm tracking down what might be another
this week.
The common theme is that C extension code that depends on
PyGILState_Ensure() for GIL management runs the risk of causing latent
crashes in any situation that involves preserving a frame beyond the
lifetime of the thread that created it.
In our case, the culprits have usually been the frames attached to
exceptions thrown across the extension->Python->embedding app
boundaries.
From a theoretical standpoint, I can't quite decide what the real error is:
1) the fact that PyGILState_Release() destroys a temporary thread
state that may still be referenced by some objects, or
2) the fact that some code is trying to keep frame objects after the
creating thread state no longer exists.
This week I've been leaning toward 2), but then I realized that
keeping frames post-thread-death is not that uncommon (for example,
debuggers and other diagnostic techniques like
http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html).
Locally we added some (unfortunate) code to our 3.1.2 port to wrap
PyGILState_Ensure(), which I thought had sidestepped the issue for us:
void takeGIL()
{
PyGILState_Ensure();
// This has the side effect of keeping such thread states alive until
// the interpreter is finalized; however, all thread state objects get
// unconditionally deleted during Py_Finalize, so they won't leak.
PyThreadState* pThreadState = PyGILState_GetThisThreadState();
if (pThreadState->gilstate_counter == 1)
{
++pThreadState->gilstate_counter;
}
}
But clearly that can't be a correct answer (and it may not even be a
functioning one, given that I'm seeing a similar issue again).
--
Tim Lesher
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in generator if the generator in created in a C thread
On Thu, Mar 29, 2012 at 17:26, Victor Stinner wrote: > The problem is not the frame, but the Python thread state referenced by the > frame. It's a private attribute. My patch just updates this reference before > running the generator (and it clears the reference when the generator > excution is stopped or finished). Right--my thought was the the situation we saw might be similarly triggered because we were storing an exception with traceback (and associated frames) generated by thread A, and then re-throwing it from thread B some time after thread A has exited. The frame attached to the exception's traceback would still, then, be referencing a nonexistent thread state, correct? My concern was that this might one instance of a broader problem for folks who embed Python and see the attractive PyGILState_Ensure() API. It already prevents us from using subinterpreters (which for us might have been a better solution than repeated initialize/finalize, with its known issues). We recently made a change to dispose of the traceback before storing the exception, and that appears to eliminate the corruption we were seeing, so that's making me more suspicious. > You may leak memory if your threads have a short lifetime and you create > many threads. For example if one thread is only used to process one request > and then is destroyed. Absolutely--this particular hack was only used for a thread created outside Python that had to call into the VM. Their behavior is well-defined in our case--two particular OS threads, with lifetimes longer than those of the interpreters we create and finalize. -- Tim Lesher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python-dev Summary for 2005-02-01 through 2005-02-14 [draft]
Here are sample summaries of two "skipped threads". If anyone has
comments, criticisms, or rotten tomatoes, let me know.
---
Replacing list of constants with tuple of constants
---
Skip Montanaro noticed that Raymond Hettinger had made a number of
library checkins replacing lists with tuples in constructs like ``for
a in [1, 2, 3]:`` and ``if a in [1, 2, 3]:``. He agreed with the
motivation (the peephole optimizer can convert a tuple of constants
into a single constant, eliminating construction time), but questioned
hardcoding the optimization. Instead, he suggested teaching the
optimizer about "throwaway" lists in ``for`` and ``if`` statements.
Guido agreed with the sentiment. Raymond accepted the suggestion, and
checked in code to implement it.
Contributing threads:
- `list of constants -> tuple of constants
<http://mail.python.org/pipermail/python-dev/2005-February/051442.html>`__
---
Complex I/O problem
---
Neal Becker asked why the result of printing a ``complex`` is not an
acceptable input to constructing a complex. For example, ``print
complex(2,2)`` yields ``'(2+2j)'``; but ``complex('(2+2j)')`` throws a
``ValueError``.
A. M. Kuchling responded that many types, including ``str`` and
``file`` share this behavior, and that in any event, parsing string
representations is a job more suited to ``eval`` than to classes
themselves.
Guido `reiterated the rules`_ for ``str()`` (used by ``print``) and
``repr()``. Since both ``complex.__str__`` and ``complex.__repr__``
pass the ``eval()`` test, he pronounced it fine.
.. _reiterated the rules:
http://mail.python.org/pipermail/python-dev/2005-February/051390.html
Contributing threads:
- `complex I/O problem
<http://mail.python.org/pipermail/python-dev/2005-February/051388.html>`__
--
Tim Lesher <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Ye don't be needin' these!
On Wed, 23 Mar 2005 11:34:09 -0500, Terry Reedy <[EMAIL PROTECTED]> wrote: > > "Herman Toothrot" <[EMAIL PROTECTED]> wrote in message > news:[EMAIL PROTECTED] > > Avast! Why be there builtins divmod and pow, when operators **, /, and % > > should be good enough for ya? It runs counter to TOOWTDI, I be thinking. > > Questions like this should be asked on comp.lang.python or the python > mailing list. I'll answer if I see it there. I have to wonder if this wasn't a tongue-in-cheek message sent from a just-created hotmail account, by an existing python-dev participant who's embroiled in the current "do we even need functionals anymore" discussion... -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Re: Ye don't be needin' these!
On Wed, 23 Mar 2005 22:58:51 +0100, Florian Schulze <[EMAIL PROTECTED]> wrote: > BTW, Herman Toothrot is from Monkey Island. Right. That's what leads me to believe 1) it's not a serious post, and 2) it's from someone who's old enough to know better. -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] python-dev Summary for 2005-04-01 through 2005-04-15 [draft]
Here's the first draft of the python-dev summary for the first half of
April. Please send any corrections or suggestions to the summarizers.
==
Summary Announcements
==
---
New python-dev summary team
---
This summary marks the first by the team of Steve Bethard, Tim Lesher,
and Tony Meyer. We're trying a collaborative approach to the
summaries: each fortnight, we'll be getting together in a virtual
smoke-filled back room to divide up the interesting threads. Then
we'll stitch together the summaries in roughly the same form as you've
seen in the past. We'll mark each editor's entries with his initials.
Thanks to Brett Cannon for sixty-one excellent python-dev summaries.
Also, thanks for providing scripts to help get the new summaries off
the ground! We're looking forward to the contributions you'll make to
the Python core, now that the summaries aren't taking up all your
time.
[TDL]
=
Summaries
=
--
Right Operator Methods
--
Greg Ewing explored an issue with new-style classes that define only
right operator methods (__radd__, __rmul__, etc.) Instances of such
a class cannot be added/multiplied/etc. together as Python raises a
TypeError. Armin Rigo explained the rule: if the instances on both sides
of an operator are of the same class, only the non-reversed method is
ever called. Armin also explained that an __add__ or __mul__ method that
returns NotImplemented may be called twice when Python attempts to
differentiate between numeric and sequence operations.
Contributing threads:
- `New style classes and operator methods
<http://mail.python.org/pipermail/python-dev/2005-April/052577.html>`__
[SJB]
--
Hierarchical groups in regular expressions
--
Chris Ottrey demoed his `pyre2 project`_ that can extract a hierarchy of
strings when nested groups match in a regular expression. The current
re module (in the stdlib) only matches the last occurrence of a group in
the string, throwing away any preceding matches. People discussed some
of pyre2's proposed API, with the main suggestion being to extend the
API to support unnamed (positional) groups in addition to named groups.
Though a number of people expressed interest in the idea, it was not
clear whether the functionality should be included in the standard
library. However, most agreed that if it was included, it should
be integrated with the existing re module. Gustavo Niemeyer offered to
perform this integration if an API could be agreed upon. Further
discussion was moved to the pyre2 `development wiki`_ and `mailing
list`_.
Contributing threads:
- `hierarchicial named groups extension to the re library
<http://mail.python.org/pipermail/python-dev/2005-April/052508.html>`__
.. _pyre2 project: http://pyre2.sourceforge.net/
.. _development wiki: http://py.redsoft.be/pyre2/wiki/
.. _mailing list: http://lists.sourceforge.net/lists/listinfo/pyre2-devel
[SJB]
---
Security capabilities in Python
---
The issue of security came up again, and Ka-Ping Yee suggested that in
Python's restricted execution mode secure proxies can be created by
using lexical scoping. He posted `some code`_ for revealing only
certain "facets" of an object by using a function to declare a proxy
class that used function local variables to build the proxy. Thus to
access the attributes used in the proxy class, you need to access
things like im_func or func_closure, which are not accessible in
restricted execution mode.
James Y Knight illustrated how strategic overriding of __eq__ in a
subclass of str could allow access to the hidden "facets". Eyal Lotem
suggested that such an attack could be countered by implementing
"facets" in C, but having to turn to C every time you needed a
particular security construct seemed unappealing.
Contributing threads:
- `Security capabilities in Python
<http://mail.python.org/pipermail/python-dev/2005-April/052580.html>`__
.. _some code: http://zesty.ca/python/facet.py
[SJB]
-
Improving GilState API Robustness
-
Michael Hudson noted that his changes to thread handling in the
readline module appeared to trigger `bug 1176893`_ ("Readline
segfault"). However, he believed the problem lay in the GilState API,
rather than in his changes: PyGilState_Release crashes if
PyEval_InitThreads wasn't called, even if the code you're writing
doesn't use multiple threads.
He proposed several solutions, none of which met with resounding
approbation, and Tim Peters noted that `PEP 311`_, Simplified Global
Interpreter Lock Acquisition for Extensions, "specifically disowns
res
[Python-Dev] python-dev Summary for 2005-05-01 through 2005-05-15 [draft]
a non-feature request <http://mail.python.org/pipermail/python-dev/2005-May/053653.html>`__ - `Python 2.4 set objects and cyclic garbage <http://mail.python.org/pipermail/python-dev/2005-May/053630.html>`__ - `CHANGE BayPIGgies: May *THIRD* Thurs <http://mail.python.org/pipermail/python-dev/2005-May/053628.html>`__ - `Python continually calling sigprocmask() on FreeBSD 5 <http://mail.python.org/pipermail/python-dev/2005-May/053615.html>`__ - `Weekly Python Patch/Bug Summary <http://mail.python.org/pipermail/python-dev/2005-May/053213.html>`__ - `problems with memory management <http://mail.python.org/pipermail/python-dev/2005-May/053408.html>`__ - `Adding DBL_MANTISSA and such to Python <http://mail.python.org/pipermail/python-dev/2005-May/053372.html>`__ - `python-dev Summary for 2005-04-16 through 2005-04-30 [draft] <http://mail.python.org/pipermail/python-dev/2005-May/053383.html>`__ - `Python Language track at Europython, still possibilities to submit talks <http://mail.python.org/pipermail/python-dev/2005-May/053303.html>`__ - `(no subject) <http://mail.python.org/pipermail/python-dev/2005-May/053196.html>`__ - `Kernel panic writing to /dev/dsp with cmpci driver <http://mail.python.org/pipermail/python-dev/2005-May/053627.html>`__ -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thoughts on stdlib evolvement
On 6/6/05, Reinhold Birkenfeld <[EMAIL PROTECTED]> wrote: > - Flat namespace: Should we tend to a more hierarchic library (e.g. > inet.url, inet.http, inet.nntp)? This would increase clarity when > searching for a module. -1. I feel the opposite way: when trying to figure out where something "lives", I prefer Python's flat namespace to (for example) Java's com.company.baz.bar.foo hierarchy. > - 3rd party modules: There are many superb modules out there, some of which > really do have a "standard" character. Examples include PIL, numarray, > ElementTree, [wxPython - I know this is a hairy issue], I think the most important question for each of these is "is the module's release schedule at least as stable as Python's?". For many of these, I suspect the answer is "no". -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] should doc string content == documentation content?
On 7/24/05, Tim Peters <[EMAIL PROTECTED]> wrote: > I'm sure there is , but via a different route: tools to extract > text from the full documentation, not to burden docstrings with an > impossible task. Channeling Guido, docstrings are best when they have > a "quick reference card" feel, more memory aid than textbook. That > doesn't mean it wouldn't also be nice to have "the textbook" online > from pydoc/help too, it means that manuals and docstrings serve > different purposes. While I agree that docstrings shouldn't be a deep copy of _Python in a Nutshell_, there are definitely some areas of the standard library that could use some help. threading comes to mind immediately. -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bare except clauses in PEP 348
On 8/25/05, Sjoerd Mullender <[EMAIL PROTECTED]> wrote: > There is an important point, though. Recently I read complaints about > the lack of backward compatibility in Python on the fedora-list (mailing > list for users of Fedora Core). Somebody asked what language he should > learn and people answered, don't learn Python because it changes too > often in backward incompatible ways. They even suggested using that > other P language because that was much more backward compatible. I think you're overstating what actually happened there. Here's the actual quote from the thread: : perl is more portable than python - programs written for perl are far : more likely to run on a new version of perl than the equivalent for : python. However, python is probably more readable and writable than perl : for a new user, and is the language most Fedora system utilities (e.g. : yum) are written in. Both perl and python run on Windows too. : : You have to be very careful about how you write your code to make it : portable to both environments. If you need a GUI, you'll need a : cross-platform GUI toolkit like Qt too. : : If it's only one language to learn, and you're a Fedora user, I'd go for : python. Yes, later there were additional posts about portability and backwards-compatibility, but they were for the most part factually incorrect (reliance on new 2.x features, not backwards-incompatibility, were the issue with CML1) and relied to "I heard that..." information So your point is well-taken, but the problem is one of user perception. That's not a dismissal of the problem--witness the "JAVA/LISP/Python is too slow" and "all PERL code is cryptic" memes. To me, this perception problem alone raises the bar on backwards compatibility. Even if obsoleted features are seldom useed, "$language breaks old code!" is a virulent meme, in both senses of the word. -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/19/05, Michael Hudson <[EMAIL PROTECTED]> wrote: > I was disappointed that that article (hey, it was the only issue of > ddj I've ever actually bought! :) didn't consider any concurrency > models other than shared memory threading. The problem is that, for all its limitations, shared-memory threading is the most popular concurrency model on the most popular operating system, so future hardware platforms targeting that system will be optimizing for that case. We can either rail against the sea, or accept it. -- Tim Lesher <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
