Re: [Python-Dev] Access control for buildbot
On Dec 10, 2014, at 6:56 AM, jacob toft pedersen wrote: > Hi there > > I was visiting you buildbot page for inspiration and found that i apparently > have the option to force stop/start all your builds without any access > control. > > You may want to put something to enforce access control? > Nah, as far as I know, no-one has abused it, and it’s definitely useful when you need to legitimately use it. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes & bytearray
On Tue, Jan 20, 2015 at 11:48:10AM +0200, Paul Sokolovsky wrote:
> Hello,
>
> On Tue, 20 Jan 2015 18:15:02 +1300
> Greg Ewing wrote:
>
> > Guido van Rossum wrote:
> > > On Mon, Jan 19, 2015 at 11:43 AM, Paul Sokolovsky
> > > mailto:pmis...@gmail.com>> wrote:
> > >
> > > b.lower_inplace()
> > > b.lower_i()
> > >
> > > Please don't go there. The use cases are too rare.
> >
> > And if you have such a use case, it's not too
> > hard to do
> >
> >b[:] = b.lower()
>
> The point of inplace operations (memoryview's, other stuff already in
> Python) is to avoid unneeded memory allocation and copying. For 1Tb
> bytearray with 1Tb of RAM, it will be very hard to do. (Ditto for 100K
> bytearray with 150K RAM.)
You can just loop through the bytearray and assign elements. I use
something along the lines of this for PyParallel where I'm operating on
bytearrays that are backed by underlying socket buffers, where I don't
want to do any memory allocations/reallocations:
def toupper_bytes(data):
assert isinstance(data, bytearray)
a = ord('a')
z = ord('z')
for i in range(0, len(data)):
c = data[i]
if c >= a and c <= z:
data[i] = c - 32
Low overhead, mostly stays within the same ceval frame. Should be a
walk in the park for PyPy, Cython or Numba to optimize, too.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Can we clean up the buildbots please?
On Fri, May 22, 2015 at 10:06:48PM +, Steve Dower wrote: > The Windows 7 buildbots are failing on test_asdl_parser, but I have no > idea why – the test works for me just fine. Yury and Benjamin made the > most recent changes to Python.asdl, but I have no idea what effect > they would have here, or why it’s Windows only. > > The WS2K3 machine needs a reboot – I pinged Trent about that months > ago – and the XP one isn’t supported for 3.5. Gave it a little bit of love just then (haven't been able to access it for months as the main switch needed a reboot). There were like, 155 cl.exe processes wedged and a bunch of error reporting dialogs. Do we still support WS2K3? (Can I even install VS 2015 on that? I would have thought not.) Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Can we clean up the buildbots please?
On Fri, May 22, 2015 at 05:24:53PM -0700, Larry Hastings wrote: > On 05/22/2015 05:11 PM, Trent Nelson wrote: > >Do we still support WS2K3? (Can I even install VS 2015 on that? I would > >have thought not.) > > According to PCbuild/readme.txt, no. It says: > >This directory is used to build CPython for Microsoft Windows NT >version 6.0 or higher (Windows Vista, Windows Server 2008, or later) >on 32 and 64 bit platforms. Ah, yeah, thought so. Pity, that box is probably the only one that hasn't had any form of hardware failure during its tenure ;-) Tried to get the W2K8 one back up on Monday when I had some remote hands but alas, no luck. Think it has balked HDDs or something. The Solaris 11 AMD64 one Solaris 10 SPARC ones are back up now though and I just cleared out their 700+ build backlogs, FWIW. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyParallel update
[CC'ing python-dev@ for those that are curious; please drop and keep follow-up discussion to python-ideas@] Hi folks, I've made a lot of progress on PyParallel since the PyCon dev summit (https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I fixed the outstanding breakage with generators, exceptions and whatnot. I got the "instantaneous Wiki search server" working[1] and implemented the entire TechEmpower Frameworks Benchmark Suite[2], including a PyParallel-friendly pyodbc module, allowing database connections and querying in parallel. [1]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py [2]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/tefb/tefb.py I set up a landing page for the project: http://pyparallel.org And there was some good discussion on reddit earlier this week: https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experimental_proofofconcept_fork_of/ I've put together some documentation on the project, its aims, and the key parts of the solution regarding the parallelism through simple client/server paradigms. This documentation is available directly on the github landing page for the project: https://github.com/pyparallel/pyparallel Writing that documentation forced me to formalize (or at least commit) to the restrictions/trade-offs that PyParallel would introduce, and I'm pretty happy I was basically able to boil it down into a single rule: Don't persist parallel objects. That keeps the mental model very simple. You don't need to worry about locking or ownership or races or anything like that. Just don't persist parallel objects, that's the only thing you have to remember. It's actually really easy to convert existing C code or Python code into something that is suitable for calling from within a parallel callback by just ensuring that rule isn't violated. It took about four hours to figure out how NumPy allocated stuff and add in the necessary PyParallel-aware tweaks, and not that much longer for pyodbc. (Most stuff "just works", though.) (The ABI changes would mean this is a Python 4.x type of thing; there are fancy ways we could avoid ABI changes and get this working on Python 3.x, but, eh, I like the 4.x target. It's realistic.) The other thing that clicked is that asyncio and PyParallel would actually work really well together for exploiting client-driven parallelism (PyParallel really is only suited to server-oriented parallelism at the moment, i.e. serving HTTP requests in parallel). With asyncio, though, you could keep the main-thread/single-thread client-drives-computation paradigm, but have it actually dispatch work to parallel.server() objects behind the scenes. For example, in order to process all files in a directory in parallel, asyncio would request a directory listing (i.e. issue a GET /) which the PyParallel HTTP server would return, it would then create non-blocking client connections to the same server and invoke whatever HTTP method is desired to do the file processing. You can either choose to write the new results from within the parallel context (which could then be accessed as normal files via HTTP), or you could have PyParallel return json/bytes, which could then be aggregated by asyncio. Everything is within the same process, so you get all the benefits that provides (free access to anything within scope, like large data structures, from within parallel contexts). You can synchronously call back into the main thread from a parallel thread, too, if you wanted to update a complex data structure directly. The other interesting thing that documentation highlights is the advantage of the split brain "main thread vs parallel thread" GC and non-GC allocators. I'm not sure if I've ever extolled the virtue of such an approach on paper or in e-mail. It's pretty neat though and allows us to avoid a whole raft of problems that need to be solved when you have a single GC/memory model. Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase. That'll require a bit of churn, so if there's enough interest from others, I figured we'd use the opportunity to at least get it building again on POSIX (Linux/OSX/FreeBSD). From there people can start implementing the missing bits for implementing the parallel machinery behind the scenes. The parallel interpreter thread changes I made are platform agnostic, the implementation just happens to be on Windows at the moment; don't let the Windows-only thing detract from what's actually being pitched: a (working, demonstrably-performant) solution to "Python's GIL problem". Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Tue, Sep 08, 2015 at 10:12:37AM -0400, Gary Robinson wrote: > There was a huge data structure that all the analysis needed to > access. Using a database would have slowed things down too much. > Ideally, I needed to access this same structure from many cores at > once. On a Power8 system, for example, with its larger number of > cores, performance may well have been good enough for production. In > any case, my experimentation and prototyping would have gone more > quickly with more cores. > > But this data structure was simply too big. Replicating it in > different processes used memory far too quickly and was the limiting > factor on the number of cores I could use. (I could fork with the big > data structure already in memory, but copy-on-write issues due to > reference counting caused multiple copies to exist anyway.) This problem is *exactly* the type of thing that PyParallel excels at, just FYI. PyParallel can load large, complex data structures now, and then access them freely from within multiple threads. I'd recommended taking a look at the "instantaneous Wikipedia search server" example as a start: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py That loads trie with 27 million entries, creates ~27.1 million PyObjects, loads a huge NumPy array, and has a WSS of ~11GB. I've actually got a new version in development that loads 6 tries of the most frequent terms for character lengths 1-6. Once everything is loaded, the data structures can be accessed for free in parallel threads. There are more details regarding how this is achieved on the landing page: https://github.com/pyparallel/pyparallel I've done a couple of consultancy projects now that were very data science oriented (with huge data sets), so I really gained an appreciation for how common the situation you describe is. It is probably the best demonstration of PyParallel's strengths. > Gary Robinson gary...@me.com http://www.garyrobinson.net Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Wed, Sep 09, 2015 at 01:43:19PM -0700, Ethan Furman wrote: > On 09/09/2015 01:33 PM, Trent Nelson wrote: > > >This problem is *exactly* the type of thing that PyParallel excels at [...] > > Sorry if I missed it, but is PyParallel still Windows only? Yeah, still Windows only. Still based off 3.3.5. I'm hoping to rebase off 3.5 after its tagged and get it into a state where it can at least build on POSIX (i.e. stub enough functions such that it'll compile). That's going to be a lot of work though, would love to get some help with it. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Wed, Sep 09, 2015 at 04:52:39PM -0400, Gary Robinson wrote: > I’m going to seriously consider installing Windows or using a > dedicated hosted windows box next time I have this problem so that I > can try your solution. It does seem pretty ideal, although the STM > branch of PyPy (using http://codespeak.net/execnet/ to access SciPy) > might also work at this point. I'm not sure how up-to-date this is: http://pypy.readthedocs.org/en/latest/stm.html But it sounds like there's a 1.5GB memory limit (or maybe 2.5GB now, I just peaked at core.h linked in that page) and a 4-core segment limit. PyParallel has no memory limit (although it actually does have support for throttling back memory pressure by not accepting new connections when the system hits 90% physical memory used) and no core limit, and it scales linearly with cores+concurrency. PyPy-STM and PyParallel are both pretty bleeding edge and experimental though so I'm sure we both crash as much as each other when exercised outside of our comfort zones :-) I haven't tried getting the SciPy stack running with PyParallel yet. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion related to memory leaks requested
Gflags/umdh is pretty useful on Windows, I used it to track down a few quirky PyParallel memory leaks. Steps: 1. Enable global flags: gflags –i python.exe +ust 2. Launch Python. 3. Enable the umdh tracer: umdh –p: -f:d1.log 4. Kill it after a short run. 5. Re-launch Python. 6. Enable it again: umdh –p: -f:d2.log 7. Let it run for longer – long enough to make sure it’s leaking memory. 8. Kill it, then generate a dump file: umdh –d d1.log d2.log > dump.txt (Those steps were pretty specific to my particular situation, but it should at least be a reasonable starting point for what to google to find out more.) Here are two sample outputs that pin-pointed the exact leak path: + 49116 ( 49116 - 0) 6 allocsBackTrace9763CA0 + 6 ( 6 - 0)BackTrace9763CA0allocations ntdll!RtlpCallInterceptRoutine+40 ntdll!RtlAllocateHeap+79846 SQLSRV32!SQLAllocateMemory+26 markSQLSRV32!SQLAllocConnect+F6 SQLSRV32!SQLAllocHandle+83 ODBC32!RetcodeDriverInit+2D9 ODBC32!SQLInternalDriverConnectW+2F ODBC32!CDispenser::CreateResource+DB comsvcs!CHolder::SafeDispenserDriver::CreateResource+43 comsvcs!CHolder::AllocResource+24D ODBC32!CDispenser::TryAllocResource+6E ODBC32!CDispenser::GetActiveConnection+72 ODBC32!SQLDriverConnectW+9D4 pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85) pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166) pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378) python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84) python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500) python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639) python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036) python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353) python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036) python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011) python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128) + 48432 ( 48432 - 0) 6 allocsBackTrace97635E0 + 6 ( 6 - 0)BackTrace97635E0allocations ntdll!RtlpCallInterceptRoutine+40 ntdll!RtlAllocateHeap+79846 SQLSRV32!SQLAllocateMemory+26 SQLSRV32!SQLAllocConnect+4D SQLSRV32!SQLAllocHandle+83 ODBC32!RetcodeDriverInit+2D9 ODBC32!SQLInternalDriverConnectW+2F ODBC32!CDispenser::CreateResource+DB comsvcs!CHolder::SafeDispenserDriver::CreateResource+43 comsvcs!CHolder::AllocResource+24D ODBC32!CDispenser::TryAllocResource+6E ODBC32!CDispenser::GetActiveConnection+72 ODBC32!SQLDriverConnectW+9D4 pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85) pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166) pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378) python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84) python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500) python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\fu
Re: [Python-Dev] The pysandbox project is broken
On Tue, Nov 12, 2013 at 01:16:55PM -0800, Victor Stinner wrote: > pysandbox cannot be used in practice > > > To protect the untrusted namespace, pysandbox installs a lot of > different protections. Because of all these protections, it becomes > hard to write Python code. Basic features like "del dict[key]" are > denied. Passing an object to a sandbox is not possible to sandbox, > pysandbox is unable to proxify arbitary objects. > > For something more complex than evaluating "1+(2*3)", pysandbox cannot > be used in practice, because of all these protections. Individual > protections cannot be disabled, all protections are required to get a > secure sandbox. This sounds a lot like the work I initially did with PyParallel to try and intercept/prevent parallel threads mutating main-thread objects. I ended up arriving at a much better solution by just relying on memory protection; main thread pages are set read-only prior to parallel threads being able to run. If a parallel thread attempts to mutate a main thread object; a SEH is raised (SIGSEV on POSIX), which I catch in the ceval loop and convert into an exception. See slide 138 of this: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 I'm wondering if this sort of an approach (which worked surprisingly well) could be leveraged to also provide a sandbox environment? The goals are the same: robust protection against mutation of memory allocated outside of the sandbox. (I'm purely talking about memory mutation; haven't thought about how that could be extended to prevent file system interaction as well.) Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The pysandbox project is broken
On Nov 15, 2013, at 12:34 PM, Victor Stinner wrote:
> 2013/11/15 Trent Nelson :
>>This sounds a lot like the work I initially did with PyParallel to
>>try and intercept/prevent parallel threads mutating main-thread
>>objects.
>>
>>I ended up arriving at a much better solution by just relying on
>>memory protection; main thread pages are set read-only prior to
>>parallel threads being able to run. If a parallel thread attempts
>>to mutate a main thread object; a SEH is raised (SIGSEV on POSIX),
>>which I catch in the ceval loop and convert into an exception.
>
> Read-only is not enough, an attack must not be able to read sensitive data.
Well you could remove both write *and* read perms from pages, such that you
would trap on read attempts too. What's an example of sensitive data that
you'd need to have residing in the same process that you also want to sandbox?
I was going to suggest something like:
with memory.protected:
htpasswd = open('htpasswd', 'r').read()
...
But then I couldn't think of why you'd persist the sensitive data past the
point you'd need it.
> Protections of memory pages sound very low-level, so not very portable :-/
It's a pretty fundamental provision provided by operating systems; granted, the
interface differs (mprotect() versus VirtualProtect()), but the result is the
same.
> How do you know fif SIGSEGV comes from a legal call (parallel thread
> thing) or a real bug?
You don't, but it doesn't really matter. It'll be pretty obvious from looking
at the offending line of code in the exception whether it was a legitimate
memory protection error, or a bug in an extension module/CPython internals.
And having a ProtectionError bubble all the way back up to the top of the stack
with exact details about the offending frame/line could be considered a nicer
alternative to dumping core ;-) (Unless you happen to be in an `except: pass`
block.)
> Victor
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The pysandbox project is broken
On Sat, Nov 16, 2013 at 02:53:22AM -0800, Maciej Fijalkowski wrote: > On Fri, Nov 15, 2013 at 6:56 PM, Trent Nelson wrote: > > On Tue, Nov 12, 2013 at 01:16:55PM -0800, Victor Stinner wrote: > >> pysandbox cannot be used in practice > >> > >> > >> To protect the untrusted namespace, pysandbox installs a lot of > >> different protections. Because of all these protections, it becomes > >> hard to write Python code. Basic features like "del dict[key]" are > >> denied. Passing an object to a sandbox is not possible to sandbox, > >> pysandbox is unable to proxify arbitary objects. > >> > >> For something more complex than evaluating "1+(2*3)", pysandbox cannot > >> be used in practice, because of all these protections. Individual > >> protections cannot be disabled, all protections are required to get a > >> secure sandbox. > > > > This sounds a lot like the work I initially did with PyParallel to > > try and intercept/prevent parallel threads mutating main-thread > > objects. > > > > I ended up arriving at a much better solution by just relying on > > memory protection; main thread pages are set read-only prior to > > parallel threads being able to run. If a parallel thread attempts > > to mutate a main thread object; a SEH is raised (SIGSEV on POSIX), > > which I catch in the ceval loop and convert into an exception. > > > > See slide 138 of this: > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 > > > > I'm wondering if this sort of an approach (which worked surprisingly > > well) could be leveraged to also provide a sandbox environment? The > > goals are the same: robust protection against mutation of memory > > allocated outside of the sandbox. > > > > (I'm purely talking about memory mutation; haven't thought about how > > that could be extended to prevent file system interaction as well.) > > > > > > Trent. > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > > Trent, you should read the mail more carefully. Notably the same > issues that make it impossible to create a sandbox make it impossible > to create pyparaller really work. Being read-only is absolutely not > enough - you can read some internal structures in inconsistent state > that lead to crashes and/or very unexpected behavior even without > modifying anything. What do you mean by inconsistent state? Like a dict half way through `a['foo'] = 'bar'`? That can't happen with PyParallel; parallel threads don't run when the main thread runs and vice versa. The main thread's memory (and internal object structure) will always be consistent by the time the parallel threads run. > PS. We really did a lot of work analyzing how STM-pypy can lead to > conflicts and/or inconsistent behavior. But you support free-threading though, right? As in, code that subclasses threading.Thread should be able to benefit from your STM work? I explicitly don't support free-threading. Your threading.Thread code will not magically run faster with PyParallel. You'll need to re-write your code using the parallel and async façade APIs I expose. On the plus side, I can completely control everything about the main thread and parallel thread execution environments; obviating the need to protect against internal inconsistencies by virtue of the fact that the main thread will always be in a consistent state when the parallel threads are running. (And it works really well in practice; I ported SimpleHTTPServer to use my new async stuff and it flies -- it'll automatically exploit all your cores if there is sufficient incoming load. Unexpected side-effect of my implementation is that code executing in parallel callbacks actually runs faster than normal single-threaded Python code; no need to do reference counting, GC, and the memory model is ridiculously cache and TLB friendly.) This is getting off-topic though and I don't want to hijack the sandbox thread. I was planning on sending an e-mail in a few days when the PyData video of my talk is live -- we can debate the merits of my parallel/async approach then :-) > Cheers, > fijal Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyParallel: alternate async I/O and GIL removal
Hi folks, Video of the presentation I gave last weekend at PyData NYC regarding PyParallel just went live: https://vimeo.com/79539317 Slides are here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 The work was driven by the async I/O discussions around this time last year on python-ideas. That resulted in me sending this: http://markmail.org/thread/kh3qgjbydvxt3exw#query:+page:1+mid:arua62vllzugjy2v+state:results where I attempted to argue that there was a better way of doing async I/O on Windows than the status quo of single-threaded, non-blocking I/O with an event multiplex syscall. I wasn't successful in convincing anyone at the time; I had no code to back it up and I didn't articulate my plans for GIL removal at the time either (figuring the initial suggestion would be met with enough scepticism as is). So, in the video above, I spend a lot of time detailing how IOCP works on Windows, how it presents us with a better environment than UNIX for doing asynchronous I/O, and how it paired nicely with the other work I did on coming up with a way for multiple threads to execute simultaneously across all cores without introducing any speed penalties. I'm particularly interested to hear if the video/slides helped UNIX-centric people gain a better understanding of how Windows does IOCP and why it would be preferable when doing async I/O. The reverse is also true: if you still think single-threaded, non- blocking synchronous I/O via kqueue/epoll is better than the approach afforded by IOCP, I'm interested in hearing why. As crazy as it sounds, my long term goal would be to try and influence Linux and BSD kernels to implement thread-agnostic I/O support such that an IOCP-like mechanism could be exposed; Solaris and AIX already do this via event ports and AIX's verbatim copy of Windows' IOCP API. (There is some promising work already being done on Linux; see recent MegaPipe paper for an example.) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Wed, Feb 27, 2013 at 08:51:16AM -0800, Michael Foord wrote: > If you have other items you'd like to discuss please let me know and I > can add them to the agenda. Hmm, seems like this might be a good forum to introduce the parallel/async stuff I've been working on the past few months. TL;DR version is I've come up with an alternative approach for exploiting multiple cores that doesn't rely on GIL-removal or STM (and has a negligible performance overhead when executing single-threaded code). (For those that are curious, it lives in the px branch of the sandbox/trent repo on hg.p.o, albeit in a very experimental/prototype/proof-of-concept state (i.e. it's an unorganized, undocumented, uncommented hackfest); on the plus side, it works. Sort of.) Second suggestion: perhaps a little segment on Snakebite? What it is, what's available to committers, feedback/kvetching from those who have already used it, etc. (I forgot the format of these summits -- is there a projector?) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Slides from today's parallel/async Python talk
Just posted the slides for those that didn't have the benefit of attending the language summit today: https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 05:21:09AM -0700, Christian Heimes wrote: > Am 14.03.2013 03:05, schrieb Trent Nelson: > > Just posted the slides for those that didn't have the benefit of > > attending the language summit today: > > > > > > https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async > > Wow, neat! Your idea with Py_PXCTC is ingenious. Yeah, it's funny how the viability and performance of the whole approach comes down to a quirky little trick for quickly detecting if we're in a parallel thread ;-) I was very chuffed when it all fell into place. (And I hope the quirkiness of it doesn't detract from the overall approach.) > As far as I remember the FS and GS segment registers are used by most > modern operating systems on x86 and x86_64 platforms nowadays to > distinguish threads. TLS is implemented with FS and GS registers. I > guess the __read[gf]sdword() intrinsics do exactly the same. Yup, in fact, if I hadn't come up with the __read[gf]sword() trick, my only other option would have been TLS (or the GetCurrentThreadId /pthread_self() approach in the presentation). TLS is fantastic, and it's definitely an intrinsic part of the solution (the "Y" part of "if we're a parallel thread, do Y"), but it definitely more costly than a simple FS/GS register read. > Reading > registers is super fast and should have a negligible effect on code. Yeah the actual instruction is practically free; the main thing you pay for is the extra branch. However, most of the code looks like this: if (Py_PXCTX) something_small_and_inlineable(); else Py_INCREF(op); /* also small and inlineable */ In the majority of the cases, all the code for both branches is going to be in the same cache line, so a mispredicted branch is only going to result in a pipeline stall, which is better than a cache miss. > ARM CPUs don't have segment registers because they have a simpler > addressing model. The register CP15 came up after a couple of Google > searches. Noted, thanks! > IMHO you should target x86, x86_64, ARMv6 and ARMv7. ARMv7 is going to > be more important than x86 in the future. We are going to see more ARM > based servers. Yeah that's my general sentiment too. I'm definitely curious to see if other ISAs offer similar facilities (Sparc, IA64, POWER etc), but the hierarchy will be x86/x64 > ARM > * for the foreseeable future. Porting the Py_PXCTX part is trivial compared to the work that is going to be required to get this stuff working on POSIX where none of the sublime Windows concurrency, synchronisation and async IO primitives exist. > Christian Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Wed, Mar 13, 2013 at 07:05:41PM -0700, Trent Nelson wrote: > Just posted the slides for those that didn't have the benefit of > attending the language summit today: > > > https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async Someone on /r/python asked if I could elaborate on the "do Y" part of "if we're in a parallel thread, do Y, if not, do X", which I (inadvertently) ended up replying to in detail. I've included the response below. (I'll work on converting this into a TL;DR set of slides soon.) > Can you go into a bit of depth about "X" here? That's a huge topic that I'm hoping to tackle ASAP. The basic premise is that parallel 'Context' objects (well, structs) are allocated for each parallel thread callback. The context persists for the lifetime of the "parallel work". The "lifetime of the parallel work" depends on what you're doing. For a simple ``async.submit_work(foo)``, the context is considered complete once ``foo()`` has been called (presuming no exceptions were raised). For an async client/server, the context will persist for the entirety of the connection. The context is responsible for encapsulating all resources related to the parallel thread. So, it has its own heap, and all memory allocations are taken from that heap. For any given parallel thread, only one context can be executing at a time, and this can be accessed via the ``__declspec(thread) Context *ctx`` global (which is primed by some glue code as soon as the parallel thread starts executing a callback). No reference counting or garbage collection is done during parallel thread execution. Instead, once the context is finished, it is scheduled to be released, which means it'll be "processed" by the main thread as part of its housekeeping work (during ``async.run()`` (technically, ``async.run_once()``). The main thread simply destroys the entire heap in one fell swoop, releasing all memory that was associated with that context. There are a few side effects to this. First, the heap allocator (basically, the thing that answers ``malloc()`` calls) is incredibly simple. It allocates LARGE_PAGE_SIZE chunks of memory at a time (2MB on x64), and simply returns pointers to that chunk for each memory request (adjusting h->next and allocation stats as it goes along, obviously). Once the 2MB has been exhausted, another 2MB is allocated. That approach is fine for the ``submit_(work|timer|wait)`` callbacks, which basically provide a way to run a presumably-finite-length function in a parallel thread (and invoking callbacks/errbacks as required). However, it breaks down when dealing with client/server stuff. Each invocation of a callback (say, ``data_received(...)``) may only consume, say, 500 bytes, but it might be called a million times before the connection is terminated. You can't have cumulative memory usage with possibly-infinite-length client/server-callbacks like you can with the once-off ``submit_(work|wait|timer)`` stuff. So, enter heap snapshots. The logic that handles all client/server connections is instrumented such that it takes a snapshot of the heap (and all associated stats) prior to invoking a Python method (via ``PyObject_Call()``, for example, i.e. the invocation of ``data_received``). When the method completes, we can simply roll back the snapshot. The heap's stats and next pointers et al all get reset back to what they were before the callback was invoked. That's how the chargen server is able to pump out endless streams of data for every client whilst keeping memory usage static. (Well, every new client currently consumes at least a minimum of 2MB (but down the track that can be tweaked back down to SMALL_PAGE_SIZE, 4096, for servers that need to handle hundreds of thousands of clients simultaneously). The only issue with this approach is detecting when the callback has done the unthinkable (from a shared-nothing perspective) and persisted some random object it created outside of the parallel context it was created in. That's actually a huge separate technical issue to tackle -- and it applies just as much to the normal ``submit_(wait|work|timer)`` callbacks as well. I've got a somewhat-temporary solution in place for that currently: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.
Re: [Python-Dev] Slides from today's parallel/async Python talk
Cross-referenced to relevant bits of code where appropriate.
(And just a quick reminder regarding the code quality disclaimer:
I've been hacking away on this stuff relentlessly for a few months;
the aim has been to make continual forward progress without getting
bogged down in non-value-add busy work. Lots of wildly inconsistent
naming conventions and dead code that'll be cleaned up down the
track. And the relevance of any given struct will tend to be
proportional to how many unused members it has (homeless hoarder +
shopping cart analogy).)
On Thu, Mar 14, 2013 at 11:45:20AM -0700, Trent Nelson wrote:
> The basic premise is that parallel 'Context' objects (well, structs)
> are allocated for each parallel thread callback.
The 'Context' struct:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel_private.h#l546
Allocated via new_context():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4211
also relevant, new_context_for_socket() (encapsulates a
client/server instance within a context).
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4300
Primary role of the context is to isolate the memory management.
This is achieved via 'Heap':
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel_private.h#l281
(Which I sort of half started refactoring to use the _HEAD_EXTRA
approach when I thought I'd need to have a separate heap type for
some TLS avenue I explored -- turns out that wasn't necessary).
> The context persists for the lifetime of the "parallel work".
>
> The "lifetime of the parallel work" depends on what you're doing. For
> a simple ``async.submit_work(foo)``, the context is considered
> complete once ``foo()`` has been called (presuming no exceptions were
> raised).
Managing context lifetime is one of the main responsibilities of
async.run_once():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3841
> For an async client/server, the context will persist for the entirety
> of the connection.
Marking a socket context as 'finished' for servers is the job of
PxServerSocket_ClientClosed():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l6885
> The context is responsible for encapsulating all resources related to
> the parallel thread. So, it has its own heap, and all memory
> allocations are taken from that heap.
The heap is initialized in two steps during new_context(). First,
a handle is allocated for the underlying system heap (via
HeapCreate):
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4224
The first "heap" is then initialized for use with our context via
the Heap_Init(Context *c, size_t n, int page_size) call:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l1921
Heaps are actually linked together via a doubly-linked list. The
first heap is a value member (not a pointer) of Context; however,
the active heap is always accessed via the '*h' pointer which is
updated as necessary.
struct Heap {
Heap *prev;
Heap *next;
void *base;
void *next;
int allocated;
int remaining;
...
struct Context {
Heap heap;
Heap *h;
...
> For any given parallel thread, only one context can be executing at a
> time, and this can be accessed via the ``__declspec(thread) Context
> *ctx`` global (which is primed by some glue code as soon as the
> parallel thread starts executing a callback).
Glue entry point for all callbacks is _PyParallel_EnteredCallback:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3047
On the topic of callbacks, the main workhorse for the
submit_(wait|work) callbacks is _PyParallel_WorkCallback:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3120
The interesting logic starts at start:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3251
The interesting part is the error handling. If the callback raises
an exception, we check to see if an errback has been provided. If
so, we call the errback with the error details.
If the callback completes successfully (or it fails, but the errback
completes successfully), that is treated as successful callback or
errback completion, respectively:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3270
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3294
If the errback fails, or no errback was provided, the exception
percolates back to the m
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 02:30:14PM -0700, Trent Nelson wrote: > Then it dawned on me to just add the snapshot/rollback stuff to > normal Context objects. In retrospect, it's silly I didn't think of > this in the first place -- the biggest advantage of the Context > abstraction is that it's thread-local, but not bindingly so (as in, > it'll only ever run on one thread at a time, but it doesn't matter > which one, which is essential, because the ). > > Once I switched ... $10 if you can guess when I took a break for lunch. "but it doesn't matter which one, which is essential, because there are no guarantees with regards to which thread runs which context." Is along the lines of what I was going to say. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 12:59:57PM -0700, Stefan Ring wrote: > > Yup, in fact, if I hadn't come up with the __read[gf]sword() trick, > > my only other option would have been TLS (or the GetCurrentThreadId > > /pthread_self() approach in the presentation). TLS is fantastic, > > and it's definitely an intrinsic part of the solution (the "Y" part > > of "if we're a parallel thread, do Y"), but it definitely more > > costly than a simple FS/GS register read. > > I think you should be able to just take the address of a static > __thread variable to achieve the same thing in a more portable way. Sure, but, uh, that's kinda' trivial in comparison to all the wildly unportable Windows-only functionality I'm using to achieve all of this at the moment :-) For the record, here are all the Windows calls I'm using that have no *direct* POSIX equivalent: Interlocked singly-linked lists: - InitializeSListHead() - InterlockedFlushSList() - QueryDepthSList() - InterlockedPushEntrySList() - InterlockedPushListSList() - InterlockedPopEntrySlist() Synchronisation and concurrency primitives: - Critical sections - InitializeCriticalSectionAndSpinCount() - EnterCriticalSection() - LeaveCriticalSection() - TryEnterCriticalSection() - Slim read/writer locks (some pthread implements have rwlocks)*: - InitializeSRWLock() - AcquireSRWLockShared() - AcquireSRWLockExclusive() - ReleaseSRWLockShared() - ReleaseSRWLockExclusive() - TryAcquireSRWLockExclusive() - TryAcquireSRWLockShared() - One-time initialization: - InitOnceBeginInitialize() - InitOnceComplete() - Generic event, signalling and wait facilities: - CreateEvent() - SetEvent() - WaitForSingleObject() - WaitForMultipleObjects() - SignalObjectAndWait() Native thread pool facilities: - TrySubmitThreadpoolCallback() - StartThreadpoolIo() - CloseThreadpoolIo() - CancelThreadpoolIo() - DisassociateCurrentThreadFromCallback() - CallbackMayRunLong() - CreateThreadpoolWait() - SetThreadpoolWait() Memory management: - HeapCreate() - HeapAlloc() - HeapDestroy() Structured Exception Handling (#ifdef Py_DEBUG): - __try/__except Sockets: - ConnectEx() - AcceptEx() - WSAEventSelect(FD_ACCEPT) - DisconnectEx(TF_REUSE_SOCKET) - Overlapped WSASend() - Overlapped WSARecv() Don't get me wrong, I grew up with UNIX and love it as much as the next guy, but you can't deny the usefulness of Windows' facilities for writing high-performance, multi-threaded IO code. It's decades ahead of POSIX. (Which is also why it bugs me when I see select() being used on Windows, or IOCP being used as if it were a poll-type "generic IO multiplexor" -- that's like having a Ferrari and speed limiting it to 5mph!) So, before any of this has a chance of working on Linux/BSD, a lot more scaffolding will need to be written to provide the things we get for free on Windows (threadpools being the biggest freebie). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 03:50:27PM -0700, "Martin v. Löwis" wrote: > Am 14.03.13 12:59, schrieb Stefan Ring: > > I think you should be able to just take the address of a static > > __thread variable to achieve the same thing in a more portable way. > > That assumes that the compiler supports __thread variables, which > isn't that portable in the first place. FWIW, I make extensive use of __declspec(thread). I'm aware of GCC and Clang's __thread alternative. No idea what IBM xlC, Sun Studio and others offer, if anything. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 03:56:33PM -0700, "Martin v. Löwis" wrote:
> Am 14.03.13 11:23, schrieb Trent Nelson:
> > Porting the Py_PXCTX part is trivial compared to the work that is
> > going to be required to get this stuff working on POSIX where none
> > of the sublime Windows concurrency, synchronisation and async IO
> > primitives exist.
>
> I couldn't understand from your presentation why this is essential
> to your approach. IIUC, you are "just" relying on the OS providing
> a thread pool, (and the sublime concurrency and synchronization
> routines are nothing more than that, ISTM).
Right, there's nothing Windows* does that can't be achieved on
Linux/BSD, it'll just take more scaffolding (i.e. we'll need to
manage our own thread pool at the very least).
[*]: actually, the interlocked singly-linked list stuff concerns me;
the API seems straightforward enough but the implementation becomes
deceptively complex once you factor in the ABA problem. (I'm not
aware of a portable open source alternative for that stuff.)
> Implementing a thread pool on top of select/poll/kqueue seems
> straight-forward.
Nod, that's exactly what I've got in mind. Spin up a bunch of
threads that sit there and call poll/kqueue in an endless loop.
That'll work just fine for Linux/BSD/OSX.
Actually, what's really interesting is the new registered IO
facilities in Windows 8/2012. The Microsoft recommendation for
achieving the ultimate performance (least amount of jitter, lowest
latency, highest throughput) is to do something like this:
while (1) {
if (!DequeueCompletionRequests(...)) {
YieldProcessor();
continue;
} else {
/* Handle requests */
}
}
That pattern looks a lot more like what you'd do on Linux/BSD (spin
up a thread per CPU and call epoll/kqueue endlessly) than any of the
previous Windows IO patterns.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Mon, Mar 18, 2013 at 05:27:33PM -0700, Christian Tismer wrote:
> Hi Trent,
Hi Christian! Thanks for taking the time to read my walls of text
;-)
> > So, the remaining challenge is preventing the use case alluded to
> > earlier where someone tries to modify an object that hasn't been "async
> > protected". That's a bit harder. The idea I've got in mind is to
> > instrument the main CPython ceval loop, such that we do these checks as
> > part of opcode processing. That allows us to keep all the logic in the
> > one spot and not have to go hacking the internals of every single
> > object's C backend to ensure correctness.
> >
> > Now, that'll probably work to an extent. I mean, after all, there are
> > opcodes for all the things we'd be interested in instrumenting,
> > LOAD_GLOBAL, STORE_GLOBAL, SETITEM etc. What becomes challenging is
> > detecting arbitrary mutations via object calls, i.e. how do we know,
> > during the ceval loop, that foo.append(x) needs to be treated specially
> > if foo is a main-thread object and x is a parallel thread object?
> >
> > There may be no way to handle that *other* than hacking the internals of
> > each object, unfortunately. So, the viability of this whole approach
> > may rest on whether or that's deemed as an acceptable tradeoff (a
> > necessary evil, even) to the Python developer community.
>
> This is pretty much my concern:
> In order to make this waterproof, as required for CPython, you will quite
> likely have to do something on very many objects, and this is hard
> to chime into CPython.
Actually, I think I was unnecessarily pessimistic here. When I sent
that follow-up mail with cross-references, I realized I'd forgotten
the nitty gritty details of how I implemented the async protection
support.
It turns out I'd already started on protecting lists (or rather,
PySequenceMethods), but decided to stop as the work I'd done on the
PyMappingMethods was sufficient for my needs at the time.
All I *really* want to do is raise an exception if a parallel object
gets assigned to a main-thread container object (list/dict etc) that
hasn't been "async protected". (As opposed to now, where it'll
either segfault or silently corrupt stuff, then segfault later.)
I've already got all the infrastructure in place to test that (I use
it extensively within pyparallel.c):
Py_ISPY(obj) - detect a main-thread object
Py_ISPX(obj) - detect a parallel-thread object
Py_IS_PROTECTED(obj) - detect if a main-thread object has
been protected*
[*]: actually, this isn't in a macro form right now, it's a
cheeky inline:
__inline
char
_protected(PyObject *obj)
{
return (obj->px_flags & Py_PXFLAGS_RWLOCK);
}
As those macros are exposed in the public ,
they can be used in other parts of the code base. So, it's just a
matter of finding the points where an `lvalue = rvalue` takes place;
where: ``Py_ISPY(lvalue) && Py_ISPX(rvalue)``. Then a test to see
if lvalue is protected; if not, raise an exception. If so, then
nothing else needs to be done.
And there aren't that many places where this happens. (It didn't
take long to get the PyMappingMethods intercepts nailed down.)
That's the idea anyway. I need to get back to coding to see how it
all plays out in practice. "And there aren't many places where this
happens" might be my famous last words.
> > If it's not, then it's unlikely this approach will ever see the light of
> > day in CPython. If that turns out to be the case, then I see this
> > project taking the path that Stackless took (forking off and becoming a
> > separate interpreter).
>
> We had that discussion quite often for Stackless, and I would love to find
> a solution that allows to add special versions and use cases to CPython
> in a way that avoids the forking as we did it.
>
> It would be a nice thing if we could come up with a way to keep CPython
> in place, but to swap the interpreter out and replace it with a specialized
> version, if the application needs it. I wonder to what extent that would be
> possible.
> What I would like to achieve, after having given up on Stackless integration
> is a way to let it piggyback onto CPython that works like an extension
> module, although it hat effectively replace larger parts of the interpreter.
> I wonder if that might be the superior way to have more flexibility,
> without forcing
> everything and all go into CPython.
> If we can make the interpreter somehow pluggable at runtime, a lot of issues
> would become much simpler.
>
> >
> > There's nothing wrong with that; I am really excited about the
> > possibilities afforded by this approach, and I'm sure it will pique the
> > interest of commercial entities out there that h
Re: [Python-Dev] Slides from today's parallel/async Python talk
That's good to hear :-) (It's a fantastic facility, I couldn't imagine having to go back to manual TLS API stuff after using __thread/__declspec(thread).) This e-mail was sent from a wireless device. On 21 Mar 2013, at 09:30, "Baptiste Lepilleur" mailto:baptiste.lepill...@gmail.com>> wrote: 2013/3/15 Trent Nelson mailto:tr...@snakebite.org>> On Thu, Mar 14, 2013 at 03:50:27PM -0700, "Martin v. Löwis" wrote: > Am 14.03.13 12:59, schrieb Stefan Ring: > > I think you should be able to just take the address of a static > > __thread variable to achieve the same thing in a more portable way. > > That assumes that the compiler supports __thread variables, which > isn't that portable in the first place. FWIW, I make extensive use of __declspec(thread). I'm aware of GCC and Clang's __thread alternative. No idea what IBM xlC, Sun Studio and others offer, if anything. IBM xlC and Sun Studio also support this feature. From memory, it's also __thread keyword. This features is also supported by the new C11/C++11 standards. Baptiste. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
No, I haven't. I'd lose the excellent Windows pairing of thread pool IO and overlapped IO facilities if I did that. Not saying it isn't an option down the track for the generic "submit work" API though; that stuff will work against any thread pool without too much effort. But for now, the fact that all I need to call is TrySubmitThreadpoolCallback and Windows does *everything* else is pretty handy. Lets me concentrate on the problem instead of getting distracted by scaffolding. This e-mail was sent from a wireless device. On 21 Mar 2013, at 05:53, "Sturla Molden" wrote: > Den 14. mars 2013 kl. 23:23 skrev Trent Nelson : > >> >> For the record, here are all the Windows calls I'm using that have >> no *direct* POSIX equivalent: >> >> Interlocked singly-linked lists: >> - InitializeSListHead() >> - InterlockedFlushSList() >> - QueryDepthSList() >> - InterlockedPushEntrySList() >> - InterlockedPushListSList() >> - InterlockedPopEntrySlist() >> >> Synchronisation and concurrency primitives: >> - Critical sections >> - InitializeCriticalSectionAndSpinCount() >> - EnterCriticalSection() >> - LeaveCriticalSection() >> - TryEnterCriticalSection() >> - Slim read/writer locks (some pthread implements have >> rwlocks)*: >> - InitializeSRWLock() >> - AcquireSRWLockShared() >> - AcquireSRWLockExclusive() >> - ReleaseSRWLockShared() >> - ReleaseSRWLockExclusive() >> - TryAcquireSRWLockExclusive() >> - TryAcquireSRWLockShared() >> - One-time initialization: >> - InitOnceBeginInitialize() >> - InitOnceComplete() >> - Generic event, signalling and wait facilities: >> - CreateEvent() >> - SetEvent() >> - WaitForSingleObject() >> - WaitForMultipleObjects() >> - SignalObjectAndWait() >> >> Native thread pool facilities: >> - TrySubmitThreadpoolCallback() >> - StartThreadpoolIo() >> - CloseThreadpoolIo() >> - CancelThreadpoolIo() >> - DisassociateCurrentThreadFromCallback() >> - CallbackMayRunLong() >> - CreateThreadpoolWait() >> - SetThreadpoolWait() >> >> Memory management: >> - HeapCreate() >> - HeapAlloc() >> - HeapDestroy() >> >> Structured Exception Handling (#ifdef Py_DEBUG): >> - __try/__except >> >> Sockets: >> - ConnectEx() >> - AcceptEx() >> - WSAEventSelect(FD_ACCEPT) >> - DisconnectEx(TF_REUSE_SOCKET) >> - Overlapped WSASend() >> - Overlapped WSARecv() >> >> >> Don't get me wrong, I grew up with UNIX and love it as much as the >> next guy, but you can't deny the usefulness of Windows' facilities >> for writing high-performance, multi-threaded IO code. It's decades >> ahead of POSIX. (Which is also why it bugs me when I see select() >> being used on Windows, or IOCP being used as if it were a poll-type >> "generic IO multiplexor" -- that's like having a Ferrari and speed >> limiting it to 5mph!) >> >> So, before any of this has a chance of working on Linux/BSD, a lot >> more scaffolding will need to be written to provide the things we >> get for free on Windows (threadpools being the biggest freebie). >> >> >> > > > Have you considered using OpenMP instead of Windows API or POSIX threads > directly? OpenMP gives you a thread pool and synchronization primitives for > free as well, with no special code needed for Windows or POSIX. > > OpenBLAS (and GotoBLAS2) uses OpenMP to produce a thread pool on POSIX > systems (and actually Windows API on Windows). The OpenMP portion of the C > code is wrapped so it looks like sending an asynch task to a thread pool; the > C code is not littered with OpenMP pragmas. If you need something like > Windows threadpools on POSIX, just look at the BSD licensed OpenBLAS code. It > is written to be scalable for the world's largest supercomputers (but also > beautifully written and very easy to read). > > Cython has code to register OpenMP threads as Python threads, in case that is > needed. So that problem is also solved. > > > Sturla > > > > > > > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
http://c2.com/cgi/wiki?BlubParadox ;-) Sent from my iPhone On 21 Mar 2013, at 06:18, "Antoine Pitrou" wrote: > Le Thu, 14 Mar 2013 15:23:37 -0700, > Trent Nelson a écrit : >> >>Don't get me wrong, I grew up with UNIX and love it as much as the >>next guy, but you can't deny the usefulness of Windows' facilities >>for writing high-performance, multi-threaded IO code. It's >> decades ahead of POSIX. > > I suppose that's why all high-performance servers run under Windows. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/trent%40snakebite.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Post-PyCon updates to PyParallel
[ python-dev: I've set up a new list for pyparallel discussions: https://lists.snakebite.net/mailman/listinfo/pyparallel. This e-mail will be the last I'll send to python-dev@ regarding the on-going pyparallel work; please drop python-dev@ from the CC and just send to pyparal...@lists.snakebite.net -- I'll stay on top of the posts-from-unsubscribed-users moderation for those that want to reply to this e-mail but not subscribe. ] Hi folks, Wanted to give a quick update on the parallel work both during and after PyCon. During the language summit when I presented the slides I uploaded to speakerdeck.com, the majority of questions from other developers revolved around the big issues like data integrity and what happens when parallel objects interact with main-thread objects and vice-versa. So, during the sprints, I explored putting guards in place to throw an exception if we detect that a user has assigned a parallel object to a non-protected main-thread object. (I describe the concept of 'protection' in my follow up posts to python-dev last week: http://mail.python.org/pipermail/python-dev/2013-March/124690.html. Basically, protecting a main-thread object allows code like this to work without crashing: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.rdtsc() async.submit_work(foo) async.submit_work(bar) ) It was actually pretty easy, far easier than I expected. It was achieved via Px_CHECK_PROTECTION(): https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LInclude/object.hT1072 Various new tests related to the protection functionality: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LLib/async/test/test_primitives.pyT58 The type of changes I had to make to other parts of CPython to perform the protection checks: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LObjects/abstract.cT170 That was all working fine... until I started looking at adding support for lists (i.e. appending a parallel thread object to a protected, main-thread list). The problem is that appending to a list will often involve a list resize, which is done via PyMem_REALLOC() and some custom fiddling. That would mean if a parallel thread attempts to append to a list and it needs resizing, all the newly realloc'd memory would be allocated from the parallel context's heap. Now, this heap would stick around as long as the parallel objects have a refcount > 0. However, as soon as the last parallel object's refcount hits 0, the entire context will be scheduled for the cleanup/release/free dance, which will eventually blow away the entire heap and all the memory allocated against that heap... which means all the **ob_item stuff that was reallocated as part of the list resize. Not particularly desirable :-) As I was playing around with ways to potentially pre-allocate lists, it occurred to me that dicts would be affected in the exact same way; I just hadn't run into it yet because my unit tests only ever assigned a few (<5) objects to the protected dicts. Once the threshold gets reached (10?), a "dict resize" would take place, which would involve lots of PyMem_REALLOCs, and we get into the exact same situation mentioned above. So, at that point, I concluded that whole async protection stuff was not a viable long term solution. (In fact, the reason I first added it was simply to have an easy way to test things in unit tests.) The new solution I came up with: new thread-safe, interlocked data types that are *specifically* designed for this exact use case; transferring results from computation in a parallel thread back to a main thread 'container' object. First up is a new list type: xlist() (PyXListObject/PyXList_Type). I've just committed the work-in-progress stuff I've been able to hack out whilst traveling the past few days: https://bitbucket.org/tpn/pyparallel/commits/5b662eba4efe83e94d31bd9db4520a779aea612a It's not finished, and I'm pretty sure it doesn't even compile yet, but the idea is something like this:
Re: [Python-Dev] Post-PyCon updates to PyParallel
On Wed, Mar 27, 2013 at 11:26:51PM -0700, Trent Nelson wrote: > [ python-dev: I've set up a new list for pyparallel discussions: > https://lists.snakebite.net/mailman/listinfo/pyparallel. This > e-mail will be the last I'll send to python-dev@ regarding the > on-going pyparallel work; please drop python-dev@ from the CC > and just send to pyparal...@lists.snakebite.net -- I'll stay on > top of the posts-from-unsubscribed-users moderation for those that > want to reply to this e-mail but not subscribe. ] Gah, wrong e-mail address, it's pyparal...@snakebite.net. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
Hi Charles-François,
On Thu, Apr 04, 2013 at 01:18:58AM -0700, Charles-François Natali wrote:
> Just a quick implementation question (didn't have time to read through
> all your emails :-)
>
> async.submit_work(func, args, kwds, callback=None, errback=None)
>
> How do you implement arguments passing and return value?
>
> e.g. let's say I pass a list as argument: how do you iterate on the
> list from the worker thread without modifying the backing objects for
> refcounts (IIUC you use a per-thread heap and don't do any
> refcounting).
Correct, nothing special is done for the arguments (apart from
incref'ing them in the main thread before kicking off the parallel
thread (then decref'ing them in the main thread once we're sure the
parallel thread has finished)).
> Same thing for return value, how do you pass it to the
> callback?
For submit_work(), you can't :-) In fact, an exception is raised if
the func() or callback() or errback() attempts to return a non-None
value.
It's worth noting that I eventually plan to have the map/reduce-type
functionality (similar to what multiprocessing offers) available via
a separate 'parallel' façade. This will be geared towards programs
that are predominantly single-threaded, but have lots of data that
can be processed in parallel at various points.
Now, with that being said, there are a few options available at the
moment if you want to communicate stuff from parallel threads back
to the main thread. Originally, you could do something like this:
d = async.dict()
def foo():
d['foo'] = async.rdtsc()
def bar():
d['bar'] = async.rdtsc()
async.submit_work(foo)
async.submit_work(bar)
But I recently identified a few memory-management flaws with that
approach (I'm still on the fence with this issue... initially I was
going to drop all support, but I've since had ideas to address the
memory issues, so, we'll see).
There's also this option:
d = dict()
@async.call_from_main_thread_and_wait
def store(k, v):
d[str(k)] = str(v)
def foo():
store('foo', async.rdtsc())
def bar():
store('bar', async.rdtsc())
async.submit_work(foo)
async.submit_work(bar)
(Not a particularly performant option though; the main-thread
instantly becomes the bottleneck.)
Post-PyCon, I've been working on providing new interlocked data
types that are specifically designed to bridge the parallel/main-
thread divide:
xl = async.xlist()
def foo():
xl.push(async.rdtsc())
def bar():
xl.push(async.rdtsc())
async.submit_work(foo)
async.submit_work(bar)
while True:
x = xl.pop()
if not x:
break
process(x)
What's interesting about xlist() is that it takes ownership of the
parallel objects being pushed onto it. That is, it basically clones
them, using memory allocated from its own internal heap (allowing
the parallel-thread's context heap to be freed, which is desirable).
The push/pop operations are interlocked at the C level, which
obviates the need for any explicit locking.
I've put that work on hold for now though; I want to finish the
async client/server stuff (it's about 60-70% done) first. Once
that's done, I'll tackle the parallel.*-type façade.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Apr 04, 2013 at 11:53:01PM -0700, Charles-Fran?ois Natali wrote: > Hello, > > >> async.submit_work(func, args, kwds, callback=None, errback=None) > >> > >> How do you implement arguments passing and return value? > >> > >> e.g. let's say I pass a list as argument: how do you iterate on the > >> list from the worker thread without modifying the backing objects for > >> refcounts (IIUC you use a per-thread heap and don't do any > >> refcounting). > > > > Correct, nothing special is done for the arguments (apart from > > incref'ing them in the main thread before kicking off the parallel > > thread (then decref'ing them in the main thread once we're sure the > > parallel thread has finished)). > > IIUC you incref the argument from the main thread before publishing it > to the worker thread: but what about containers like list? How do you > make sure the refcounts of the elements don't get deallocated while > the worker thread iterates? Ah, so, all of my examples were missing async.run(). They should have looked like this: async.submit_work(foo) async.submit_work(bar) async.run() async.run() is called from the main thread, with the GIL held, and it blocks until all parallel threads (well, parallel contexts, to be exact) have completed. The parallel 'work' doesn't actually start until async.run() is called either. (That's completely untrue at the moment; async.submit_work(foo) will execute foo() in a parallel thread immediately. Fixing that is on the todo list.) With only parallel threads running, no main-thread objects could ever be deallocated*, as no decref'ing is ever done. [*]: unless you went out of your way to delete/deallocate main thread objects via the @async.call_from_main_thread facility. At the moment, that's firmly in the category of "Don't Do That". (And, thinking about it a little more, I guess I could augment the ceval loop in such a way that in order for the main thread to run things scheduled via @async.call_from_main_thread, all parallel threads need to be suspended. Or I could just freeze/thaw them (although I don't know if there are POSIX counterparts to those Windows methods). That would definitely impede performance, but it would assure data integrity. Perhaps it should be enabled by default, with the option to disable it for consenting adults.) > More generally, how do you deal with non-local objects? Read-only ops against non-local (main-thread) objects from parallel threads are free, which is nice. Things get tricky when you try to mutate main-thread objects from parallel threads. That's where all the context persistence, interlocked data types, object protection etc stuff comes in. Is... that what you mean by how do I deal with non-local objects? I took a guess ;-) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pyparallel and new memory API discussions...
The new memory API discussions (and PEP) warrant a quick pyparallel
update: a couple of weeks after PyCon, I came up with a solution for
the biggest show-stopper that has been plaguing pyparallel since its
inception: being able to detect the modification of "main thread"
Python objects from within a parallel context.
For example, `data.append(4)` in the example below will generate an
AssignmentError exception, because data is a main thread object, and
`data.append(4)` gets executed from within a parallel context::
data = [ 1, 2, 3 ]
def work():
data.append(4)
async.submit_work(work)
The solution turned out to be deceptively simple:
1. Prior to running parallel threads, lock all "main thread"
memory pages as read-only (via VirtualProtect on Windows,
mprotect on POSIX).
2. Detect attempts to write to main thread pages during parallel
thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
and raise an exception instead (detection is done in the ceval
frame exec loop).
3. Prior to returning control back to the main thread (which will
be paused whilst all the parallel threads are running), unlock
all the "main thread" pages.
4. Pause all parallel threads while the main thread runs.
5. Go back to 1.
I got a proof-of-concept working on Windows a while back (and also
played around with large page support in the same commit). The main
changes were to obmalloc.c:
https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe808c1c11779f#chg-Objects/obmalloc.c
The key was the introduction of two new API calls, intended to be
called by the pyparallel.c infrastructure:
_PyMem_LockMainThreadPages()
_PyMem_UnlockMainThreadPages()
The implementation is pretty simple:
+int
+_PyMem_LockMainThreadPages(void)
+{
+DWORD old = 0;
+
+if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, &old)) {
+PyErr_SetFromWindowsErr(0);
+return -1;
+}
Note the `base_addr` and `nbytes_committed` argument. Basically, I
re-organized obmalloc.c a little bit such that we never actually
call malloc() directly. Instead, we exploit the ability to reserve
huge virtual address ranges without actually committing the memory,
giving us a fixed `base_addr` void pointer that we can pass to calls
like VirtualProtect or mprotect.
We then incrementally commit more pages as demand increases, and
simply adjust our `nbytes_committed` counter as we go along. The
net effect is that we can call VirtualProtect/mprotect once, with a
single base void pointer and size_t range, and immediately affect the
protection of all memory pages that fall within that range.
As an added bonus, we also get a very cheap and elegant way to test
if a pointer (or any arbitrary memory address, actually) belongs to
the main thread's memory range (at least in comparison to the
existing _PyMem_InRange black magic). (This is very useful for my
pyparallel infrastructure, which makes extensive use of conditional
logic based on address tests.)
(Side-bar: a side-effect of the approach I've used in the proof-
of-concept (by only having a single base addr pointer) is that
we effectively limit the maximum memory we could eventually
commit.
I actually quite like this -- in fact, I'd like to tweak it
such that we can actually expose min/max memory values to the
Python interpreter at startup (analogous to the JVM).
Having known upper bounds on maximum memory usage will vastly
simplify some other areas of my pyparallel work (like the async
socket stuff).
For example, consider network programs these days that take a
"max clients" configuration parameter. That seems a bit
backwards to me.
It would be better if we simply said, "here, Python, you have
1GB to work with". That allows us to calculate how many
clients we could simultaneously serve based on socket memory
requirements, which allows for much more graceful behavior
under load than leaving it open-ended.
Maximum memory constraints would also be useful for the
parallel.map(callable, iterable) stuff I've got in the works,
as it'll allow us to optimally chunk work and assign to threads
based on available memory.)
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
Regards,
Trent.
Re: [Python-Dev] pyparallel and new memory API discussions...
Hi Charles-François! Good to hear from you again. It was actually your e-mail a few months ago that acted as the initial catalyst for this memory protection idea, so, thanks for that :-) Answer below. On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-François Natali wrote: > 2013/6/19 Trent Nelson : > > > > The new memory API discussions (and PEP) warrant a quick pyparallel > > update: a couple of weeks after PyCon, I came up with a solution for > > the biggest show-stopper that has been plaguing pyparallel since its > > inception: being able to detect the modification of "main thread" > > Python objects from within a parallel context. > > > > For example, `data.append(4)` in the example below will generate an > > AssignmentError exception, because data is a main thread object, and > > `data.append(4)` gets executed from within a parallel context:: > > > > data = [ 1, 2, 3 ] > > > > def work(): > > data.append(4) > > > > async.submit_work(work) > > > > The solution turned out to be deceptively simple: > > > > 1. Prior to running parallel threads, lock all "main thread" > > memory pages as read-only (via VirtualProtect on Windows, > > mprotect on POSIX). > > > > 2. Detect attempts to write to main thread pages during parallel > > thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), > > and raise an exception instead (detection is done in the ceval > > frame exec loop). > > Quick stupid question: because of refcounts, the pages will be written > to even in case of read-only access. How do you deal with this? Easy: I don't refcount in parallel contexts :-) There's no need, for two reasons: 1. All memory allocated in a parallel context is localized to a private heap. When the parallel context is finished, the entire heap can be blown away in one fell-swoop. There's no need for reference counting or GC because none of the objects will exist after the parallel context completes. 2. The main thread won't be running when parallel threads/contexts are executing, which means main thread objects being accessed in parallel contexts (read-only access is fine) won't be suddenly free()'d or GC-collected or whatever. You get credit for that second point; you asked a similar question a few months ago that made me realize I absolutely couldn't have the main thread running at the same time the parallel threads were running. Once I accepted that as a design constraint, everything else came together nicely... "Hmmm, if the main thread isn't running, it won't need write-access to any of its pages! If we mark them read-only, we could catch the traps/SEHs from parallel threads, then raise an exception, ahh, simple!". I'm both chuffed at how simple it is (considering it was *the* major show-stopper), and miffed at how it managed to elude me for so long ;-) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: > > 1. All memory allocated in a parallel context is localized to a > > private heap. > > How do you allocate memory in this "private" heap? Did you add new > functions to allocate memory? Yup: _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. All memory operations (PyObject_New/Malloc etc) get intercepted during parallel thread execution and redirected to _PyHeap_Malloc(), which is a very simple slab allocator. (No need for convoluted buckets because we never free individual objects during parallel execution; instead, we just blow everything away at the end.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
> """
> So, Victor, I'm interested to hear how the new API you're proposing
> will affect this solution I've come up with for pyparallel; I'm
> going to be absolutely dependent upon the ability to lock main
> thread pages as read-only in one fell-swoop -- am I still going to
> be able to do that with your new API in place?
> """
>
> 2013/6/19 Trent Nelson :
> > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> >> > 1. All memory allocated in a parallel context is localized to a
> >> > private heap.
> >>
> >> How do you allocate memory in this "private" heap? Did you add new
> >> functions to allocate memory?
> >
> > Yup:
> > _PyHeap_Malloc():
> > http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
> >
> > All memory operations (PyObject_New/Malloc etc) get intercepted
> > during parallel thread execution and redirected to _PyHeap_Malloc(),
> > which is a very simple slab allocator. (No need for convoluted
> > buckets because we never free individual objects during parallel
> > execution; instead, we just blow everything away at the end.)
>
> Ok, so I don't think that the PEP 445 would change anything for you.
>
> The following change might have an impact: If _PyHeap_Malloc is not
> thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
> GIL is not held would avoid bugs in your code.
Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by
design, but I'm not sure if we're referring to the same sort of
thread-safe problem here.
For one, _PyHeap_Malloc won't ever run if the GIL isn't being held.
(Parallel threads are only allowed to run when the main thread has
the GIL held and has relinquished control to parallel threads.)
Also, I interpret PyMem_RawMalloc() as a direct shortcut to
malloc() (or something else that returns void *s that are then
free()'d down the track). Is that right?
I don't think that would impact pyparallel.
> If you want to choose dynamically the allocator at runtime, you can
> replace PyObject_Malloc allocator using:
> -- 8< -
> static void *
> _PxMem_AllocMalloc(void *ctx, size_t size)
> {
> PyMemBlockAllocator *ctx;
> if (Py_PXCTX)
> return _PxMem_Malloc(size))
> else
> return alloc->malloc(alloc->ctx, size);
> }
>
> ...
>
> PyMemBlockAllocator pyparallel_pyobject;
>
> static void *
> setup_pyparallel_allocator(void)
> {
> PyMemBlockAllocator alloc;
> PyObject_GetAllocator(&pyparallel_pyobject);
> alloc.ctx = &pyparallel_pyobject;
> alloc.malloc = _PxMem_AllocMalloc;
> ...
> PyObject_SetAllocator(&alloc);
> }
> -- 8< -
>
> But I don't know if you want pyparallel to be an "optional" feature
> chosen at runtime...
Hmmm, those code snippets are interesting. Time for some more
homework.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou wrote: > > > On Wed, 15 Dec 2021 14:13:03 +0100 > > Antoine Pitrou wrote: > > > > > Did you try to take into account the envisioned project for adding a > > > "complete" GC and removing the GIL? > > > > Sorry, I was misremembering the details. Sam Gross' proposal > > (posted here on 07/10/2021) doesn't switch to a "complete GC", but it > > changes reference counting to a more sophisticated scheme (which > > includes immortalization of objects): > > > > > > https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit > > > > A note about this: Sam's immortalization covers exactly the objects that > Eric is planning to move into the interpreter state struct: "such as > interned strings, small integers, statically allocated PyTypeObjects, and > the True, False, and None objects". (Well, he says "such as" but I think so > does Eric. :-) > > Sam's approach is to use the lower bit of the ob_refcnt field to indicate > immortal objects. This would not work given the stable ABI (which has > macros that directly increment and decrement the ob_refcnt field). In fact, > I think that Sam's work doesn't preserve the stable ABI at all. However, > setting a very high bit (the bit just below the sign bit) would probably > work. Say we're using 32 bits. We use the value 0x_6000_ as the initial > refcount for immortal objects. The stable ABI will sometimes increment > this, sometimes decrement it. But as long as the imbalance is less than > 0x_2000_, the refcount will remain in the inclusive range [ > 0x_4000_ , 0x_7FFF_ ] and we can test for immortality by testing a > single bit: > > if (o->ob_refcnt & 0x_4000_) > > I don't know how long that would take, but I suspect that a program that > just increments the refcount relentlessly would have to run for hours > before hitting this range. On a 64-bit machine the same approach would > require years to run before a refcount would exceed the maximum allowable > imbalance. (These estimates are from Mark Shannon.) I did some research on this a few years back. I was curious what sort of "max reference counts" were encountered in the wild, in long-running real life programs. For the same reason: I wanted to get some insight into how many unused bits could possibly be repurposed for future shenanigans (I had PyParallel* in the mind at the time). I added some logic to capture* the max reference counts of the None, True, and Zero objects (in a trace callback), then ran a really long simulation program of a client's (it ran for about 5-6 hours). The results were as follows: MaxNoneRefCount 9,364,132 MaxTrueRefCount 204,215 MaxZeroRefCount36,784 I thought that was pretty interesting. Potentially many, many upper bits for the taking. The code also had some logic that would int 3 as soon as a 32-bit refcnt overflowed, and that never hit either (obviously, based on the numbers above). I also failed to come up with real-life code that would result in a Python object having a reference count higher than None's refcnt, but that may have just been from lack of creativity. Just thought I'd share. Regards, Trent. [*] 1: https://github.com/pyparallel/pyparallel [*] 2: https://github.com/tpn/tracer/blob/master/PythonTracer/PythonTracer.h#L690 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U2WHO5LYMBF6A6AFM36HOQQCNVGLXG6M/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Wed, Jan 05, 2022 at 01:59:21PM -0800, Trent Nelson wrote: > > I did some research on this a few years back. I was curious what sort > of "max reference counts" were encountered in the wild, in long-running > real life programs. For the same reason: I wanted to get some insight > into how many unused bits could possibly be repurposed for future > shenanigans (I had PyParallel* in the mind at the time). > > I added some logic to capture* the max reference counts of the None, > True, and Zero objects (in a trace callback), then ran a really long > simulation program of a client's (it ran for about 5-6 hours). The > results were as follows: > > MaxNoneRefCount 9,364,132 > MaxTrueRefCount 204,215 > MaxZeroRefCount36,784 Just double-checked my results, there were a handful of runs with higher counts: MaxNoneRefCount 59,834,444 MaxTrueRefCount 1,072,467 MaxZeroRefCount 3,460,921 Regards, Trent. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UJPQVBF5I4PGZTBX3EOVLLCAOQVNHVGZ/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] [snakebite] snakebite for GSoC?
On Thu, Mar 19, 2009 at 10:32:03AM -0700, ajaksu wrote: > Does anyone have good ideas for assigning students to snakebite? Is it > too early? Perhaps a little too early, python-dev@ won't know anything about Snakebite yet as I haven't publicly announced it there ;-) Watch this space closer to PyCon. FWIW, though, we're planning for Snakebite to be *very* involved with GSoC/GHOP. > I think the client-side 'Snakebite daemon' and server-side stuff > described at http://tinyurl.com/beyond-buildbot would be great > projects. Indeed. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_ssize_t support for ctypes arrays and pointers
On Fri, Mar 20, 2009 at 08:00:46PM +0100, Thomas Heller wrote: > Since I do not have a machine with so much memory: Does one > of the buildbots allow to run tests for this feature, or > do I have to wait for the snakebite farm? Will you be at PyCon? The wait might not be as bad as you think ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
> What happened to the big-ass computer farm for Python which was > being put together by someone at (I think) Michigan State? That sounds a lot like Snakebite (www.snakebite.org), which is still... uhhh, a work in progress ;-) We've run into an issue recently that's thwarted progress, but that'll hopefully be resolved in the next couple of weeks. And then... full steam ahead! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
> > What happened to the big-ass computer farm for Python which was > > being put together by someone at (I think) Michigan State? > > That sounds a lot like Snakebite (www.snakebite.org), which is > still... uhhh, a work in progress ;-) Actually, for those that are interested, here's a copy of the presentation I gave at the Testing in Python session at PyCon a few months ago: http://www.snakebite.org/presentations/snakebite-pycon2010-tip.pptx (Office 2007-2010) http://www.snakebite.org/presentations/snakebite-pycon2010-tip.ppt (Office 97-2003) If anything, it'll shed some light on all the unforeseen issues we've been running into since the project's inception. The presentation is a little out of date -- I spent three months earlier this year on the network and it's definitely in the most respectable state it's been in yet. Coupla' photos for those that are interested: http://snakebite.org/images/IMG_4384.JPG http://snakebite.org/images/IMG_4392.JPG http://snakebite.org/images/IMG_4393.JPG http://snakebite.org/images/IMG_4394.JPG http://snakebite.org/images/IMG_4395.JPG http://snakebite.org/images/IMG_4396.JPG http://snakebite.org/images/IMG_4401.JPG http://snakebite.org/images/IMG_4402.JPG http://snakebite.org/images/IMG_4403.JPG http://snakebite.org/images/IMG_4405.JPG http://snakebite.org/images/IMG_4410.JPG http://snakebite.org/images/IMG_4418.JPG http://snakebite.org/images/IMG_4424.JPG http://snakebite.org/images/IMG_4425.JPG We've got three racks filled to the brim with all sorts of servers: - 4xItanium 2 @ 1.5GHz, 16GB RAM, HP-UX 11iv3 - 4xItanium 2 @ 1.5GHz, 30GB RAM, RHEL 5.3 - 2xUltraSPARC III 900MHz, 8GB, Solaris 10 (file/zfs/nfs server -- 16x146GB 2Gb FC) - 2xUltraSPARC III 1.2GHz, 4GB, Solaris 10 - 2xPA-RISC 875MHz, 8GB, HP-UX 11iv1 - 4 AIX boxes w/ 2x1.5GHz, 8GB, AIX 5.1, 5.2, 5.3 & 6.1 - 10 dedicated VMware x86/64 boxes, ranging from dual core 8GB to 8 core monsters with 64GB - 4x667MHz AlphaServer, 8GB, Tru64 - 4x600MHz SGI Octane 300, IRIX 6.22 - and lots of other stuff. Actually, the only platform we don't have is Mac OS X. Although I've got a contact at Apple that I'll start harassing again once I'm back in East Lansing. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python on Windows with CoApp
Howdy folks, Quick e-mail at 34,000ft (aren't wifi-enabled flights great?) to mention a new initiative that's been started by Microsoft called CoApp (Common Opensource Application Publishing Platform). The aim is simple: make open source software rock on Windows ;-) It's probably easiest to think of it as a Microsoft-endorsed-but-community-run open source distribution for Windows, akin to all the various package managers for Linux distributions and ports/packages for the *BSDs. There are specific user and developer experiences we'll be addressing -- like making it easy to install and use open source software, or use it within your own project (open source or not). CoApp will affect Python in one of two ways. Once there's a clear-cut specification for open source projects to follow, Python can either decide to follow it, or not. The same applies to all open source packages, actually. For those that follow it, great! If not, no problem -- the plan is to shallow-fork such projects via launchpad and the CoApp community will take responsibility for getting releases of open source projects into CoApp shape. It's in its infancy at the moment -- it took the chap (Garrett Serack) who's spearheading it at Microsoft about six months to get it all signed off by the lawyers and platform/server VPs. So, for those of you out there who are Windows-inclined, now's a perfect time to get involved to help shape the direction of CoApp going forward. The website/wiki is http://coapp.org/ and the launchpad project site is http://launchpad.net/coapp (which is where the mailing list is hosted). We're actually having a 'CoApp Development Summit' tomorrow and Friday in Seattle (that Microsoft's graciously sponsored). The event will be accessible via Live Meeting for those that are interested: http://coapp.org/Project_Planning/CoApp_Design_and_Development_Summit Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 updated
Does anyone know of a way to teach vim that C sources in a python checkout
should have 4-space indents without changing the defaults for other C files?
I use this in my vimrc:
""
" indentation: use detectindent plugin if possible
""
set autoindent
set smartindent
try
let g:detectindent_preferred_expandtab = 1
let g:detectindent_preferred_tabsize = 8
let g:detectindent_preferred_indent = 4
source $VIMRUNTIME/plugin/detectindent.vim
au BufNewFile,BufRead * .* DetectIndent
catch
set smarttab
set expandtab
set tabstop=8
set shiftwidth=4
set softtabstop=4
set textwidth=80
endtry
*** And this is plugin/detectindent.vim:
" Name: detectindent (global plugin)
" Version: 1.0
" Author:Ciaran McCreesh
" Updates: http://dev.gentoo.org/~ciaranm/vim/
" Purpose: Detect file indent settings
"
" License: You may redistribute this plugin under the same terms
as Vim
"itself.
"
" Usage: :DetectIndent
"
"" to prefer expandtab to noexpandtab when detection is
"" impossible:
":let g:detectindent_preferred_expandtab = 1
"
"" to set a preferred indent level when detection is
"" impossible:
":let g:detectindent_preferred_indent = 4
"
" Requirements: Untested on Vim versions below 6.2
fun! IsCommentStart(line)
" &comments isn't reliable
if &ft == "c" || &ft == "cpp"
return -1 != match(a:line, '/\*')
else
return 0
endif
endfun
fun! IsCommentEnd(line)
if &ft == "c" || &ft == "cpp"
return -1 != match(a:line, '\*/')
else
return 0
endif
endfun
fun! DetectIndent()
let l:has_leading_tabs= 0
let l:has_leading_spaces = 0
let l:shortest_leading_spaces_run = 0
let l:longest_leading_spaces_run = 0
let l:idx_end = line("$")
let l:idx = 1
while l:idx <= l:idx_end
let l:line = getline(l:idx)
" try to skip over comment blocks, they can give really screwy
indent
" settings in c/c++ files especially
if IsCommentStart(l:line)
while l:idx <= l:idx_end && ! IsCommentEnd(l:line)
let l:line = getline(l:idx)
let l:idx = l:idx + 1
endwhile
let l:idx = l:idx + 1
continue
endif
let l:leading_char = strpart(l:line, 0, 1)
if l:leading_char == "\t"
let l:has_leading_tabs = 1
elseif l:leading_char == " "
" only interested if we don't have a run of spaces followed
by a
" tab.
if -1 == match(l:line, '^ \+\t')
let l:has_leading_spaces = 1
let l:spaces = strlen(matchstr(l:line, '^ \+'))
if l:shortest_leading_spaces_run == 0 ||
\ l:spaces < l:shortest_leading_spaces_run
let l:shortest_leading_spaces_run = l:spaces
endif
if l:spaces > l:longest_leading_spaces_run
let l:longest_leading_spaces_run = l:spaces
endif
endif
endif
let l:idx = l:idx + 1
endwhile
if l:has_leading_tabs && ! l:has_leading_spaces
" tabs only, no spaces
set noexpandtab
if exists("g:detectindent_preferred_tabsize")
let &shiftwidth = g:detectindent_preferred_indent
let &tabstop = g:detectindent_preferred_indent
endif
elseif l:has_leading_spaces && ! l:has_leading_tabs
" spaces only, no tabs
set expandtab
let &shiftwidth = l:shortest_leading_spaces_run
elseif l:has_leading_spaces && l:has_leading_tabs
" spaces and tabs
set noexpandtab
let &shiftwidth = l:shortest_leading_spaces_run
" , time to guess how big tabs are
if l:longest_leading_spaces_run < 2
let &tabstop = 2
elseif l:longest_leading_spaces_run < 4
let &tabstop = 4
else
let &tabstop = 8
endif
else
" no spaces, no tabs
if exists("g:detectindent_preferred_tabsize")
let &shiftwidth = g:detectindent_preferred_indent
let &tabstop = g:detectindent_preferred_indent
endif
if exists("g:detectindent_preferred_expandtab")
set expandtab
endif
endif
endfun
command! -nargs=0 DetectIndent call DetectIndent()
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IBM P-690 server looking for a home
On 19-Aug-10 10:48 AM, Randall Walls wrote: Greetings, The company I work for has an IBM P-690 server that is in the process of being retired. It is still a viable server, and has seen almost 0 use (it was our failover machine). Unfortunately for us, this machine has little to no resale value, and will probably be junked. I'd rather it go to a good home, and having taken advantage of the work of the python development community for a number of years (we use python extensively in system admin and database work), I saw this as an opportunity to give back a little. So, If anyone is interested in this machine, please let me know. We are looking at perhaps a November time frame for when it will be removed from our remote site. The P690 is no small machine, it is the size of a full rack and has 32 Power4 processors in it and takes (I believe) 2 or 3 phase 220 Volt power. It weighs nearly a ton. We are running AIX5.3 on it, but I believe that the machine is capable of running a PowerPC flavor of Linux as well. This would make a great test machine for python HPC modules or as a community box where developers could test their code against a PowerPC architecture. It has lots of life left and I'd rather see it put to use then thrown away. Snakebite[1]'s always got an eye out for free hardware, but dang, that's one chunky piece of kit. I'll follow up in private. (And yeah, I'm still working on Snakebite, for those that are interested. Turns out hosting three racks of heavy-duty hardware in the corner room of a (graciously donated) science lab takes a bit longer than originally anticipated. Who would have thought.) Regards, Trent "no-news-is-good-news" Nelson. [1]: http://www.snakebite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 07-Nov-10 1:55 AM, Nick Coghlan wrote: On Sun, Nov 7, 2010 at 3:53 AM, Giampaolo Rodolà wrote: In such cases I would find more easy to be able to connect to the machine and test myself rather than create a separate branch, commit, schedule a buildbot run, wait for it to complete and see whether everything is "green". On the other side I perfectly understand how opening up blanket ssh access is not something everyone is comfortable with doing. AFAICR there was someone who was setting up an evironment to solve exactly this problem but I'm not sure whether this is already usable. Dealing with exactly this problem is one of the goals of the Snakebite project. As far as I know, the folks behind that project are still working on it - I've cc'ed Trent Nelson to see if he can provide any additional info on the topic. Thanks for the ping Nick, I might have missed this otherwise. Good timing, too, as Titus and I were just discussing which low hanging fruit/pain points Snakebite should tackle first (now that all the server room stuff has finally been taken care of). Luckily, the problems that we faced 2.5 years ago when I came up with the idea of Snakebite are still just as ever present today ;-) 1. Not having access to buildbots is a pain when something doesn't work right. Doing dummy debug commits against trunk to try and coerce some more information out of a failing platform is painful. Losing a build slave entirely due to a particularly hard crash and requiring the assistance of the owner is also super frustrating. 2. The buildbot web interface for building non-(trunk|2.x|py3k) branches is also crazy unfriendly. Per-activity branches are a great way to isolate development, even with Subversion, but it kinda' blows that you don't *really* get any feedback about how your code behaves on other platforms until you re-integrate your changes back into a mainline branch. (I'm sure none of us have been masochistic enough to manually kick off individual builds for every platform via the buildbot web page after every commit to a non-standard branch.) So, enter Snakebite. We've got three racks filled with way more hardware than I should have ever purchased. Ignoring the overhead of having to set machines up and whatnot, let's just assume that over the next couple of months, if there's a platform we need a stable buildbot for, Snakebite can provide it. (And if we feel like bringing IRIX/MIPS and Tru64/Alphas back as primary platforms, we've got the hardware to do that, too ;-).) Now, the fact that they're all in the one place and under my complete control is a big advantage, as I can start addressing some of the pain points that lead me down this twisted path 2.5 years ago. I'd like to get some feedback from the development community on what they'd prefer. In my mind, I could take one of the following two steps: 1. Set up standard build slaves on all the platforms, but put something in place that allowed committers to ssh/mstsc in to said slaves when things go wrong in order to aid with debugging and/or maintaining general buildbot health (OK'ing modal crash dialogues on Windows, for example). 2. Address the second problem of the buildbot web interface sucking for non-standard branches. I'm thinking along the lines of a hack to buildbot, such that upon creation of new per-activity branches off a mainline, something magically runs in the background and sets up a complete buildbot view at python.snakebite.org/dev/buildbot/, just as if you were looking at a trunk buildbot page. I'm not sure how easy the second point will be when we switch to hg; and I'll admit if there have been any python-dev discussions about buildbot once we're on hg, I've missed them. Of course there's a third option, which is using the infrastructure I've mentioned to address a similarly annoying pain point I haven't thought of -- so feel free to mention anything else you'd like to see first instead of the above two things. Titus, for example, alluded to some nifty way for a committer to push his local hg branch/changes somewhere, such that it would kick off builds on multiple platforms in the same sorta' vein as point 2, but able to leverage cloud resources like Amazon's EC2, not just Snakebite hardware. Look forward to hearing some feedback! Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 14-Nov-10 3:48 AM, David Bolen wrote: This is a completely separate issue, though probably around just as long, and like the popup problem its frequency changes over time. By "hung" here I'm referring to cases where something must go wrong with a test and/or its cleanup such that a python_d process remains running, usually several of them at the same time. My guess: the "hung" (single-threaded) Python process has called select() without a timeout in order to wait for some data. However, the data never arrives (due to a broken/failed test), and the select() never returns. On Windows, processes seem harder to kill when they get into this state. If I purposely wedge a Windows process via select() via the interactive interpreter, ctrl-c has absolutely no effect (whereas on Unix, ctrl-c will interrupt the select()). As for why kill_python.exe doesn't seem to be able to kill said wedged processes, the MSDN documentation on TerminateProcess[1] states the following: The terminated process cannot exit until all pending I/O has been completed or canceled. (sic) It's not unreasonable to assume a wedged select() constitutes pending I/O, so that's a possible explanation as to why kill_python.exe isn't able to terminate the processes. (Also, kill_python currently assumes TerminateProcess() always works; perhaps this optimism is misplaced. Also note the XXX TODO regarding the fact that we don't kill processes that have loaded our python*.dll, but may not be named python_d.exe. I don't think that's the issue here, though.) On 14-Nov-10 5:32 AM, David Bolen wrote: > "Martin v. Löwis" writes: > >> This is what kill_python.exe is supposed to solve. So I recommend to >> investigate why it fails to kill the hanging Pythons. > > Yeah, I know, and I can't say I disagree in principle - not sure why > Windows doesn't let the kill in that module work (or if there's an > issue actually running it under all conditions). > > At the moment though, I do know that using the sysinternals pskill > utility externally (which is what I currently do interactively) > definitely works so to be honest, That's interesting. (That kill_python.exe doesn't kill the wedged processes, but pskill does.) kill_python is pretty simple, it just calls TerminateProcess() after acquiring a handle with the relevant PROCESS_TERMINATE access right. That being said, that's the recommended way to kill a process -- I doubt pskill would be going about it any differently (although, it is sysinternals... you never know what kind of crazy black magic it's doing behind the scenes). Are you calling pskill with the -t flag? i.e. kill process and all dependents? That might be the ticket, especially if killing the child process that wedged select() is waiting on causes it to return, and thus, makes it killable. Otherwise, if it happens again, can you try kill_python.exe first, then pskill, and confirm if the former fails but the latter succeeds? Trent. [1]: http://msdn.microsoft.com/en-us/library/ms686714(VS.85).aspx ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
Hi,
Has anyone else built Python with Visual Studio 2005 and played around
with Profile Guided Optimization? I had to build Python from source w/
VS 2005 as I had a few .pyd's built with VS 2005 that I wanted to load;
I ended up playing around with Profile Guided Optimization, running
``python.exe pystones.py'' to collect call-graph data after
python.exe/Python24.dll had been instrumented, then recompiling with the
optimizations fed back in.
Results were interesting, an average speedup of around 33% was
noticeable:
ActiveState 2.4.3 python.exe:
C:\Python24>python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.980119 This machine
benchmarks at
51014.2 pystones/second
The python compiled from branches/release24-maint with VS 2005 + profile
guided optimization:
C:\Python24>python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.73261 This machine benchmarks
at
68249.2 pystones/second
Is there any motivation in the Win32 Python dev camp to switch from VC6
to VS 2005?
FWIW, although there were a shed-load of warnings when compiling python
and pythoncore (and a lot more errors when compiling other modules), I
only had to apply one patch to get it working well enough to run
pystone.py. Without this patch, the VC8 CRT aborts at runtime as soon
as an invalid signal is passed to signal(); which is inevitable given
the current code in the initsignal() method:
for (i = 1; i < NSIG; i++) {
void (*t)(int);
t = PyOS_getsig(i);
Regards,
Trent.
--
http://www.onresolve.com
Index: signalmodule.c
===
--- signalmodule.c (revision 47196)
+++ signalmodule.c (working copy)
@@ -280,7 +280,21 @@
{NULL, NULL} /* sentinel */
};
+#define WIN32VS2005HACK
+#ifdef WIN32VS2005HACK
+#include
+#include
+#include
+void dummy_handler(const wchar_t *exp,
+ const wchar_t *fn,
+ const wchar_t *file,
+ unsigned int line,
+ uintptr_t reserved)
+{
+}
+#endif
+
PyDoc_STRVAR(module_doc,
"This module provides mechanisms to use signal handlers in Python.\n\
\n\
@@ -339,6 +353,12 @@
goto finally;
Py_INCREF(IntHandler);
+#ifdef WIN32VS2005HACK
+(void)_set_invalid_parameter_handler(&dummy_handler);
+_CrtSetReportMode(_CRT_ASSERT, 0);
+#endif
+
+
Handlers[0].tripped = 0;
for (i = 1; i < NSIG; i++) {
void (*t)(int);
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any tips to tell sprinter at PyCon about developing on Windows?
Feb 2, 2008 7:34 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > > Brett Cannon wrote: > > It would be really cool if you can recruit some experienced Windows > > developers. :] > That's the point in all of this. =) > -Brett I'll be around for the sprints -- didn't really have a plan as to what I'd like to sprint on but if there's some interest in farming Windows developers, I'll raise my hand. Anything in particular you can point myself or others in the Windows camp at such that we're a bit better prepared come sprint time (i.e. open issues)? (Also, I'm looking to acquire a new reasonably well-spec'd Windows box for work. If it's available in time for PyCon, I should be able to set up a couple of virtual 64-bit Vista/Server 2008 images with VS 2008 dev environments that non-Windows developers could use, if that would be desirable.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding a new Windows x64 buildbot
Hi, I've got a Windows Server 2008 x64 server I'd like to contribute as a buildbot. As per the recommendation on http://wiki.python.org/moin/BuildBot, it sounds like I'm looking for Martin, Anthony or Neal to sort me out with slave credentials. Feel free to drop me a line! Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Python 2.6 and 3.0
> (unless a complete working solution is presented in that other technology, > and as long as that other technology still creates MSI files with > free-as-in-beer tools). Just out of interest, what's the reason for enforcing that the installer must be an MSI? Or, rather, if I were to present an alternative .exe installer that ticks all of the above boxes, exceeds the capabilities of the current installer and above all is easier to extend and maintain -- would that be a non-starter because it's not an MSI? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Howdy, I'm going through the motions of getting my newly added build slave in a half decent state. The external.bat and external-amd64.bat files needed the following in order to build db-4.4.20: Index: external.bat === --- external.bat(revision 61125) +++ external.bat(working copy) @@ -10,7 +10,8 @@ @rem Sleepycat db if not exist db-4.4.20 svn export http://svn.python.org/projects/external/db-4.4.20 if not exist db-4.4.20\build_win32\debug\libdb44sd.lib ( - vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug /project db_static + devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln + devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug /project db_static ) @rem OpenSSL (This is against trunk, same thing would apply to py3k I guess, given that we're using %VS90COMNTOOLS%vsvars32.bat there too.) Regards, Trent. -- http://www.onresolve.com external.bat.patch Description: external.bat.patch external-amd64.bat.patch Description: external-amd64.bat.patch ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
> > I'm going through the motions of getting my newly added build slave > in a half decent state. > > I think the buildbot should have a name different from 'x86 XP'. > (Martin, Neal?) > > Thomas Yeah, I've dropped Martin a note regarding this. The community bots refer to Windows Server 2003 boxes as just that, so perhaps a rename to 'x86 Windows Server 2008' is appropriate. FWIW as it's a 64-bit box, I'm hoping to get a slave set up for 'x64 Windows Server 2008' as well. (As far as I can see, the x64/x86 nature of the slave is dictated by the master, correct? i.e. I can't tweak/clone this myself?) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Christian Heimes: > Trent Nelson wrote: > > - vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > /project db_static > > + devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln > > + devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > /project db_static > > The upgrade is requires only once. It probably belongs next to the > checkout or svn up and not in the build section. Makes sense. So we're looking at something like: @rem Sleepycat db if not exist db-4.4.20 ( svn export http://svn.python.org/projects/external/db-4.4.20 devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln ) if not exist db-4.4.20\build_win32\debug\libdb44sd.lib ( devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug ) I'll test this when I get to work and report back. Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Christian Heimes: > Thomas Heller wrote: > > What's the difference between these two? > > > > vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > > > > devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > > Devenv is the name of the VS GUI executable but it can *also* be used as > a CLI to build stuff. devenv doesn't work for Express Edition. > > vcbuild seems to be the preferred CLI app to build a project but it's > limited. I think it doesn't support /upgrade. Hummm. My answer would be more along the lines of "devenv works, vcbuild doesn't" ;-) S:\buildbots\python\trunk.nelson-windows\db-4.4.20\build_win32>vcbuild Berkeley_DB.sln /build Debug /project db_static Microsoft (R) Visual C++ Project Builder - Command Line Version 9.00.21022 Copyright (C) Microsoft Corporation. All rights reserved. vcbuild.exe : warning VCBLD6002: invalid option /build specified. The option was ignored. vcbuild.exe : warning VCBLD6002: invalid option /project specified. The option was ignored. vcbuild.exe : warning VCBLD6002: invalid option db_static specified. The option was ignored. vcbuild.exe : error VCBLD0006: invalid configuration name: DEBUG. Compare this to: S:\buildbots\python\trunk.nelson-windows\db-4.4.20\build_win32>devenv Berkeley_DB.sln /build Debug /project db_static Microsoft (R) Visual Studio Version 9.0.21022.8. Copyright (C) Microsoft Corp. All rights reserved. == Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped == I don't know how the existing vcbuild line ever worked, given the following output from vcbuild /?: Usage: vcbuild [options] [project|solution] [config|$ALL] Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
Spent some time on my buildbot (x86 2k8 trunk) this morning trying to track down why test_bsddb3 is failing (trunk with db-4.4.20). The first test that fails is this: test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ERROR That's slightly misleading though as the test runs fine -- the actual exception is being thrown in test_basics.BasicTestCase.tearDown() when os.remove() is called against the first db file (i.e. '__db.001'): WindowsError: [Error 5] Access is denied: 'c:\users\trent~1.nel\appdata\local\temp\2\db_home2808\__db.001 This isn't surprising, given that the python_d.exe process still seems to have __db.001, __db.002 and __db.003 open at the time os.remove() is called. The aforementioned tearDown() method looks like this: def tearDown(self): self.d.close() if self.env is not None: test_support.rmtree(self.homeDir) self.env.close() ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) else: os.remove(self.filename) If I switch the order of statements such that self.env.close() is called before test_suppot.rmtree(self.homeDir), this particular test and a host of others that were also failing now pass (a runtime abort is no longer raised by the CRT half way into the tests either). (Note that the order was switched to that shown above by Neal in r61052 on Feb 24th, which is when these issues started occurring.) That said, there are still a lot more test failures down the track once this change has been made, either via the access denied WindowsError, or a DBInvalidArgError, e.g.: ERROR: test02_WithSource (bsddb.test.test_recno.SimpleRecnoTestCase) -- Traceback (most recent call last): File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_recno.py", line 33, in tearDown test_support.rmtree(self.homeDir) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\test\test_support.py", line 70, in rmtree shutil.rmtree(path) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 184, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 182, in rmtree os.remove(fullname) WindowsError: [Error 5] Access is denied: 'c:\\users\\trent~1.nel\\appdata\\local\\temp\\2\\db_home4656\\tmp04_knk' == ERROR: test01_1WriterMultiReaders (bsddb.test.test_thread.BTreeConcurrentDataStore) -- Traceback (most recent call last): File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_thread.py", line 62, in setUp self.env.open(homeDir, self.envflags | db.DB_CREATE) DBInvalidArgError: (22, 'Invalid argument -- configured environment flags incompatible with existing environment') The DBInvalidArgError exception is only raised after a previous WindowsError is encountered, so I assume it's a side-effect of the tearDown() method not cleaning the environment correctly. It seems this comment in tearDown() is quite pertinent to our situation: ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) Not sure why this is commented out -- quick search of svn history indicates it's been like that for at least the last year and a half. Will have some more time this evening to spend on this, however, work calls at the moment. Regards, Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Facundo Batista [EMAIL PROTECTED] Sent: 26 February 2008 06:22 To: Thomas Hervé Cc: python-dev@python.org Subject: Re: [Python-Dev] Buildbots for trunk are all red 2008/2/25, Thomas Hervé <[EMAIL PROTECTED]>: > I've worked on that problem during the bug day. I've open a ticket with > a patch at http://bugs.python.org/issue2168. Most of the buildbots are green now!!! Thank you all! This community is as awesome as Python itself, ;) Three remains in red, though: - Alpha Tru64: test_smtplib.py is flaky, and _ssl.c is not compiled correctly. Neil is hunting this, I think. - X86 XP-3: seems to crash after test_bsddb3.py. - X86 XP-4: idem. For this two, how can be tried if the bsddb lib in those windows is correctly installed? Thanks again. -- .Fac
Re: [Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
> Trent, thanks for working on the buildbot. I fixed the first case you > mentioned in r61233 wrt removing the directory before closing the > file. It would be great if you could submit a patch when you are able > to fix the remaining problems. Nod, found a few more things now that test_bsddb3 isn't causing a CRT abortion. tmpfile() needs to be reworked on Windows, see http://bugs.python.org/issue2232. Going to spend some more time on it this evening. I'm determined to see a flippin' green build/test status for my slave if it kills me :> Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
> Trent, thanks for working on the buildbot. I fixed the first case you > mentioned in r61233 wrt removing the directory before closing the > file. It would be great if you could submit a patch when you are able > to fix the remaining problems. % svn diff Index: test_dbshelve.py === --- test_dbshelve.py(revision 61233) +++ test_dbshelve.py(working copy) @@ -267,8 +267,8 @@ def tearDown(self): +self.do_close() test_support.rmtree(self.homeDir) -self.do_close() class EnvBTreeShelveTestCase(BasicEnvShelveTestCase): Index: test_thread.py === --- test_thread.py (revision 61233) +++ test_thread.py (working copy) @@ -73,9 +73,9 @@ self.d.open(self.filename, self.dbtype, self.dbopenflags|db.DB_CREATE) def tearDown(self): -test_support.rmtree(self.homeDir) self.d.close() self.env.close() +test_support.rmtree(self.homeDir) def setEnvOpts(self): pass I'm getting 100% success rate with test_bsddb3 on Windows now with this patch. Yay! Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] signal.alarm(3) in trunk test_socketserver.py
r61099 added the following to trunk/Lib/test/test_socketserver.py: if __name__ == "__main__": test_main() + signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. which breaks platforms that don't have signal.alarm, like, say, !unix ;-) Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] signal.alarm(3) in trunk test_socketserver.py
> r61099 added the following to trunk/Lib/test/test_socketserver.py: > > if __name__ == "__main__": > test_main() > + signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. > Actually, signal.alarm() was introduced all over the place in that revision. I understand the intent of this commit was to speed up the runtime of this test (something like 28s -> 0.3s was quoted in the commit log). FWIW, runtime of the test with the following patch on Windows is 0.125s: Index: test_socketserver.py === --- test_socketserver.py(revision 61233) +++ test_socketserver.py(working copy) @@ -28,6 +28,9 @@ HAVE_UNIX_SOCKETS = hasattr(socket, "AF_UNIX") HAVE_FORKING = hasattr(os, "fork") and os.name != "os2" +def signal_alarm(n): +if hasattr(signal, 'alarm'): +signal.alarm(n) def receive(sock, n, timeout=20): r, w, x = select.select([sock], [], [], timeout) @@ -99,7 +102,7 @@ """Test all socket servers.""" def setUp(self): -signal.alarm(20) # Kill deadlocks after 20 seconds. +signal_alarm(20) # Kill deadlocks after 20 seconds. self.port_seed = 0 self.test_files = [] @@ -112,7 +115,7 @@ except os.error: pass self.test_files[:] = [] -signal.alarm(0) # Didn't deadlock. +signal_alarm(0) # Didn't deadlock. def pickaddr(self, proto): if proto == socket.AF_INET: @@ -267,4 +270,4 @@ if __name__ == "__main__": test_main() -signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. +signal_alarm(3) # Shutdown shouldn't take more than 3 seconds. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Patch for trunk test_winsound.py (fixes my buildbot)
winsound.Beep fails for me on the 'x86 2k8 trunk' build slave, which is a virtual Windows Server 2008 instance running under Hyper-V. Not surprisingly, there's not a single audio-related device on this system. The attached patch to test_winsound.py incorporates the _have_soundcard() checks to the BeepTest class, which fixes the problem for me. (I've also tested the patch on a Vista system (that does have a soundcard) and everything works as expected.) Trent. test_winsound.py.patch Description: test_winsound.py.patch ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] signal.alarm(3) in trunk test_socketserver.py
> Yep, the alarm is only there to prevent what would be deadlocks from > running forever. Sorry for breaking !unix. Your patch looks fine to > me. Do you want to submit it or shall I? I'm not a committer, so it's all yours. Thanks for the quick turnaround! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors?
I've started to see my build slave dying every so often with a twisted error half way through tests: ... test_htmlparser test_httplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] Examples: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/46/step-test/0 http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/36/step-test/0 I'm not sure if I should read into the fact that it's occurring after networking-oriented tests like test_httplib and test_ftplib. Running rt.bat on the resulting build manually doesn't indicate any errors in these tests. Have other Windows buildbot owners seen this? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors?
Had a chat with some Twisted/buildbot folk and they can confirm they've seen it as well on Windows. They've given me a few things to look into. Out of interest, how are you running your buildbot? Via the command line in an interactive desktop session, as a service, or as a scheduled task, or some other way? From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Thomas Heller [EMAIL PROTECTED] Sent: 05 March 2008 10:03 To: python-dev@python.org Subject: Re: [Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors? Trent Nelson schrieb: > I've started to see my build slave dying every so often with a > twisted error half way through tests: ... test_htmlparser > test_httplib > > remoteFailed: [Failure instance: Traceback (failure with no frames): > twisted.internet.error.ConnectionLost: Connection to the other side > was lost in a non-clean fashion. ] > > Examples: > http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/46/step-test/0 > > http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/36/step-test/0 > > > I'm not sure if I should read into the fact that it's occurring after > networking-oriented tests like test_httplib and test_ftplib. Running > rt.bat on the resulting build manually doesn't indicate any errors in > these tests. Have other Windows buildbot owners seen this? > > Trent. I have not observed this behaviour on my buildbots. Have you looked into the twistd.log logfile? Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r61264 - in python/trunk: Lib/test/test_os.py Misc/NEWS
Hurrah, 'x86 W2k8 trunk' has just experienced its first green build and test! Thanks to everyone that committed the various patches I sent out in such a timely fashion. Martin, does this mean I can have a slave set up for x64 now? }:> Trent. > -Original Message- > From: [EMAIL PROTECTED] [mailto:python-checkins- > [EMAIL PROTECTED] On Behalf Of martin.v.loewis > Sent: 06 March 2008 01:55 > To: [EMAIL PROTECTED] > Subject: [Python-checkins] r61264 - in python/trunk: > Lib/test/test_os.py Misc/NEWS > > Author: martin.v.loewis > Date: Thu Mar 6 07:55:22 2008 > New Revision: 61264 > > Modified: >python/trunk/Lib/test/test_os.py >python/trunk/Misc/NEWS > Log: > Patch #2232: os.tmpfile might fail on Windows if the user has no > permission to create files in the root directory. > Will backport to 2.5. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Request for another build slave
Can someone set me up with a build slave for an x86 FreeBSD box (6.2-STABLE, although we'll be migrating to 7.x in a week or so)? Thanks. [Suggestion: perhaps we could set up a [EMAIL PROTECTED] list for discussing buildbot administrative minutiae, rather than polluting python-dev?] Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows x64 & bsddb 4.4.20 woes
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0021bbc0, ...)
ntdll.dll!773ae13d()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!773aea57()
ntdll.dll!773b59f8()
_bsddb_d.pyd!__os_strdup() + 0x18 bytes
_bsddb_d.pyd!__os_tmpdir() + 0x281 bytes
You'd think placing breakpoints in db 4.4.20's __os_strdup and __os_tmpdir
methods would do something, but alas, the bufferoverflow exception is raised
before any breakpoints are set. This makes me suspect there's something funky
going on with the entire build and linking of db_static (VS should honour those
breakpoints if the code is being entered, I even added db_static to pcbuild.sln
and rebuilt but no dice). I've noticed that they're not using consistent
compiler flags by default (we use /GS, they use /GS-, we allow function level
linking, they don't -- note that I did change db_static's options to align with
_bsddb's but the bufferoverflow exception is still being thrown).
Greg, Jesús, I'm CC'ing you guys as stfw'ing seems to bring back you two the
most when it comes to bsddb issues. I've still got a list of things to try
with regarding to debugging this x64 issue, but I wanted to reach out now to
see if anyone else had encountered it before. Has bsddb ever been built
successfully on Win64 and passed all tests or am I venturing into new ground?
Martin, you've changed externals/bsddb-4.4.20 with regards to x64 builds
recently -- have you been able to get things working in your x64 environments?
Regards,
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
Hey Greg,
I'm at PyCon indeed, staying through the sprints 'til next Thursday. I'll drop
you a note offline re catching up.
The other query I had was whether or not I should try a later version of
BerkeleyDB -- are we committed to 4.4.20 (or 4.4.x?) for 2.6/3.0 or is it worth
investigating newer versions? Martin/Jesus, any thoughts on this?
Regarding the db_static build and conflicting compile/link options -- I'm going
to bring the db_static source directly into the _bsddb project (for now) which
should make this a lot easier to debug.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 13 March 2008 22:00
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
I haven't built the bsddb stuff on windows myself in a few years and have never
had access to a windows x64 system so I'm no silver bullet. Making the
BerkeleyDB compile and link options match with those of python is the first
place I'd start. Also you should be able to make a debug build of BerkeleyDB
(though it sounds like you may have tried that already?). Next off in the
debugging i'd take a look at the assembly to see what exactly it was failing to
do.
If you're at PyCon right now we should meet up and try to figure it out (I just
arrived).
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0021bbc0, ...)
ntdll.dll!773ae13d()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!773aea57()
ntdll.dll!773b59f8()
_bsddb_d.pyd!__os_strdup() + 0x18 bytes
_bsddb_d.pyd!__os_tmpdir() + 0x281 bytes
You'd think placing breakpoints in db 4.4.20's __os_strdup and __os_tmpdir
methods would do something, but alas, the bufferoverflow exception is raised
before any breakpoints are set. This makes me suspect there's something funky
going on with the entire build and linking of db_static (VS should honour those
breakpoints if the code is being entered, I even added db_static to pcbuild.sln
and rebuilt but no dice). I've noticed that they're not using consistent
compiler flags by default (we use /GS, they use /GS-, we allow function level
linking, they don't -- note that I did change db_static's options to align with
_bsddb's but the bufferoverflow exception is still being thrown).
Greg, Jesús, I'm CC'ing you guys as stfw'ing seems to bring back you two the
most when it comes to bsddb issues. I've still got a list of things to try
with regarding to debugging this x64 issue, but I wanted to reach out now to
see if anyone else had encountered it before. Has bsddb ever been built
successfully on Win64 and passed all tests or am I venturing into new ground?
Martin, you've changed externals/bsddb-4.4.20 with regards to x64 builds
recently -- have you been able to get things working in your x64 environments?
Regards,
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
Ah, and to think I just fixed 4.4.20 ;-)
Removing the dependency on db_static.vcproj and merging the relevant source
code files into _bsddb.vcproj did the trick -- all x64 bsddb-related tests now
pass. The only issue with this approach is that it locks _bsddb.vcproj into
4.4.20. However, considering that this approach (i.e. bringing their source
files into our build instead of linking against a static lib compiled with
wildly incompatible flags) only took me about two minutes to implement and
immediately fixed every bsddb problem I was encoutering, I'm convinced it's the
right way to go. (I can separate the dependencies easily enough.)
Woeful PyCon/hotel connectivity is preventing me from getting to
bugs.python.org at the moment; I'll raise a ticket later to capture this stuff
and we can move the discussion there once I've attached some patches.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 14 March 2008 00:23
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
Hey Greg,
I'm at PyCon indeed, staying through the sprints 'til next Thursday. I'll drop
you a note offline re catching up.
The other query I had was whether or not I should try a later version of
BerkeleyDB -- are we committed to 4.4.20 (or 4.4.x?) for 2.6/3.0 or is it worth
investigating newer versions? Martin/Jesus, any thoughts on this?
Python 2.6/3.0 should be built on Windows using BerkeleyDB 4.5.x for now.
4.6.x is out but has some bugs on some platforms so i don't recommend shipping
our release using it; 4.7.x is in beta and some bugs are being worked on; if
its out and shows no signs of obvious issues before the 2.6/3.0 beta period is
over I recommend we build our binary releases using it. Otherwise 4.5 it will
be. There is no reason to use 4.4.x.
Regarding the db_static build and conflicting compile/link options -- I'm going
to bring the db_static source directly into the _bsddb project (for now) which
should make this a lot easier to debug.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 13 March 2008 22:00
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
I haven't built the bsddb stuff on windows myself in a few years and have never
had access to a windows x64 system so I'm no silver bullet. Making the
BerkeleyDB compile and link options match with those of python is the first
place I'd start. Also you should be able to make a debug build of BerkeleyDB
(though it sounds like you may have tried that already?). Next off in the
debugging i'd take a look at the assembly to see what exactly it was failing to
do.
If you're at PyCon right now we should meet up and try to figure it out (I just
arrived).
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
> > Removing the dependency on db_static.vcproj and merging the relevant > > source code files into _bsddb.vcproj did the trick -- all x64 > > bsddb-related tests now pass. The only issue with this approach is > > that it locks _bsddb.vcproj into 4.4.20. However, considering that > > this approach (i.e. bringing their source files into our build > > instead of linking against a static lib compiled with wildly > > incompatible flags) only took me about two minutes to implement and > > immediately fixed every bsddb problem I was encoutering, I'm > > convinced it's the right way to go. (I can separate the dependencies > > easily enough.) > > I'm convinced this is the wrong approach. Are you sure you copied > all compiler settings over to the project correctly? What is the > procedure to upgrade such a setup? What is the procedure for people > who want to build with a different version of bsddb? I reviewed all the compiler options used by db_static.vcproj -- the only thing I needed to bring over was -DDIAGNOSTIC for debug builds. Everything else either had no impact and could be safely dropped, or conflicted with compiler options used by the rest of the python build (function level linking, buffer overflow checks, etc). Regarding support for users who want to build with different versions of bsddb; if they want a functional build that passes tests they're going to have to do something similar to the work I've done anyway. As it stands now, the .lib generated by db_static.vcproj for x64 builds just straight out does not work. That can be fixed in two ways: coerce db_static.vcproj into matching our build, or mimicking db_static in a new .vcproj that's contained with our build, inheriting our property sheets. I chose the latter. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 3.0 buildbots all red
http://www.python.org/dev/buildbot/3.0/ New sprint idea: getting all (inc. trunk) the buildbots green by Thursday. Anyone interested? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.6 and 3.0 tasks
> > * Replace Windows API calls with wide versions to support unicode > >for file names, environment etc. > > +1. This should be broken into separate tasks for each API. What are we referring to here? Calling the W versions explicitly and using wchar_t for everything, or using the TCHAR/TEXT() approach and keeping the API calls the same, letting the #define UNICODE do the work behind the scenes? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
> > New sprint idea: getting all (inc. trunk) the buildbots green by > Thursday. Anyone interested? > > I think the chance to achieve that is close to zero. Sounds like a challenge if ever I've heard one -- care to wager a beer on it? (Only applies to buildbots that are connected/online.) (FWIW, I've got the x64 Windows build green on my dev server, tcl/tk and bsddb required patching, so did some tests, and so did some C code -- I'm in the process of filtering the efforts back into the tracker.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
Yeah test_tokenize is weird, I've been looking into it as well. Here's a
sample failure from a Windows buildbot:
File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_tokenize.py",
line ?, in test.test_tokenize.__test__.doctests
Failed example:
for testfile in testfiles:
if not roundtrip(open(testfile)): break
else: True
Exception raised:
Traceback (most recent call last):
File "S:\buildbots\python\3.0.nelson-windows\build\lib\doctest.py", line
1227, in __run
compileflags, 1), test.globs)
File "", line 2, in
if not roundtrip(open(testfile)): break
File "", line 3, in
roundtrip
token_list = list(generate_tokens(f.readline))
File "S:\buildbots\python\3.0.nelson-windows\build\lib\tokenize.py", line
264, in generate_tokens
line = readline()
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1467,
in readline
readahead, pending = self._read_chunk()
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1278,
in _read_chunk
pending = self._decoder.decode(readahead, not readahead)
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1081,
in decode
output = self.decoder.decode(input, final=final)
File
"S:\buildbots\python\3.0.nelson-windows\build\lib\encodings\cp1252.py", line
23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 17:
character maps to
The following is at the end of the doctests in test_tokenize:
>>> tempdir = os.path.dirname(f) or os.curdir
>>> testfiles = glob.glob(os.path.join(tempdir, "test*.py"))
>>> if not test_support.is_resource_enabled("compiler"):
... testfiles = random.sample(testfiles, 10)
...
>>> for testfile in testfiles:
... if not roundtrip(open(testfile)): break
... else: True
True
On that first line, 'f' is lib/test/tokenize_tests.txt, so basically, it's
grabbing ten random test*.py files in lib/test and running
untokenize(generate_tokens(f.readline)) on each one. In order to figure out
which file it's dying on, I added the following to test_tokenize.py:
def test_tokenize_all():
import glob
import os
tempdir = os.path.dirname(__file__) or os.curdir
testfiles = glob.glob(os.path.join(tempdir, "test*.py"))
for file in testfiles:
print("processing file: " + file)
print("roundtrip(open(file)): " + roundtrip(open(file)))
This results in different results:
Python 3.0a3+ (py3k, Mar 16 2008, 10:41:45) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_tokenize
[50808 refs]
>>> test_tokenize.test_tokenize_all()
processing file:
s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\testcodec.py
Traceback (most recent call last):
File "", line 1, in
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line 565, in test_tokenize_all
print("roundtrip(open(file)): " + roundtrip(open(file)))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line 514, in roundtrip
source = untokenize(generate_tokens(f.readline))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
238, in untokenize
return ut.untokenize(iterable)
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
183, in untokenize
self.add_whitespace(start)
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
172, in add_whitespace
assert row <= self.prev_row
AssertionError
[52668 refs]
Yay. And to make this even more interesting:
s:\src\svn\svn.python.org\projects\python\branches\py3k\PCbuild>python_d
..\Lib\test\test_tokenize.py
doctest (test.test_tokenize) ... 62 tests with zero failures
[61919 refs]
Oh, and while we're here:
s:\src\svn\svn.python.org\projects\python\branches\py3k\PCbuild>python_d
..\lib\test\regrtest.py -q -uall -rw test_tokenize
**
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line ?, in test.test_tokenize.__test__.doc
tests
Failed example:
for testfile in testfiles:
if not roundtrip(open(testfile)): break
else: True
Exception raised:
Traceback (most recent call last):
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\doctest.py", line
1227, in __run
compileflags, 1), test.globs)
File "", line 2, in
if not roundtrip(open(testfile)): break
File "", line 3, in
roundtrip
token_list = list(generate_tokens(f.readline))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
264, in generate_tokens
Re: [Python-Dev] 3.0 buildbots all red
As it turns out, it's not memory related, but has to do with tokenize not supporting coding cookies in files. Mark picked up on this and linked it to an issue already in roundup that was raised way back in 2003: http://bugs.python.org/issue71988. I've just finished patching test_tokenizer.py to better represent this test case -- the current implementation doesn't lend itself very well to being debugged when things go wrong (I think Mark and I both felt like we were on a bit of a wild goose chase). I've fixed that and have a bunch of text files with various utf-8/bom sig permutations that are now used to test tokenizer's compliance with PEP 0263. I'll upload that now then move on to actually patching tokenizer.py. Trent "wishes-there-was-somewhere-to-get-some-food-after-11pm-at-pycon" Nelson. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of ocean [EMAIL PROTECTED] Sent: 17 March 2008 01:34 To: Neal Norwitz; Mark Dickinson Cc: Python Dev Subject: Re: [Python-Dev] 3.0 buildbots all red > Yeah, sounds like a memory issue. Did you try running with valgrind > or purify? I haven't done so for a long time, perhaps never on 3k > branch. It would be a good thing to run a tool soon. Maybe is this related? [Potential overflows due to incorrect usage of PyUnicode_AsString] http://bugs.python.org/issue1950 Thank you. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
> As it turns out, it's not memory related, but has to do with > tokenize not supporting coding cookies in files. > Mark picked up on this and linked it to an issue already > in roundup that was raised way back in 2003: > http://bugs.python.org/issue71988. Oops, left off an 8. That's meant to read http://bugs.python.org/issue719888. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows)
+1 for avoiding a bikeshed, so +1 to AMD64. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Christian Heimes [EMAIL PROTECTED] Sent: 18 March 2008 13:54 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; python-dev@python.org Subject: Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows) [EMAIL PROTECTED] schrieb: > So, at the risk of painting a bike-shed, I'd like to propose that we adopt > 'AMD64' in distutils (needs a change), platform.py (needs a change to use > sys.getwindowsversion() in preference to pywin32, if possible, anyway), > and the Python banner (which already uses AMD64). +1 for AMD64 If we ever need names for Itanium and i386 compatible arch I propose IA64 and X86. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r61577 - in python/trunk: Include/code.h Include/compile.h Include/parsetok.h Include/pythonrun.h Lib/__future__.py Lib/test/test_print.py Misc/ACKS Misc/NEWS Parser
This change breaks all the trunk buildbots: == ERROR: testCompileLibrary (test.test_compiler.CompilerTest) -- Traceback (most recent call last): File "S:\buildbots\python\trunk.nelson-windows\build\lib\test\test_compiler.py", line 52, in testCompileLibrary compiler.compile(buf, basename, "exec") File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 64, in compile gen.compile() File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 112, in compile gen = ModuleCodeGenerator(tree) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 1275, in __init__ self.futures = future.find_futures(tree) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 59, in find_futures walk(node, p1) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 106, in walk walker.preorder(tree, visitor) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 63, in preorder self.dispatch(tree, *args) # XXX *args make sense? File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 57, in dispatch return meth(node, *args) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 27, in visitModule if not self.check_stmt(s): File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 37, in check_stmt "future feature %s is not defined" % name SyntaxError: future feature print_function is not defined From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of eric.smith [EMAIL PROTECTED] Sent: 18 March 2008 19:45 To: [EMAIL PROTECTED] Subject: [Python-checkins] r61577 - in python/trunk: Include/code.h Include/compile.h Include/parsetok.h Include/pythonrun.h Lib/__future__.py Lib/test/test_print.py Misc/ACKS Misc/NEWSParser/parser.c Parser/parsetok.c Python/bltinmodule.c Python/future.c Pyth... Author: eric.smith Date: Wed Mar 19 00:45:49 2008 New Revision: 61577 Added: python/trunk/Lib/test/test_print.py Modified: python/trunk/Include/code.h python/trunk/Include/compile.h python/trunk/Include/parsetok.h python/trunk/Include/pythonrun.h python/trunk/Lib/__future__.py python/trunk/Misc/ACKS python/trunk/Misc/NEWS python/trunk/Parser/parser.c python/trunk/Parser/parsetok.c python/trunk/Python/bltinmodule.c python/trunk/Python/future.c python/trunk/Python/pythonrun.c Log: Backport of the print function, using a __future__ import. This work is substantially Anthony Baxter's, from issue 1633807. I just freshened it, made a few minor tweaks, and added the test cases. I also created issue 2412, which is to check for 2to3's behavior with the print function. I also added myself to ACKS. Modified: python/trunk/Include/code.h == --- python/trunk/Include/code.h (original) +++ python/trunk/Include/code.h Wed Mar 19 00:45:49 2008 @@ -48,11 +48,12 @@ #define CO_FUTURE_DIVISION 0x2000 #define CO_FUTURE_ABSOLUTE_IMPORT 0x4000 /* do absolute imports by default */ #define CO_FUTURE_WITH_STATEMENT 0x8000 +#define CO_FUTURE_PRINT_FUNCTION 0x1 /* This should be defined if a future statement modifies the syntax. For example, when a keyword is added. */ -#if 0 +#if 1 #define PY_PARSER_REQUIRES_FUTURE_KEYWORD #endif Modified: python/trunk/Include/compile.h == --- python/trunk/Include/compile.h (original) +++ python/trunk/Include/compile.h Wed Mar 19 00:45:49 2008 @@ -24,6 +24,8 @@ #define FUTURE_DIVISION "division" #define FUTURE_ABSOLUTE_IMPORT "absolute_import" #define FUTURE_WITH_STATEMENT "with_statement" +#define FUTURE_PRINT_FUNCTION "print_function" + struct _mod; /* Declare the existence of this type */ PyAPI_FUNC(PyCodeObject *) PyAST_Compile(struct _mod *, const char *, Modified: python/trunk/Include/parsetok.h == --- python/trunk/Include/parsetok.h (original) +++ python/trunk/Include/parsetok.h Wed Mar 19 00:45:49 2008 @@ -27,6 +27,10 @@ #define PyPARSE_WITH_IS_KEYWORD0x0003 #endif +#define PyPARSE_PRINT_IS_FUNCTION 0x0004 + + + PyAPI_FUNC(node *) PyParser_ParseString(const char *, grammar *, int, perrdetail *); PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int, Modified: python/trunk/Include/pythonrun.h == --- python/trunk/Include/pythonrun.h(o
Re: [Python-Dev] 3.0 buildbots all red
> > Sounds like a challenge if ever I've heard one -- care to wager a beer on > > it? > > (Only applies to buildbots that are connected/online.) > Make sure you get a screen shot for OnYourDesktop if/when they *do* go > green! Screenshot? I'm going to buy a pack of iron-on transfers and sell t-shirts of it online. "All the buildbots were green momentarily after PyCon 2008... and all I got was this lousy t-shirt." Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] First green Windows x64 buildbots!
We've just experienced our first 2.6 green x64 Windows builds on the build slaves! Well, almost green. Thomas's 'amd64 XP trunk' ran out of disk: 304 tests OK. 1 test failed: test_largefile == ERROR: test_seek (test.test_largefile.TestCase) -- Traceback (most recent call last): File "C:\buildbot\trunk.heller-windows-amd64\build\lib\test\test_largefile.py", line 42, in test_seek f.flush() IOError: [Errno 28] No space left on device Sorry about that Thomas ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to build extensions for Windows?
Having recently sunk a lot of time into the Windows build process, I'd recommend going with Visual C++ Express 2008 rather than MinGW, as this is the official compiler for 2.6/3.0. (You can download a free copy.) FWIW, I've probably been working on the Windows build side of things on and off for the past month or so, and we've only just reached a point where 32bit and 64bit Windows builds are compiling with all extension modules (bsddb, tcl/tk, ssl etc) and passing all tests (most work has gone into the x64 builds though, the 32-bit ones were already green on XP and below for 32bit). Using MinGW/gcc on Windows hasn't seen anywhere near so much attention, so, YMWV. In terms of my Windows-oriented priorities, they are as follows: - Get 3.0 32/64 Windows builds actually compiling successfully and then passing all tests (given that all build slaves for 3.0 are red that's not exactly a quick action). - Move on to the MSI installer improvements for 2.6/3.0, specifically with regards to the VCRT9 runtime and signing of the installer/binaries. - Maybe putting some cycles into Python builds on MinGW. To be honest though, the main motivation for doing that will be to demonstrate that a Python executable compiled with Visual Studio 2008 Professional with Profile Guided Optimisation will outperform a MinGW/gcc build ;-) Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Bill Janssen [EMAIL PROTECTED] Sent: 19 March 2008 20:02 To: python-dev@python.org Subject: [Python-Dev] how to build extensions for Windows? I've set up a Parallels virtual machine on my Mac, and have succeeded in getting Windows XP running in it! And I've installed MinGW, as well. Now I'd like to learn how to build the SSL module from source on Windows for Python 2.5.2. Is there any documentation on the process of building an extension from scratch that's appropriate for someone who doesn't know much about Windows? I'm looking for step-by-step. What about this? http://www.mingw.org/MinGWiki/index.php/Python%20extensions Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] trunk buildbot status
Quick update on the status of the trunk buildbots: Failing: [x86 gentoo trunk (Neal Norwitz)] This has been failing at the same point for the past couple of days now: test_sqlite command timed out: 1800 seconds without output, killing pid 15168 process killed by signal 9 program finished with exit code -1 None of the other buildbots seem to be encountering the same problem. Neal, got any idea what's going on with this one? [alpha True64 5.1 trunk (Neal Norwitz)] test_tarfile started failing recently (within the past few days) with CRC checks. See http://www.python.org/dev/buildbot/trunk/alpha%20Tru64%205.1%20trunk/builds/2712/step-test/0. Greg updated the test such that it prints out some more detail about the failure so we're waiting on that at the moment. [hppa Ubuntu trunk (Matthias Klose)] This has been consistently failing in test_socketserver for as long as I can remember: test_socketserver make: *** [buildbottest] Alarm clock program finished with exit code 2 I just updated that test such that it waits 20 seconds instead of 3 seconds at the end of the test if the server hasn't shutdown -- waiting for the test results of this still. [x86 XP trunk (Joseph Armbruster)] This box didn't survive the recent build changes, but I can't figure out why, none of the other Windows boxes encounter this error: The following error has occurred during XML parsing: File: C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj Line: 179 Column: 1 Error Message: Illegal qualified name character. The file 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' has failed to load. Can someone check a clean trunk build on a Windows system that *only* has Visual C++ Express 2008? The latest build system updates don't rely on any features of Visual Studio Professional, but the tools use a lot of common files, and perhaps a Service Pack needs to be applied or something. [amd64 XP trunk (Thomas Heller)] Builds fine, all tests pass except for test_largefile, which is failing as there's no more space left on the drive ;-) [x86 XP-4 trunk (David Bolen)] This is currently flagged as having failed test, but I don't think it's finished building since the finalised build updates, so hopefully the BSDDB errors in the last run will be resolved when it finished the latest build. [x86 FreeBSD 2 trunk (Jeroen Ruigrok van der Werven)] This is a FreeBSD 6.3-STABLE box (which switched to curses 5.6 from 5.2) -- there's been an ongoing thread with regards to why curses has started failing, Jeroen can probably provide more info on that. Either way I don't anticipate a quick fix for this particular slave, unfortuantely. Neal/Martin, I'd like to promote the following slaves to the stable list: [g4 osx.4] [x86 W2k8] [AMD64 W2k8] [ppc Debian unstable] [sparc Ubuntu] [sparc Debian] [PPC64 Debian] [S-390 Debian] [x86 XP-3] [amd64 XP] [x86 FreeBSD] [x86 FreeBSD 3] The trunk builds of these slaves have been the most reliable since I've been tracking. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] trunk buildbot status
I'd recommend cd'ing to your trunk root directory and running Tool\buildbot\build.bat from there -- it'll automatically check out all the dependencies and build via command line with vcbuild (building via Visual Studio usually always Does The Right Thing, command line builds often take a bit more coercing). From: Eric Smith [EMAIL PROTECTED] Sent: 19 March 2008 20:49 To: Trent Nelson Cc: python-dev@python.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: [Python-Dev] trunk buildbot status Trent Nelson wrote: > Quick update on the status of the trunk buildbots: > > [x86 XP trunk (Joseph Armbruster)] > This box didn't survive the recent build changes, but I can't figure out why, > none of the other Windows boxes encounter this error: > The following error has occurred during XML parsing: > File: C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj > Line: 179 > Column: 1 > Error Message: > Illegal qualified name character. > The file > 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' > has failed to load. > > Can someone check a clean trunk build on a Windows system that *only* has > Visual C++ Express 2008? The latest build system updates don't rely on any > features of Visual Studio Professional, but the tools use a lot of common > files, and perhaps a Service Pack needs to be applied or something. I just built the trunk on a Windows XP x86 box that only has Visual C++ Express 2008 installed. I got a bunch of errors with sqlite, tcl, db-4.4.20, and ssl, but the interpreter built and appears to run ok. But since I don't have bsddb installed, I don't think I'm executing the portion of the build process that you find failing. I don't have time to install bsddb tonight, but I can do that in about 24 hours if you still need me to. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] trunk buildbot status
Thanks Eric, very useful to know. I guess it's just that particular build slave... From: Eric Smith [EMAIL PROTECTED] Sent: 20 March 2008 02:55 To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] trunk buildbot status Trent Nelson wrote: > I'd recommend cd'ing to your trunk root directory and running > Tool\buildbot\build.bat from there -- it'll automatically check out all the > dependencies and build via command line with vcbuild (building via Visual > Studio usually always Does The Right Thing, command line builds often take a > bit more coercing). Okay, that's extremely helpful. With that (and installing nasmw.exe), a trunk checkout builds correctly and passes all tests (although skipping test_tcl) on my box. As I said, it's XP x86 with 2008 Express Edition only. Let me know if I can provide any other information. Unfortunately I don't have access to this box during the work day (EDT), and I'm leaving for vacation tomorrow (Friday). But I'll help as best I can. Eric. > > > From: Eric Smith [EMAIL PROTECTED] > Sent: 19 March 2008 20:49 > To: Trent Nelson > Cc: python-dev@python.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL > PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: [Python-Dev] trunk buildbot status > > Trent Nelson wrote: >> Quick update on the status of the trunk buildbots: >> >> [x86 XP trunk (Joseph Armbruster)] >> This box didn't survive the recent build changes, but I can't figure out >> why, none of the other Windows boxes encounter this error: >> The following error has occurred during XML parsing: >> File: >> C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj >> Line: 179 >> Column: 1 >> Error Message: >> Illegal qualified name character. >> The file >> 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' >> has failed to load. >> >> Can someone check a clean trunk build on a Windows system that *only* has >> Visual C++ Express 2008? The latest build system updates don't rely on any >> features of Visual Studio Professional, but the tools use a lot of common >> files, and perhaps a Service Pack needs to be applied or something. > > I just built the trunk on a Windows XP x86 box that only has Visual C++ > Express 2008 installed. I got a bunch of errors with sqlite, tcl, > db-4.4.20, and ssl, but the interpreter built and appears to run ok. > > But since I don't have bsddb installed, I don't think I'm executing the > portion of the build process that you find failing. > > I don't have time to install bsddb tonight, but I can do that in about > 24 hours if you still need me to. > > Eric. > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
> def rename_and_remove (filename): > os.rename (filename, filename + ".deleted") > os.remove (filename + ".deleted") > Isn't this still going to run into problems when the rename > fails because the earlier tests remove still left the .deleted > file around due to some other running desktop search service > that now has the .deleted file open? I haven't looked into all the various places the tests write temp files to, but if we could localise everything to a common root directory, i.e. %TEMP%\python-regrtest, we could then attempt to blow this away at the start of regrtest.py before any tests run, and refuse to run if this fails. This would be in combination with the unlinking/renaming approach discussed. This approach seems like it would cover all bases a bit more effectively. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No time for svn merge
> > Yes, that's all I meant: make it the committer's job > > to merge or block as appropriate. I just wasn't sure if > > there was some reason that this would be difficult or > > undesirable. > > Ah, yes. It is indeed difficult or undesirable, or was > so in the past: Some committers don't care (didn't care) > at all about 3k. They would have to setup sandboxes, > learn what the nature of changes is, and invest some > regular time into forward-porting. Is this *really* the case still? Who are these rogue committers? ;-) I think holding a developer accountable for merging or blocking to py3k when they commit to trunk is a great idea. Who better to pass judgement on such an activity than the person closest to it? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows
> Further, I > assert that there are a greater number of build tools which do not support > cross-compilation, but will build natively on x64 and expect 'PCBuild' > to have libraries they can link with to create an x64 binary. I'm with Martin on this one as well I think. If I understand correctly, you're proposing: PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86 And what we've currently got is: PCbuild/- always x86 PCbuild/amd64 - always x64 I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] the release gods are angry at python
> > In the py3k branch I've assigned the audio resource to the winsound > > tests. Only regrtest.py -uall or -uaudio runs the winsound test. > Reason: > > the test sound was freaking out my poor cat. :/ > > I feel with your cat ;-). > This would not help on the buildbot since it runs 'rt.bat -d -q -uall - > rw'. I feel for the poor NOC engineers at my colo that freak out when some random server in a farm of thousands starts making bizarre sounds. I detest test_winsound. There are so many corner cases you need to account for that makes the test pointless as you end up wrapping everything in except: pass blocks. Does the system have a legacy beep driver? Is it enabled? Is it disabled? Is there a sound card? Is it enabled or disabled? Pah! +1 to removing audio out of -uall, if only for the sake of cats, erroneously red buildbots, and poor ServerCentral NOC engineers. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No time for svn merge
Christian Heimes [mailto:[EMAIL PROTECTED]: > Trent Nelson schrieb: > > I think holding a developer accountable for merging or blocking to > py3k when they commit to trunk is a great idea. Who better to pass > judgement on such an activity than the person closest to it? > > Blocking a revision makes my job as The Merger easier. > > I'm not so sure about the merging part. It takes some experience with > the Python 3.0 code base to know the subtle differences in the C API. > Most merges are straight forward for me. If you enforce the forward > merging on every developer it may slow down development. > Each regular merge takes me about 45 minutes of computer time but less > than 15 supervisor time. The computer time is mostly compile and test > time in the background. If everybody merges her own code to 3.0 it > still takes *everybody* about 10 minutes of time and 45 minutes of computer > time. Ah, right, I wasn't thinking about the implication of code affecting the C base for some reason, but that's entirely reasonable. Perhaps each developer should be accountable for either: a) blocking b) merging, if they're able to do so c) if they're unable to merge, replying to the relevant python-checkins@ e-mail indicating that they're unable to handle trunk -> py3k for whatever reason (e.g. not familiar with py3k code base) Other developers could then pitch in and help merge if someone requests it via e-mail. I'd think that would make The Merger's life easier. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
Looking into some of the recent Windows buildbot failures, I see things like
this:
sqlite3 : error PRJ0008 : Could not delete file
'c:\buildbot\trunk.heller-windows-amd64\build\PCbuild\amd64\sqlite3_d.dll'.
build-amd64.bat doesn't go through the kill_python.c hoopla, so I figure the
above error is being caused by the fact that an erroneous/stalled python_d.exe
from a previous run is still open. I was looking at modifying kill_python.c to
accept an 'x64' argument if we want to kill amd64\python_d.exe instead of the
usual 32-bit exe, however, this caught my attention:
if ((strstr(path, "pcbuild\\python_d.exe") != NULL) ||
(strstr(path, "\\build\\python.exe") != NULL)) {
printf("Terminating %s (pid %d)\n", path, pids[i]);
if (!TerminateProcess(hProcess, 1)) {
printf("Termination failed: %d\n", GetLastError());
return 1;
}
return 0;
That'll kill the first python_d.exe instance it finds matching the given path;
given that our buildbots run trunk/release25-maint/py3k in parallel, it seems
as though it wouldn't be hard for us to get into a situation where
kill_python.exe ends up killing the wrong python_d.exe (i.e. trunk checkin,
trunk builds, starts testing, py3k checkin, calls kill_python.exe, kills
trunk's python_d.exe that was in the middle of testing).
That can't be helping our cause, unless I'm missing something... Unless anyone
advises otherwise, I'll start on a patch.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
> > That'll kill the first python_d.exe instance it finds matching the > > given path; given that our buildbots run trunk/release25-maint/py3k > > in parallel > > That's actually not a given: we currently *don't* run multiple builds > simultaneously on the same slave. I thought the slave lock only applied per branch, not per host? > > Unless anyone advises otherwise, I'll start on a patch. > > If you can make it less error-prone, sure, go ahead. Spent a bit of time on it this evening; as it happens, in order to enumerate 64-bit processes, you need to be a 64-bit process yourself. As it's much easier managing 32-bit vs. x64 binaries when they're a .vcproj part of pcbuild.sln, I'm looking into adding kill_python as a .vcproj and configure the solution such that it builds and runs this before any other project. That'll automatically take care of choosing the right version to run depending on whether 'Win32' or 'x64' is selected as the platform. It'll also simplify the verification logic that checks if it's the right python_d.exe -- the path of the .exe needs to match the path of the running kill_python.exe. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
[Disclaimer: thought dump e-mail, signal to noise ratio may be subpar.] Sounds like you're at least making steps forward in the right direction, despite the activity probably being quite disheartening. Based on what you've said below and the rest of the conversation, here are my thoughts for an approach: 1. For a given python[_d].exe, always use the same test directory, but hash it against the entire python process path such that it's unique only for a given python instance. 2. Make sure every time a test wants a temp file, it gets a unique one in this directory. Sounds like your TESTFN modification would take care of this for those tests using TESTFN; if TESTFN is the preferred approach then any other tests using tempfile or hardcoding file names would then be changed to use this instead. 3. In order to address tests that either call the test_support methods for removing files/directories, or those that call os.(unlink|remove), do what ever needs to be done to make these no-ops on Windows if an error occurs. 4. At the end of the regrtest.py run, create a suspended arbitrary process (i.e. explorer.exe), hijack the main thread context of the process and inject a routine (i.e. overwrite the thread context's instruction pointers) that takes care of removing the temporary directory that was used for testing -- patiently re-trying if any failures occur until all rogue processes also accessing the file(s) stop doing so. Resume the thread before exiting python. Heh. Sounds crazy? It did to me as well, until I came across http://www.catch22.net/tuts/selfdel.asp, which documents the approach. It's not particularly necessary in our case, we could simply spawn another python process at the end of regrtest.py that patiently attempts to remove the test directory we just used when the python process that was executing regrtest.py exits. We could then modify regrtest.py such that it will use the same approach if the hashed test directory already exists at the start of a run and it can't remove it via os.unlink. If we *still* run into issues here on the buildbots, say if regrtest.py blocks on our helper process, which for the life of it can't remove some/all of the test files -- it'd be interesting to write something that grok's all open handles for all processes and attempts to figure out what it is that keeps these files open -- i.e. same thing that procexp.exe does when you search for a handle. Or, keeping it simple, rather than a separate process and hashed test directory based on python process path, just have a common directory, i.e. %TEMP%\python-regrtest, and use an incrementing sequence number for each test run's test directory, i.e. if there are directories 001, 002 and 003 in the temp dir, when regrtest.py starts, it'll try delete all of these -- if it can't (which is what we'd want if another test is already running), it adds 1 to the highest numbered directory it couldn't delete. Guess it all depends on how much effort we want to put into cleaning up our test directory really -- just ensuring tests get a clean area and unique file names each run is the easy part. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Tim Golden [EMAIL PROTECTED] Sent: 03 April 2008 09:39 To: Python-Dev Subject: Re: [Python-Dev] fixing tests on windows [re tests which fail because something's holding a file open with SHARE_DELETE] Well I've tried my best, but I can't come up with a solution which guarantees to overcome this obstacle. I set up a fairly aggressive directory watcher which, when it sees a test file being created, takes a SHARE_DELETE handle on it and releases it immediately. (Guessing that this is what the other apps are doing). Running the test suite from HEAD, this generates all manner of access-denied type errors as per the original output. I used tempfile to generate a random TESTFN in the current directory rather than the static @test. And implemented the rename-shim discussed previously, renaming to a different random name, courtesy of mktemp. With those in place, most tests run without error. But I'm still getting errors in the same sort of areas which Steven B originally reported. The one error I can't see a way round is the access denied (which manifests as Permission Error) which is the documented result of attempting to open a file with a pending delete -- ie the delete succeeded but hasn't completed yet. An additional complication is that there are hundreds of instances throughout the tests where the test simply calls os.unlink/os.remove to undo the test file. To have some more robust central deletion I had to go through and update 68 tests. I'll keep trying, but in the current state I'm not convinced the situation's improved enough for me to bother uploading a patch. TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/m
Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
Committed new version of kill_python to trunk in r62129. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 02 April 2008 14:39 To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause > That'll kill the first python_d.exe instance it finds matching the > given path; given that our buildbots run trunk/release25-maint/py3k > in parallel That's actually not a given: we currently *don't* run multiple builds simultaneously on the same slave. > Unless anyone advises otherwise, I'll start on a patch. If you can make it less error-prone, sure, go ahead. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I started looking into this:
http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0
Pertinent part:
test_asyncore
test_asynchat
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
I tried to replicate it on the buildbot in order to debug, which, surprisingly,
I could do consistently by just running rt.bat -q -d -uall test_asynchat. As
the log above indicates, the python process becomes completely and utterly
wedged, to the point that I can't even attach a remote debugger and step into
it.
Digging further, I noticed that if I ran the following code in two different
python consoles, EADDRINUSE was *NOT* being raised by socket.bind():
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 54322))
However, take out the setsockopt line, and wallah, the second s.bind() will
raise EADDRINUSE, as expected. This manifests into a really bizarre issue with
test_asynchat in particualr, as subsequent sock.accept() calls on the socket
put python into the uber wedged state (can't even ctrl-c out at the console,
need to kill the process directly).
Have to leave the office and head home so I don't have any more time to look at
it tonight -- just wanted to post here for others to mull over.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
I agree with Tim, you can jump through as many hoops as you want (setting directories private, using %TEMP% exclusively, etc), but I doubt anything is going to change the behaviour of things like virus scanners, for example. Tim, let me know if you need help with anything, perhaps we could set up a temporary branch outside of trunk to play around with various approaches to see what works best. This'll ensure we don't adversely affect the main buildbots, but also give us the option to get different Windows boxes to build our test branch on demand. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Tim Golden [EMAIL PROTECTED] Sent: 04 April 2008 04:04 Cc: python-dev@python.org Subject: Re: [Python-Dev] fixing tests on windows Terry Reedy wrote: > If the testdir disallows the search indexer, then there should be no need > to disable Windows Search Service. If privatizing the dir kept other > programs out, then likewise. > > | Or were you suggesting that there is some programmatic way for the > | test suite to create directories that disallow the Search Service, > | etc.? > > I suspect, but do not know, that the dialog box effects changes through > user-programmable interfaces. So while I would start with manual changes > to see if that solves the problem, I presume there must be a system call > for changing dir attributes. The problem is, I think, that it isn't just the Indexing Service which generates this issue. TortoiseSVN is well known for doing the same thing, and there could be now and potentially will be in the future other programs. I don't think that hunting down and turning off their interference case by case is a viable solution in the long-term. Although it would obviously be a way forward in the short term, _faute de mieux_. [Tony Nelson] > I'd think that files and directories created in the TEMP > directory would normally not be indexed on any OS, including > MSWindows. But this is just a guess. I'm inclined to think you're right. And a first pass I had at producing a solution simply used tempfile to do everything. Unfortunately that's far more invasive than I was really comfortable with at the time: at the very least, you have to patch several tests which fail if there's an (escaped) backslash in the path. However, it's clear that my attempt to cause the minimum damage isn't enough to clear the problem 100%. So I think the next move is indeed to turn test_support.TESTFN into a function (in some way) which generates a unique tempfile reference, possibly with a context manager to clean up. Or something. The complication is that, while most test simply want a handy file to exist or be written to, and don't really care what happens afterwards, some tests are testing the very mechanism of creating/deleting a file etc. So a wholesale replacement isn't necessarily straightforward. On we go. TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
> I don't like the part where the solution kills the Python process during > a rebuild. It's too surprising for the user. Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. Trent. From: Christian Heimes [EMAIL PROTECTED] Sent: 04 April 2008 09:25 To: [EMAIL PROTECTED]; Trent Nelson Subject: Re: r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... trent.nelson schrieb: > Author: trent.nelson > Date: Thu Apr 3 20:27:06 2008 > New Revision: 62129 > > Added: >python/trunk/PCbuild/kill_python.c (contents, props changed) >python/trunk/PCbuild/kill_python.vcproj > Removed: >python/trunk/Tools/buildbot/Makefile >python/trunk/Tools/buildbot/kill_python.bat >python/trunk/Tools/buildbot/kill_python.c >python/trunk/Tools/buildbot/kill_python.mak > Modified: >python/trunk/PCbuild/debug.vsprops >python/trunk/PCbuild/pcbuild.sln >python/trunk/PCbuild/pythoncore.vcproj >python/trunk/PCbuild/release.vsprops >python/trunk/Tools/buildbot/build-amd64.bat >python/trunk/Tools/buildbot/build.bat >python/trunk/Tools/buildbot/buildmsi.bat > Log: > Reimplement kill_python. The existing version had a number of flaws, namely, > it didn't work for x64 and it wasn't precise about which python_d.exe it was > killing -- it just killed the first one it came across that happened to have > 'pcbuild\python_d.exe' or 'build\python_d.exe' in it's path. The new version > has been rewritten from the ground up and now lives in PCbuild, instead of > Tools\buildbot, and it has also been incorporated into the Visual Studio > solution (pcbuild.sln) as 'kill_python'. The solution has also been altered > such that kill_python is called where necessary in the build process in order > to prevent any linking errors due to open file locks. In lieu of this, all > of the existing bits and pieces in Tools\buildbot that called out to > kill_python at various points have also be > en removed as they are now obsolete. Tested on both Win32 and x64. I don't like the part where the solution kills the Python process during a rebuild. It's too surprising for the user. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I've raised issue 2550 to track this problem. I've also provided a patch on
the tracker to test_socket.py that reproduces the issue. Anyone mind if I
commit this to trunk? I'd like to observe if any other platforms exhibit
different behaviour via buildbots. It'll cause all the Windows slaves to fail
on test_socket though. (I can revert it once I've seen how the buildbots
behave until I can come up with an actual patch for Windows that fixes the
issue.)
http://bugs.python.org/issue2550
http://bugs.python.org/file9939/test_socket.py.patch
Trent.
From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL
PROTECTED]
Sent: 03 April 2008 22:40
To: python-dev@python.org
Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows
vs Unix (or why test_asynchat is sometimes dying on Windows)
I started looking into this:
http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0
Pertinent part:
test_asyncore
test_asynchat
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
I tried to replicate it on the buildbot in order to debug, which, surprisingly,
I could do consistently by just running rt.bat -q -d -uall test_asynchat. As
the log above indicates, the python process becomes completely and utterly
wedged, to the point that I can't even attach a remote debugger and step into
it.
Digging further, I noticed that if I ran the following code in two different
python consoles, EADDRINUSE was *NOT* being raised by socket.bind():
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 54322))
However, take out the setsockopt line, and wallah, the second s.bind() will
raise EADDRINUSE, as expected. This manifests into a really bizarre issue with
test_asynchat in particualr, as subsequent sock.accept() calls on the socket
put python into the uber wedged state (can't even ctrl-c out at the console,
need to kill the process directly).
Have to leave the office and head home so I don't have any more time to look at
it tonight -- just wanted to post here for others to mull over.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
Ok, I'll change the approach this weekend. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
Interesting results! I committed the patch to test_socket.py in r62152. I was expecting all other platforms except for Windows to behave consistently (i.e. pass). That is, given the following: import socket host = '127.0.0.1' sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind((host, 0)) port = sock.getsockname()[1] sock.close() del sock sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock1.bind((host, port)) sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock2.bind((host, port)) the second bind should fail with EADDRINUSE, at least according to the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed): "With TCP, we are never able to start multiple servers that bind the same IP address and same port: a completely duplicate binding. That is, we cannot start one server that binds 198.69.10.2 port 80 and start another that also binds 198.69.10.2 port 80, even if we set the SO_REUSEADDR socket option for the second server." The results: both Windows *and* Linux fail the patched test; none of the buildbots for either platform encountered an EADDRINUSE socket.error after the second bind(). FreeBSD, OS X, Solaris and Tru64 pass the test -- EADDRINUSE is raised on the second bind. (Interesting that all the ones that passed have a BSD lineage.) I've just reverted the test in r62156 as planned. The real issue now is that there are tests that are calling test_support.bind_socket() with the assumption that the port returned by this method is 'unbound', when in fact, the current implementation can't guarantee this: def bind_port(sock, host='', preferred_port=54321): for port in [preferred_port, 9907, 10243, 32999, 0]: try: sock.bind((host, port)) if port == 0: port = sock.getsockname()[1] return port except socket.error, (err, msg): if err != errno.EADDRINUSE: raise print >>sys.__stderr__, \ ' WARNING: failed to listen on port %d, trying another' % port This logic is only correct for platforms other than Windows and Linux. I haven't looked into all the networking test cases that rely on bind_port(), but I would think an implementation such as this would be much more reliable than what we've got for returning an unused port: def bind_port(sock, host='127.0.0.1', *args): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((host, 0)) port = s.getsockname()[1] s.close() del s sock.bind((host, port)) return port Actually, FWIW, I just ran a full regrtest.py against trunk on Win32 with this change in place and all the tests still pass. Thoughts? Trent. ____ From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 04 April 2008 17:07 To: python-dev@python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I've raised issue 2550 to track this problem. I've also provided a patch on the tracker to test_socket.py that reproduces the issue. Anyone mind if I commit this to trunk? I'd like to observe if any other platforms exhibit different behaviour via buildbots. It'll cause all the Windows slaves to fail on test_socket though. (I can revert it once I've seen how the buildbots behave until I can come up with an actual patch for Windows that fixes the issue.) http://bugs.python.org/issue2550 http://bugs.python.org/file9939/test_socket.py.patch Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 03 April 2008 22:40 To: python-dev@python.org Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0 Pertinent part: test_asyncore test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asy
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
> >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
Fixed in r62193. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 04 April 2008 20:41 To: "Martin v. Löwis" Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... Ok, I'll change the approach this weekend. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I've attached a patch (http://bugs.python.org/file9966/trunk.2550.patch) to issue 2550 that addresses the original problem here: test_support.bind_port() potentially returning ports that have already been bound to. The patch updates the tests that relied on this method, such that they call it with the new calling convention (test_ftplib, test_httplib, test_socket, test_ssl_socket, test_asynchat, test_telnetlib). Any objections to the patch? Would like to commit it sooner rather than later, as it'll fix my buildbots from wedging on test_asynchat at the very least. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 05 April 2008 18:22 To: Jean-Paul Calderone; python-dev@python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) > >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
