OpenBSD, Apache and Python
Hi! I'm wanting to use openbsd to run a webserver because of its high security. The website is written in python. Before I was using linux with apache 2 and mod_python and this worked well. My problem is that mod_python 3.x does not work on apache 1 which is what comes with openbsd (and has been hardened buy the openbsd guys). One option is to use mod_python 2.x, but this has not seen any development since 2004. I'm now looking at using fastcgi to run my python. This looks like it's going to work fine, though I will be using a fcgi interface such as http://jonpy.sourceforge.net/. There does not seem to be any official fastcgi module for python. Should I be worried about the stability and security of the python fcgi interfaces? Is there something better (more mature / battle tested)? Are there any other options available to me - considering that security is very important? Thanks for any help, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Best FCGI handler
Hi! Does anyone have any recommendations for FGCI interfaces for python? For example jonpy? Thanks, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Question about encoding, I need a clue ...
Hi, I use Mac OSX for development but deploy on a Linux server. (Platform details provided below). When the locale is set to FR_CA, I am not able to display a u circumflex consistently across the two machines even though the default encoding is set to "ascii" on both machines. Specifically, calendar.month_name[8] returns a ? (question mark) on the Linux server whereas it displays properly on the Mac OSX system. However, if I take the result from calendar.month_name[8] and run it through the following function unicode(calendar.month_name[8],"latin1") ... then the u circumflex displays correctly on the Linux server but does not display correctly on my Mac. Of course, I could work around this problem with a relatively simple if statement but these issues are going to show up all over my application so even a simple if statement will start to get cumbersome. I guess what it boils down to is that I would like to get a better handle on what is going on so that I will know how best to work through future encoding issues. Thanks in advance for any advice. Here are the specifics of my problem. On my Mac: Python 2.6.7 (r267:88850, Jul 30 2011, 23:46:53) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin >>> import locale >>> locale.setlocale(locale.LC_ALL,'fr_CA') 'fr_CA' >>> import sys >>> sys.getdefaultencoding() 'ascii' >>> import calendar >>> calendar.month_name[8] 'ao\xc3\xbbt' >>> print calendar.month_name[8] août >>> print unicode(calendar.month_name[8],"latin1") août On the linux server: uname -a Linux alhena 2.6.32.8-grsec-2.1.14-modsign-xeon-64 #2 SMP Sat Mar 13 00:42:43 PST 2010 x86_64 GNU/Linux Python 2.5.2 (r252:60911, Jan 24 2010, 17:44:40) [GCC 4.3.2] on linux2 >>> import locale,sys,calendar >>> locale.setlocale(locale.LC_ALL,'fr_CA') 'fr_CA' >>> sys.getdefaultencoding() 'ascii' >>> calendar.month_name[8] 'ao\xfbt' >>> print calendar.month_name[8] ao?t >>> print unicode(calendar.month_name[8],"latin1") août -- http://mail.python.org/mailman/listinfo/python-list
Read-write lock for Python
Hi all, I currently find myself needing a Python read-write lock. I note that there is none in the standard library, but googling "python read-write lock" quickly produced 6 different competing examples, including two languishing patch proposals for the standard library. I can always pick a random one and hope for the best, but I was hoping someone here might have a tip for one that has been used and debugged and is likely to work. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Read-write lock for Python
On Thu, Apr 28, 2011 at 4:28 PM, Jean-Michel Pichavant wrote: > Geoff Bache wrote: >> >> Hi all, >> >> I currently find myself needing a Python read-write lock. I note that >> there is none in the standard library, but googling "python read-write >> lock" quickly produced 6 different competing examples, including two >> languishing patch proposals for the standard library. >> >> I can always pick a random one and hope for the best, but I was hoping >> someone here might have a tip for one that has been used and debugged >> and is likely to work. >> >> Regards, >> Geoff Bache >> > > What about > > http://docs.python.org/library/threading.html#lock-objects Those aren't read-write locks. They are basic locks, which don't distinguish between readers and writers. I need to be able to lock between reader and writer operations, without readers locking each other out. /Geoff -- http://mail.python.org/mailman/listinfo/python-list
Re: Read-write lock for Python
On Thu, Apr 28, 2011 at 7:53 PM, D'Arcy J.M. Cain wrote: > On Thu, 28 Apr 2011 19:14:45 +0200 > Geoff Bache wrote: >> On Thu, Apr 28, 2011 at 4:28 PM, Jean-Michel Pichavant >> wrote: >> > What about >> > >> > http://docs.python.org/library/threading.html#lock-objects >> >> Those aren't read-write locks. They are basic locks, which don't >> distinguish between readers and writers. I need to be able to lock >> between reader and writer operations, without readers locking each >> other out. > > There really isn't any such thing as "read-write" locks. Did you google it? I even provided the exact search terms to use in my initial posting. It takes you straight to two standard library patch proposals and four other solutions, all labelled this way (sometimes "reader-writer locks" admittedly). That doesn't happen if it isn't a standard problem. I'm not looking for general advice on how to solve some specific problem. I'm asking if anyone knows anything about the relative merits of the above 6 solutions. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Read-write lock for Python
On Fri, Apr 29, 2011 at 12:38 AM, Ryan Kelly wrote: > On Thu, 2011-04-28 at 07:02 -0700, Geoff Bache wrote: >> Hi all, >> >> I currently find myself needing a Python read-write lock. I note that >> there is none in the standard library, but googling "python read-write >> lock" quickly produced 6 different competing examples, including two >> languishing patch proposals for the standard library. >> >> I can always pick a random one and hope for the best, but I was hoping >> someone here might have a tip for one that has been used and debugged >> and is likely to work. > > I wrote and have used the "SHLock" class in threading2 which should do > what you need. Don't know about "likely to work" but if it doesn't, I'd > like to hear about it so I can fix it :-) That's good enough for me :) Thanks, I'll give it a try. /Geoff -- http://mail.python.org/mailman/listinfo/python-list
Problems with imports on multiple threads, with embedded Python
Hi all, I have a multithreaded application using an embedded Python 3.6.4 (upgraded from 3.6.2 today in the hope that the problem was now solved: it doesn't seem to be). The standard library is in a zip file. So long as only one thread is running Python at a time it seems to work fine. But there seems to be a problem with the module importing when several Python threads are active. I get a variety of errors indeterministically, usually indicating that some symbol hasn't been imported. This occurs both in my own code and in the standard library. The most frequent is probably this line: dt = datetime.strptime(dtStr, fromFmt) which produces AttributeError: module '_strptime' has no attribute '_strptime_datetime' at random. Does anyone have any insight into this problem? Regards, Geoff Bache -- https://mail.python.org/mailman/listinfo/python-list
Re: Problems with imports on multiple threads, with embedded Python
On Thursday, 21 December 2017 00:33:54 UTC+1, Lawrence D’Oliveiro wrote: > On Thursday, December 21, 2017 at 5:13:33 AM UTC+13, [email protected] wrote: > > > > I have a multithreaded application using an embedded Python 3.6.4 ... > > Avoid multithreading if you can. Is your application CPU-bound? If not, you > are getting no benefit from it. Unfortunately that's not an option. As I say, Python is embedded and is only a part of what the application does. The multithreading is not there for the Python part but is a well-established fact in the application. -- https://mail.python.org/mailman/listinfo/python-list
Access violation in Python garbage collector (visit_decref) - how to debug?
Hi all, We are running Python embedded in our C++ product and are now experiencing crashes (access violation reading 0xff on Windows) in the Python garbage collector. We got this on Python 3.6.4 originally, but I can reproduce it with both Python 3.6.8 and Python 3.7.4. The chances of producing a minimal example that reproduces it reliably are currently small I would say. All attempts to simplify the set up seem to cause the problem to go away. Indeed I can only reproduce it by sending a fairly large amount of data 2 or 3 times to our server - sending either half of the data does not reproduce it. I note that there have been a lot of crashes in this method reported over the years, and that efforts are underway in Python 3.8 to make it easier to find issues like this, i.e. https://bugs.python.org/issue36389. I won't be able to use Python 3.8 yet but I could potentially use it as a debugging tool I guess, if there was documentation available how to use these new features. I tried setting PYTHONMALLOC=debug but didn't get any more information that way. The python code is calling "json.loads" on a fairly large json unicode string with plenty of Swedish characters in it. It works at least once, and then crashes on the second or third send of the same data. I paste the stacktrace from Python 3.7.4 below. Please let me know how I can debug this further. I'm using Visual Studio 2017 on Windows 10 if that helps. Regards, Geoff Bache > [Inline Frame] python37.dll!visit_decref(_object *) Line 271 C python37.dll!dict_traverse(_object * op, int(*)(_object *, void *) visit, void * arg) Line 2982 C python37.dll!subtract_refs(_gc_head * containers) Line 296 C python37.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 861 C python37.dll!collect_with_callback(int generation) Line 1029 C python37.dll!collect_generations() Line 1051 C [Inline Frame] python37.dll!_PyObject_GC_Alloc(int) Line 1708 C [Inline Frame] python37.dll!_PyObject_GC_Malloc(unsigned __int64 basicsize) Line 1718 C python37.dll!_PyObject_GC_New(_typeobject * tp) Line 1730 C python37.dll!new_dict(_dictkeysobject * keys, _object * * values) Line 584 C python37.dll!PyDict_New() Line 678 C python37.dll!_parse_object_unicode(_PyScannerObject * s, _object * pystr, __int64 idx, __int64 * next_idx_ptr) Line 730 C python37.dll!scan_once_unicode(_PyScannerObject * s, _object * pystr, __int64 idx, __int64 * next_idx_ptr) Line 1162 C python37.dll!_parse_array_unicode(_PyScannerObject * s, _object * pystr, __int64 idx, __int64 * next_idx_ptr) Line 869 C python37.dll!scan_once_unicode(_PyScannerObject * s, _object * pystr, __int64 idx, __int64 * next_idx_ptr) Line 1101 C python37.dll!scanner_call(_object * self, _object * args, _object * kwds) Line 1188 C python37.dll!_PyObject_FastCallKeywords(_object * callable, _object * const * stack, __int64 nargs, _object * kwnames) Line 199 C python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object * kwnames) Line 4619 C python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 3095 C [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 3930 C [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack, _object * const *) Line 433 C python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object * kwnames) Line 4621 C python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 3140 C [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 3930 C [Inline Frame] python37.dll!_PyFunction_FastCallKeywords(_object * stack, _object * const *) Line 433 C python37.dll!call_function(_object * * * pp_stack, __int64 oparg, _object * kwnames) Line 4621 C python37.dll!_PyEval_EvalFrameDefault(_frame * f, int throwflag) Line 3111 C [Inline Frame] python37.dll!PyEval_EvalFrameEx(_frame *) Line 547 C python37.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 3930 C
Re: Access violation in Python garbage collector (visit_decref) - how to debug?
I've so far only tried within my application, but I'm aware it would be easier if I could reproduce it outside. Even simplifying the context within the application has proved difficult though, so I suspect this will be hard. But I can try a bit more... The file isn't "large" by production standards, only by test data standards. It's about 500kb and not at all deeply nested, basically a long list of dictionaries. But I don't seem to be able to reduce it further either. /Geoff On Fri, Oct 4, 2019 at 9:53 PM Chris Angelico wrote: > On Sat, Oct 5, 2019 at 5:38 AM Geoff Bache wrote: > > > > Hi all, > > > > We are running Python embedded in our C++ product and are now > experiencing > > crashes (access violation reading 0xff on Windows) in the Python > > garbage collector. > > > > We got this on Python 3.6.4 originally, but I can reproduce it with both > > Python 3.6.8 and Python 3.7.4. > > > > The chances of producing a minimal example that reproduces it reliably > are > > currently small I would say. All attempts to simplify the set up seem to > > cause the problem to go away. > > Indeed I can only reproduce it by sending a fairly large amount of data 2 > > or 3 times to our server - sending either half of the data does not > > reproduce it. > > Have you tried to reproduce the issue outside of your application? > Even if it means creating a gigantic Python script with a whopping > triple-quoted string for the input data, it'd be helpful to try this. > If you CAN repro the problem, it'd be way easier to diagnose (since we > don't need your code, just your test case); and if you CAN'T, it may > mean an issue with the embedding aspects. > > What's your definition of "fairly large" here? How many > kilobytes/megabytes/gigabytes, and how deeply nested is the JSON > object? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Access violation in Python garbage collector (visit_decref) - how to debug?
It's possible. Our embedding code is fairly simple and we've tried to
encapsulate all the DECREFing in resource objects, i.e. that do DECREF in
their destructor when they go out of scope. We hoped this would minimise
this sort of problem.
The C++ calling code essentially tries to call progwrapper.prog with a
dictionary as arguments, and looks like
GilStateHolder pyStateHolder;
PyRefHolder module(PyImport_ImportModule("progwrapper"));
if (module._ref)
{
PyXRefHolder func(PyObject_GetAttrString(module._ref, "prog"));
PyXRefHolder pyArgs(PyTuple_New(1));
PyXRefHolder pyDict(convertDictForPython(dictIn));
PyTuple_SetItem(pyArgs._ref, 0, pyDict._ref);
PyRefHolder pyRet(PyObject_CallObject(func._ref, pyArgs._ref));
if (pyRet._ref != NULL)
{
addPythonValuesToDict(pyRet._ref, theDict);
...
where GilStateHolder, PyRefHolder etc are such resource objects as
described above, wrapping round the PyObject pointers etc.
/Geoff
On Fri, Oct 4, 2019 at 9:56 PM MRAB wrote:
> On 2019-10-04 20:32, Geoff Bache wrote:
> > Hi all,
> >
> > We are running Python embedded in our C++ product and are now
> experiencing
> > crashes (access violation reading 0xff on Windows) in the Python
> > garbage collector.
> >
> > We got this on Python 3.6.4 originally, but I can reproduce it with both
> > Python 3.6.8 and Python 3.7.4.
> >
> > The chances of producing a minimal example that reproduces it reliably
> are
> > currently small I would say. All attempts to simplify the set up seem to
> > cause the problem to go away.
> > Indeed I can only reproduce it by sending a fairly large amount of data 2
> > or 3 times to our server - sending either half of the data does not
> > reproduce it.
> >
> [snip]
> In my experience, it's most likely caused by incorrect refcounting,
> DECREFing too many times.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Access violation in Python garbage collector (visit_decref) - how to debug?
Yes, this is hard, that's why I'm here :) I've enabled the equivalent tools to valgrind in Visual Studio, and tried setting PYTHONMALLOC=debug, but neither of those seem to be showing anything either. I don't really know what else to try in this direction. /Geoff On Sat, Oct 5, 2019 at 7:22 AM dieter wrote: > Geoff Bache writes: > > ... > > We are running Python embedded in our C++ product and are now > experiencing > > crashes (access violation reading 0xff on Windows) in the Python > > garbage collector. > > Errors like this are very difficult to analyse. The main reason: > the memory corruption is likely far away from the point when > it is finally detected (by an access violation in your case). > > Python can be built in a special way to add marks to > its allocated memory blocks and verify their validity. > This increases the chance to detect a memory corruption earlier > and thereby facilitates the detection of the root cause. > > There are tools for the analysis of memory management problems > (e.g. "valgrind", though this may be for Linux). In my > experience, even with those tools, the analysis is very difficult. > > I have several times successfully analysed memory corruption > problems. In those cases, I have been lucky that the corruption > was reproducible and affected typically the same address. > Thus, I could put a (hardware) memory breakpoint at this address > stopping the program as soon as this address was written and > then analyse the state in the debugger. This way, I could detect > precisely which code caused the corruption. However, > this was quite a long time ago; nowadays, modern operating systems > employ address randomization thus reducing significantly that > the corruption affects the same address (which may mean that > you need to deactivate address randomization to get a better chance > for this kind of analysis. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Access violation in Python garbage collector (visit_decref) - how to debug?
Thankyou!!!
That was it. Saved me a lot of time debugging no doubt and potential nasty
problems at customer sites etc.
I must admit I had just pattern-matched with these functions but had no
idea that some of them stole references. I still don't really understand
why, the docs say it is a "common idiom", but not common enough to be
familiar to me and tricky to be inconsistent. But now I know, and the crash
is gone :)
Regards and many thanks again,
/Geoff
On Mon, Oct 7, 2019 at 7:32 PM MRAB wrote:
> On 2019-10-07 08:04, Geoff Bache wrote:
> > It's possible. Our embedding code is fairly simple and we've tried to
> > encapsulate all the DECREFing in resource objects, i.e. that do DECREF
> > in their destructor when they go out of scope. We hoped this would
> > minimise this sort of problem.
> > The C++ calling code essentially tries to call progwrapper.prog with a
> > dictionary as arguments, and looks like
> >
> > GilStateHolder pyStateHolder;
> > PyRefHolder module(PyImport_ImportModule("progwrapper"));
> >
> > if (module._ref)
> > {
> > PyXRefHolder func(PyObject_GetAttrString(module._ref, "prog"));
> > PyXRefHolder pyArgs(PyTuple_New(1));
> > PyXRefHolder pyDict(convertDictForPython(dictIn));
> > PyTuple_SetItem(pyArgs._ref, 0, pyDict._ref);
>
> |Do you DECREF pyDict._ref later on? I ask because PyTuple_SetItem
> steals a reference.|
>
> |Many other functions that "store" an object, such as PyList_Append,
> INCREF to retain the stored object, but PyTuple_SetItem doesn't.|
>
> ||
>
> > PyRefHolder pyRet(PyObject_CallObject(func._ref, pyArgs._ref));
> >
> > if (pyRet._ref != NULL)
> > {
> > addPythonValuesToDict(pyRet._ref, theDict);
> > ...
> >
> > where GilStateHolder, PyRefHolder etc are such resource objects as
> > described above, wrapping round the PyObject pointers etc.
> >
> > [snip]
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Access violation in Python garbage collector (visit_decref) - how to debug?
On Tue, Oct 8, 2019 at 7:27 AM dieter wrote: > Geoff Bache writes: > > Yes, this is hard, that's why I'm here :) > > > > I've enabled the equivalent tools to valgrind in Visual Studio, and tried > > setting PYTHONMALLOC=debug, but neither of those seem to be showing > > anything either. I don't really know what else to try in this direction. > > It likely is too hard to be solved remotely (in this list). > > On the contrary, MRAB just solved it :) Thanks for your replies. /Geoff -- https://mail.python.org/mailman/listinfo/python-list
Should PyImport_ImportModule be threadsafe when importing from zipfiles?
Hi all,
I have some embedded Python code which looks like this in C++
_gstate = PyGILState_Ensure();
PyImport_ImportModule("a");
...
PyGILState_Release(_gstate);
and is called from different threads which are created in C++.
My module a.py then imports another module b in python, which defines a lot
of functions.
When several threads execute this simultaneously I often get a stacktrace
saying some function near the end of module b is not defined, presumably
because the module has been imported part-initialised.
This only seems to happen when my Python modules are packaged in a zip
file, not when they are ordinary files on disk.
I have observed this in both Python 3.7 and Python 3.8. Does anyone have
any insights or suggestions for how to debug this? It seems likely to be
hard to produce a reproducible test case.
Regards,
Geoff Bache
--
https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
Hi Chris, Yes, I've tried both of these things already. I can confirm there are multiple calls, and that pre-importing the module fixes it. But pre-importing it is not a solution in practice. Regards, Geoff On Thu, Feb 20, 2020 at 4:45 PM Chris Angelico wrote: > On Fri, Feb 21, 2020 at 2:37 AM Geoff Bache wrote: > > When several threads execute this simultaneously I often get a stacktrace > > saying some function near the end of module b is not defined, presumably > > because the module has been imported part-initialised. > > This only seems to happen when my Python modules are packaged in a zip > > file, not when they are ordinary files on disk. > > > > I have observed this in both Python 3.7 and Python 3.8. Does anyone have > > any insights or suggestions for how to debug this? It seems likely to be > > hard to produce a reproducible test case. > > One easy way to probe the bug would be to pre-import the module before > starting any secondary threads. If you ever get the problem under that > pattern, then it's not a concurrency problem (IOW have fun figuring > out what *is* the problem). > > Another thing to try: Slap a print call at the top and bottom of the > module. See if you get multiple of them. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
But, as I say, I don't have the module loaded beforehand, so caching it
from sys.modules and restoring it afterwards won't be possible.
Just investigating which calls are made doesn't seem likely to provide new
information. I already know which call is made, it's the one that produces
the stacktrace...
Basically I'm looking for hints as to information about known problems and
whether this kind of thing is likely to work anyway, and also hints for
debugging - as in extracting more information about possible causes rather
than documenting the symptoms further. I think I already have a fairly good
overview of the symptoms.
/Geoff
On Fri, Feb 21, 2020 at 9:11 AM Chris Angelico wrote:
> On Fri, Feb 21, 2020 at 6:56 PM Geoff Bache wrote:
> >
> > I'm not sure I understand what you mean by "stick something" there. What
> did you have in mind? I can't just put anything there or the importing will
> fail anyhow.
> > Also, setting the module back into there afterwards isn't possible
> unless I've already imported it, and as mentioned I can't do that in
> practice.
> >
>
> The best object to put in there would be something that prints out all
> __getattr__ calls. Something like this:
>
> import sys
> _this_module = sys.modules[__name__]
> print("** Importing %s, module ID is %s **" % (__name__, id(_this_module)))
> class Marker:
> def __getattr__(self, attr):
> print("** Sentinel imported, looking for %s.%s **" % (__name__,
> attr))
> sys.modules[__name__] = Marker()
>
> ...
> # rest of module goes here
> ...
>
> sys.modules[__name__] = _this_module
> del _this_module
>
>
> Untested, might need some tweaks. The idea is that, if anything
> imports the module while it's still running, it should either print
> out a second "importing" line with a different ID, or it'll get the
> sentinel object, and any use of it will trigger the message.
>
> (Feel free to use the logging module instead of print, same difference.)
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
I'm not sure I understand what you mean by "stick something" there. What did you have in mind? I can't just put anything there or the importing will fail anyhow. Also, setting the module back into there afterwards isn't possible unless I've already imported it, and as mentioned I can't do that in practice. /Geoff On Thu, Feb 20, 2020 at 6:59 PM Chris Angelico wrote: > On Fri, Feb 21, 2020 at 3:06 AM Geoff Bache wrote: > > > > Hi Chris, > > > > Yes, I've tried both of these things already. I can confirm there are > multiple calls, and that pre-importing the module fixes it. But > pre-importing it is not a solution in practice. > > > > Cool, good to know. > > Crazy idea: What would happen if you stick something temporarily into > sys.modules["b"], then when you're done importing, set the module back > into there? Might not help, but would be interesting to try, and might > show a bit more of what's going on. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
OK, I see what you mean with the import backup change now, though it seems like it changes the semantics of Python's import mechanism a bit... But I still don't really get what information we'd gain from doing this. What call is being made and is failing is pretty obvious from the stacktrace. Thanks for your replies, anyway. /Geoff On Fri, Feb 21, 2020 at 12:25 PM Chris Angelico wrote: > On Fri, Feb 21, 2020 at 8:58 PM Geoff Bache wrote: > > > > But, as I say, I don't have the module loaded beforehand, so caching it > from sys.modules and restoring it afterwards won't be possible. > > Those lines go into the actual file you're importing. > > > Basically I'm looking for hints as to information about known problems > and whether this kind of thing is likely to work anyway, and also hints for > debugging - as in extracting more information about possible causes rather > than documenting the symptoms further. I think I already have a fairly good > overview of the symptoms. > > > > Yeah, changing the actual module that's being messed with might help > with that. I am kinda clutching at straws though. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Syntax error (The Python Book) Linux User and Developer Bookazine
Hi,
Noob at the Python thing so here goes,
I have copied a program to demonstrate control structures in Python but get a
syntax error at line 31, isint = False. I'm using Python 2.7.6 and Linux Mint
based around ubuntu14.04.1. I have pasted all the code below,
#!/usr/bin/env python2
'''
We are going to write a program that will ask for the user to input an arbitary
number of integers, store them in a collection, and then demonstrate how
the collection would be used with various control structures
'''
import sys # Used for the sys.exit function
target_int=raw_input("How many integers?")
'''
By now the variable target_int contains a string representation of
whatever the user typed. We nee to try and convert that to an integer but
be ready to deal with the error if it's not. Otherwise the program will crash
'''
try:
target_int=int(target_int)
except ValueError:
sys.exit("You must enter an integer")
ints=list() # list to store the integers
count = 0 # Track how many integers have been inputted
# Keep asking for a number until we have reached the required number
while count < target_int:
new_int=raw_input("Please enter integer {0}:".format(count +1)
isint = False
try:
new_int=int(new_int) # If the above succeeds then isint will
#be set to true: isint = True
except:
print("You must enter an integer")
'''
Only carry on if we have an integer. If not we will loop again.
The == below is a comparision operator, a single = is an asignment operator
'''
if isnit==True:
ints.append(new_int) # Adds the integer to the collection
count += 1 # Count is incremented by 1
# The for loop
print ("Using a for loop")
for values in ints:
print (str(value))
# The while loop
print ("Using a while loop")
total=len(ints) # We already have the total from above but using len we
can determine from the ints list.
count = 0
while count < total:
print (str(ints[count]))
count += 1
--
https://mail.python.org/mailman/listinfo/python-list
Re: Syntax error (The Python Book) Linux User and Developer Bookazine
On Sunday, 14 February 2016 13:39:52 UTC, Geoff Munn wrote:
> Hi,
>
> Noob at the Python thing so here goes,
>
> I have copied a program to demonstrate control structures in Python but get a
> syntax error at line 31, isint = False. I'm using Python 2.7.6 and Linux Mint
> based around ubuntu14.04.1. I have pasted all the code below,
>
>
>
> #!/usr/bin/env python2
>
> '''
> We are going to write a program that will ask for the user to input an
> arbitary
> number of integers, store them in a collection, and then demonstrate how
> the collection would be used with various control structures
> '''
>
> import sys # Used for the sys.exit function
>
> target_int=raw_input("How many integers?")
>
> '''
> By now the variable target_int contains a string representation of
> whatever the user typed. We nee to try and convert that to an integer but
> be ready to deal with the error if it's not. Otherwise the program will crash
> '''
>
> try:
> target_int=int(target_int)
> except ValueError:
> sys.exit("You must enter an integer")
>
> ints=list() # list to store the integers
>
> count = 0 # Track how many integers have been inputted
>
> # Keep asking for a number until we have reached the required number
> while count < target_int:
> new_int=raw_input("Please enter integer {0}:".format(count +1)
> isint = False
> try:
> new_int=int(new_int) # If the above succeeds then isint will
> #be set to true: isint = True
>
> except:
> print("You must enter an integer")
>
> '''
> Only carry on if we have an integer. If not we will loop again.
> The == below is a comparision operator, a single = is an asignment
> operator
> '''
> if isnit==True:
> ints.append(new_int) # Adds the integer to the collection
> count += 1 # Count is incremented by 1
> # The for loop
> print ("Using a for loop")
> for values in ints:
> print (str(value))
> # The while loop
> print ("Using a while loop")
> total=len(ints) # We already have the total from above but using len we
> can determine from the ints list.
> count = 0
> while count < total:
> print (str(ints[count]))
> count += 1
Thanks Peter and Chris, yes missed the parentheses by taking the error as being
in line 31, DOH but a lesson learned. I have checked and checked the code I
entered against the provided code and had to make some more changes to at least
go through the first while loop but have given up on the rest of it. Given your
comments do you think its worth persevering with this book or is there a better
'entry' into Python programming?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Syntax error (The Python Book) Linux User and Developer Bookazine
On Monday, 15 February 2016 15:07:03 UTC, Joel Goldstick wrote:
> On Mon, Feb 15, 2016 at 9:56 AM, Geoff Munn wrote:
>
> > On Sunday, 14 February 2016 13:39:52 UTC, Geoff Munn wrote:
> > > Hi,
> > >
> > > Noob at the Python thing so here goes,
> > >
> > > I have copied a program to demonstrate control structures in Python but
> > get a syntax error at line 31, isint = False. I'm using Python 2.7.6 and
> > Linux Mint based around ubuntu14.04.1. I have pasted all the code below,
> > >
> > >
> > >
> > > #!/usr/bin/env python2
> > >
> > > '''
> > > We are going to write a program that will ask for the user to input an
> > arbitary
> > > number of integers, store them in a collection, and then demonstrate how
> > > the collection would be used with various control structures
> > > '''
> > >
> > > import sys # Used for the sys.exit function
> > >
> > > target_int=raw_input("How many integers?")
> > >
> > > '''
> > > By now the variable target_int contains a string representation of
> > > whatever the user typed. We nee to try and convert that to an integer but
> > > be ready to deal with the error if it's not. Otherwise the program will
> > crash
> > > '''
> > >
> > > try:
> > > target_int=int(target_int)
> > > except ValueError:
> > > sys.exit("You must enter an integer")
> > >
> > > ints=list() # list to store the integers
> > >
> > > count = 0 # Track how many integers have been inputted
> > >
> > > # Keep asking for a number until we have reached the required number
> > > while count < target_int:
> > > new_int=raw_input("Please enter integer {0}:".format(count +1)
> > > isint = False
> > > try:
> > > new_int=int(new_int) # If the above succeeds then isint will
> > > #be set to true: isint = True
> > >
> > > except:
> > > print("You must enter an integer")
> > >
> > > '''
> > > Only carry on if we have an integer. If not we will loop again.
> > > The == below is a comparision operator, a single = is an asignment
> > operator
> > > '''
> > > if isnit==True:
> > > ints.append(new_int) # Adds the integer to the collection
> > > count += 1 # Count is incremented by 1
> > > # The for loop
> > > print ("Using a for loop")
> > > for values in ints:
> > > print (str(value))
> > > # The while loop
> > > print ("Using a while loop")
> > > total=len(ints) # We already have the total from above but using
> > len we can determine from the ints list.
> > > count = 0
> > > while count < total:
> > > print (str(ints[count]))
> > > count += 1
> >
> > Thanks Peter and Chris, yes missed the parentheses by taking the error as
> > being in line 31, DOH but a lesson learned. I have checked and checked the
> > code I entered against the provided code and had to make some more changes
> > to at least go through the first while loop but have given up on the rest
> > of it. Given your comments do you think its worth persevering with this
> > book or is there a better 'entry' into Python programming?
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
> Learn Python the Hard Way is pretty good some people say. Its online.
> Also Diving into Python is online written by the now offline Mark Pilgrim.
> --
> Joel Goldstick
> http://joelgoldstick.com/stats/birthdays
Thanks Joel
--
https://mail.python.org/mailman/listinfo/python-list
Customizing class attribute access in classic classes
Hi, I'm wondering if there is any way to customize class attribute access on classic classes? So this works: class Meta(type): def __getattr__(cls, name): return "Customized " + name class A: __metaclass__ = Meta print A.blah but it turns A into a new-style class. If "Meta" does not inherit from type, the customization works but A ends up not being a class at all, severely restricting its usefulness. I then hoped I could get "Meta" to inherit from types.ClassType but that wasn't allowed either. Is there any way to do this or is it just a limitation of classic classes? Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Customizing class attribute access in classic classes
On Oct 30, 4:16 am, Ben Finney wrote: > Geoff Bache writes: > > I'm wondering if there is any way to customize class attribute access > > on classic classes? > > Why do that? What is it you're hoping to achieve, and why limit it to > classic classes only? > I'm building a mocking tool, CaptureMock, which works by intercepting and capturing particular calls, recording and replaying them. A user can just say "intercept httplib for me" and it will record all the interactions with httplib and allow a test that can be run without doing anything via http or writing any handcrafted "mock-code". So I need to be able to intercept also static attribute access, say httplib.HTTPConnection.request. httplib.HTTPConnection is a classic class. I can make my intercepting version of it into a new-style class but the risk is that that will change its behaviour in subtle ways, negating the point of the tool. As for limiting it to classic classes only, I obviously need to do it on new-style classes also. But I know how to do that... Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Customizing class attribute access in classic classes
Thanks for this Steven. I'm however gettings some pretty odd effects, both on method access and inheritance. I expanded your example a bit... class Meta: def __init__(self, name, bases, namespace): self.__name__ = name self.__bases__ = bases self.__dict__ = namespace def __str__(self): return "" __repr__ = __str__ def __getattr__(self, name): return "Customized " + name def __call__(self): return self class Base: def basemethod(self): return "base" class A(Base): __metaclass__ = Meta def method(self): return "answer" The effect seems to be to make all methods of A into static methods, and to ignore its base classes altogether: >>> a = A() >>> print a.blah2 Customized blah2 >>> print a.method() Traceback (most recent call last): File "", line 1, in TypeError: method() takes exactly 1 argument (0 given) >>> print a.method(1) answer >>> print A.method(1) answer >>> print a.basemethod() Traceback (most recent call last): File "", line 1, in TypeError: 'str' object is not callable >>> isinstance(a, Base) False Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Buffering of sys.stdout and sys.stderr in python3 (and documentation)
Hi all, Short version: I'm a bit confused in general as to the changes between python2 and python3 regarding how standard output and standard error do buffering. A few things seem to have changed and I've failed to find any documentation of how and why. Also, the meaning of "python -u" seems to have changed and the docs don't seem to reflect the new behaviour (and I can't find any docs about the change either)... Long version: >From rude experiment it seems that: 1) In Python 2.x, standard error was always unbuffered while standard output was buffered by default. In python3, both are buffered. In both cases, "buffered" means line-buffered when writing to the console and not line-buffered when redirected to files. 2) In Python 2.x, the "-u" flag meant everything was totally unbuffered. In Python 3.x, it means that both stdout and stderr are line-buffered also when redirected to files. Are either of these changes documented anywhere? (1) seems important : it can lead to not seeing exception printouts, if stderr is redirected to a file and the program is subsequently terminated with SIGTERM. I just wasted quite a bit of time due to this situation... This is what the Python 3 docs have to say about the -u flag: "Force the binary layer of the stdin, stdout and stderr streams (which is available as their buffer attribute) to be unbuffered. The text I/O layer will still be line-buffered." The "still" seems misleading to me, as it is only relevant if writing to the console. It would be useful to contrast the behaviour with and without "-u" when writing to files I would say. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Buffering of sys.stdout and sys.stderr in python3 (and documentation)
Hi Terry, > The difference from 2.x should be in What's New in 3.0, except that the > new i/o module is in 2.6, so it was not exactly new. The io module existed in 2.6, but it was not used by default for standard output and standard error. The only mention of this in "What's New in 3.0" is in the section marked for changes that were already in 2.6 (which is wrong in this case), and it notes only that io.TextIOWrapper is now used, but not what implications that has for its behaviour and backward compatibility. > You might be able > to find more inhttp://python.org/dev/peps/pep-3116/ I skimmed through it but couldn't find anything relevant. It seems more "advanced" and implementation-focussed. > > You *should* be able to find sufficient info in the 3.x docs. If, after > you get other responses (or not), you think the docs need upgrading, > open an issue on the tracker at bugs.python.org with suggestions as > specific as possible, including changed or new lines of text based on > your experience and experiments. OK, I'll do that if nobody points me at some existing docs here. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Embedding pure python statically -- possibly a newbie question
I have inherited a project with an embedded python interpretter, and
don't know much about it, but have been given the task of porting it to
a new compiler/OS and have run into some problems.
I seem to have got the embedded interpretter part of it working, and
have worked out how to add c modules to extend it. My problem is
embedding pure python code so that I don't have to have external .py
files to my application.
What I mean is, if I have a module, foo.py, I'd like to somehow have its
code as part of my application executable, so when the application does,
for example,
PyImport_ImportModule("foo");
it won't look for a foo.py, but use the internal static version. The C
style modules do this, so I imagine it can be done with pure python as
well, but I've not figured out *how*. I can a C wrapper to do this
(embedding python in C in python in), but I figure there has to be
another way.
I've been going over the docs, and haven't gotten very far, so it's
possible this is answered there. If so, just give me a nudge/push/shove
in the right direction.
Thanks for any help!
--
http://mail.python.org/mailman/listinfo/python-list
Using freeze or other application to create linkable dynamic libraries for c++ applications?
Is it possible to create a DLL (pr on unix a .a, .so, etc) from python code, perhaps using freeze or some other application? Freeze seems to assume that it's creating the main program body, but I don't really understand it well enough to say. If it doesn't work, is there another application that can be used to do this? -- http://mail.python.org/mailman/listinfo/python-list
trying to check the creation date of a file
Nice going way to help! gb -- http://mail.python.org/mailman/listinfo/python-list
Re: Best Free and Open Source Python IDE
If you can take some time and master Vim, you'll be set for typing out any programming language for the rest of your life. I hear Emacs is good too, and the GNU project is great, so you could try that as well. It's supposed to be more geared towards programming -- http://mail.python.org/mailman/listinfo/python-list
Regular Expressions
What's the way to go about learning Python's regular expressions? I feel like such an idiot - being so strong in a programming language but knowing nothing about RE. -- http://mail.python.org/mailman/listinfo/python-list
Re: Hacking in python
The word "hack" can be known as a "smart/quick fix" to a problem. -- http://mail.python.org/mailman/listinfo/python-list
Re: Regular Expressions
Thanks. O'Reilly is the way I learned Python, and I'm suprised that I didn't think of a book by them earlier. -- http://mail.python.org/mailman/listinfo/python-list
Re: can't find a way to display and print pdf through python.
Are you trying to: a) Make the PDF file open in it's default application? b) Create a PDF-reader in Python? ...because your question is somewhat unclear. Report Lab has no PDF viewer. You would need a PDF/PostScript parser to do that and that's more of a job than I think you're looking for. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is python worth learning as a second language?
In article , [email protected] says... > > Hi > > I hope I won't sound trivial with asking my question. > > I am a C++ programmer and I am thinking of learning something else > because I know second language might be very helpful somehow. I have > heard a few positive things about Python but I have never writen any > single line in python so I do not know this language at all. > > Do you think python would be good complementary language for C++? Do you > think it's worth learning it or let's say try Java? and how difficult it > would be for me if I know C++ pretty well I would say? > > Thanks Every response is going to be pure opinion, of course. But learning multiple languages that have different primary purposes is always a good thing. Python would be an excellent choice to supplement your C++. There are other languages that might suit you better if you have more directed needs in mind. But as a general use language, I doubt that Python would disappoint. It is extremely functional and powerful, but still simple enough that I generally recommend it as a FIRST language for those who want to learn programming. -- http://mail.python.org/mailman/listinfo/python-list
Problems with background processes on Windows
Hi all, The following code behaves differently on Windows and Linux using Python 2.5.2. The Linux behaviour is what I expect in both places :) Perhaps somebody could help explain this. Or maybe it is a Python bug. Or a Windows feature... communicate.py --- import subprocess p = subprocess.Popen([ "python", "sleep.py" ], stdout=subprocess.PIPE, stderr=subprocess.PIPE) p.communicate() --- sleep.py import subprocess, os subprocess.Popen([ 'python', '-c', 'import time; time.sleep(10)' ], stdin=open(os.devnull), stdout=open(os.devnull, "w"), stderr=subprocess.STDOUT) In short, we start a subprocess which in turn starts a different subprocess and then returns without waiting for it. We then try to "communicate" with the first subprocess. On Windows if I run "communicate.py" it does not return for 10 seconds, i.e. until the "grandchild" background process terminates. On Linux it returns immediately as I would expect. If I replace "p.communicate()" with "p.wait()" it returns immediately on both. If I don't point stdin, stdout and stderr of the child process to os.devnull then it will wait 10 seconds on Linux also, which I'd also expect as we can't collect info from these places if they've been forwarded elsewhere. It seems like Windows somehow doesn't notice this. Any help gratefully appreciated. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
How do I test the integrity of a Python installation in Debian and Ubuntu
How do I assure myself of the integrity of a Python installation acquired using apt-get install on Debian and Ubuntu? I can run regrtest but there's nothing in the basic installation to run, viz.: geg...@gegard:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from test import regrtest >>> regrtest.main() test_grammar test_grammar skipped -- No module named test_grammar ... more of the same... 9 tests skipped: test_builtin test_doctest test_doctest2 test_exceptions test_grammar test_opcodes test_operations test_types test_unittest Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/test/regrtest.py", line 416, in main e = _ExpectedSkips() File "/usr/lib/python2.5/test/regrtest.py", line 1321, in __init__ from test import test_socket_ssl ImportError: cannot import name test_socket_ssl >>> I don't see where to go from here, so advice would be extremely helpful. Background I have a large Python-based system that I am trying to install on an externally-hosted VM. It doesn't build and install correctly most of the time, and I have tried successive images of Debian Lenny and Ubuntu Hardy with mostly different build/installation results each time. The installation proceeds Ok on a local Ubuntu VM. A number of modules are installed in addition to python, but I can't even see how to test the core python installation. There are few test_*.py files in the installation. I have previously encountered a fault on the server hosting the VM and would like to be more comfortable that the python installation itself is Ok (or have evidence that it's not). Thank you, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I test the integrity of a Python installation in Debian and Ubuntu
Aahz wrote: > How important is the apt-get requirement? That's a significant point, thank you. Two parts to the response: a) I don't feel that I am sufficiently expert to launch into compilation of Python if I can avoid it. b) I hope that I can put all platform risk into the lap of the hosting provider, and apt-get is probably the only delivery mechanism they will take any responsibility for. (I want to use their 'standard'.) All the best, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I test the integrity of a Python installation in Debian and Ubuntu
Aahz wrote: > > What directory are you running this from? What happens if you switch to > running "python Lib/test/regrtest.py"? Taking a closer look, this looks > more like a plain import error. > Thank you for your suggestion. I couldn't do quite that because there's no Lib, but instead (in Ubuntu Hardy, this time): geg...@gegard:~$ cd /usr/lib/python2.5/ geg...@gegard:/usr/lib/python2.5$ python test/regrtest.py test_grammar test_grammar skipped -- No module named test_grammar test_opcodes test_opcodes skipped -- No module named test_opcodes test_operations test_operations skipped -- No module named test_operations test_builtin test_builtin skipped -- No module named test_builtin test_exceptions test_exceptions skipped -- No module named test_exceptions test_types test_types skipped -- No module named test_types test_unittest test_unittest skipped -- No module named test_unittest test_doctest test_doctest skipped -- No module named test_doctest test_doctest2 test_doctest2 skipped -- No module named test_doctest2 9 tests skipped: test_builtin test_doctest test_doctest2 test_exceptions test_grammar test_opcodes test_operations test_types test_unittest Traceback (most recent call last): File "test/regrtest.py", line 1384, in main() File "test/regrtest.py", line 416, in main e = _ExpectedSkips() File "test/regrtest.py", line 1321, in __init__ from test import test_socket_ssl ImportError: cannot import name test_socket_ssl geg...@gegard:/usr/lib/python2.5$ Also geg...@gegard:~$ locate */test_socket_ssl.* geg...@gegard:~$ #returns nothing And geg...@gegard:~$ locate /usr/lib/python2.5/test/test_*.* /usr/lib/python2.5/test/test_support.py /usr/lib/python2.5/test/test_support.pyc geg...@gegard:~$ All the best, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I test the integrity of a Python installation in Debian and Ubuntu
Aahz wrote: > ... That seems to demonstrate that regrtest.py is indeed a good mechanism for > finding out whether it's a b0rked install! > I agree that regrtest.py looks a good mechanism. It just appears that `apt-get install python` on Debian and Ubuntu brings no tests with it. @Lawrence D'Oliveiro: `apt-get install debsums` does indeed show that the packages that it checks have matching checksums. `debsums [-a] python` checks all sorts of /usr/share/doc/python files while `debsums [-a] python2.5` checks /usr/share/doc/python2.5, /usr/lib/python2.5 and others as well. Although that wasn't what I was originally looking for it has been helpful. Thank you for your suggestions. All the best, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I test the integrity of a Python installation in Debian and Ubuntu
Lawrence D'Oliveiro wrote: > .. I expect an apology. > Otherwise, it becomes grounds for an abuse complaint to your ISP. > Yes, I do apologize profusely and publicly, and would have done so regardless of threat. I had trouble with posts making it through to the list and so was also posting in parallel to the original posters. In doing so, your address initially bounced the message - so I changed it unthinkingly in a to: line. I did not realize until this morning that they made it through python-list unchanged, as they obviously do. All the best, Geoff -- http://mail.python.org/mailman/listinfo/python-list
Finding the right Python executable on Windows
Hi all, I have a Python process on Windows and would like to start a Python subprocess using the same interpreter. I wonder how to go about this? First, I tried the obvious subprocess.Popen([ sys.executable, "subproc.py", ... ]) but that fails, because my process has a "native launcher", (i.e. C: \path\mylauncher.exe, which is just a wrapper around the Python program) and hence sys.executable returns this path instead of the interpreter location. It isn't appropriate to use the launcher for the subprocess. I also tried using sys.exec_prefix but there didn't seem to be any standard location for the interpreter under this directory in any case. I feel certain there must be some way to do this as it seems a rather basic thing somehow, can anyone give me a hint? Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding the right Python executable on Windows
On Tue, Jan 25, 2011 at 3:24 PM, Brian Curtin wrote: > On Tue, Jan 25, 2011 at 04:25, Geoff Bache wrote: >> >> Hi all, >> >> I have a Python process on Windows and would like to start a Python >> subprocess using the same interpreter. I wonder how to go about this? >> >> First, I tried the obvious >> >> subprocess.Popen([ sys.executable, "subproc.py", ... ]) >> >> but that fails, because my process has a "native launcher", (i.e. C: >> \path\mylauncher.exe, which is just a wrapper around the Python >> program) and hence sys.executable returns this path instead of the >> interpreter location. It isn't appropriate to use the launcher for the >> subprocess. >> >> I also tried using sys.exec_prefix but there didn't seem to be any >> standard location for the interpreter under this directory in any >> case. >> >> I feel certain there must be some way to do this as it seems a rather >> basic thing somehow, can anyone give me a hint? >> >> Regards, >> Geoff Bache > > If sys.executable doesn't work due to this "native launcher", you could > try something like this: >>>> import os >>>> import sys >>>> full_path = None >>>> for path in sys.path: > ... full = os.path.join(path, "python.exe") > ... if os.path.exists(full): > ... full_path = full > ... >>>> full_path > 'c:\\python31\\python.exe' Thanks Brian! It never occurred to me that the Python executable would be on sys.path, but it seems to be on the installations I've tried on Windows. It isn't on other platforms of course, but it's easy enough to only do this on Windows. I wonder why it's like this? I can't see anything in that directory I could import... Regards, Geoff -- http://mail.python.org/mailman/listinfo/python-list
A simpler logging configuration file format?
Hi all,
I wonder if there are others out there who like me have tried to use
the logging module's configuration file and found it bloated and over-
complex for simple usage (i.e. a collection of loggers writing to
files)
At the moment, if I want to add a new logger "foo" writing to its own
file "foo" to my config file, I have to repeat the name 7(!) times,
editing 3 different sections in the process:
1) Add "foo" to [loggers] and [handlers]. (These seem completely
pointless and were apparently added in anticipation of new
functionality that never happened).
2) Create the section
[logger_foo]
handler:foo
qualname:foo
level:INFO
3) Create the section
[handler_foo]
class: FileHandler
args:("foo", "a")
Wondering how it got like this, I found this blog from soon after it
was released in 2004:
http://www.mechanicalcat.net/richard/log/Python/Simple_usage_of_Python_s_logging_module
Some of the verdicts are "full of dead chickens and error prone",
"horribly complicated" , "over-engineered". So along comes the
"basicConfig" method in 2005 which is supposed to address thes
concerns. But this can only be applied globally, not even to
individual loggers, so is only helpful for extremely basic usage
(everything to one target) as far as I can see. The config file is
still much as it was then.
I'd really like to see the concept extended to encompass multiple
loggers and the config file. Allowing Logger.basicConfig should be
trivial. Writing a configuration format which took a section and
passed all the options in it to basicConfig would in my view lead to a
much friendlier syntax for normal usage.
--
http://mail.python.org/mailman/listinfo/python-list
inspect.getmodulename giving unexpected results
Hi all, I'm trying to examine some things in my stack. The information I get out of inspect.stack() gives file names and I would like to convert them to module names. I naturally assumes inspect.getmodulename would fix this for me. Unfortunately, it doesn't seem to do that in some cases. Consider the following code: ## file 'inspect_test' #!/usr/bin/env python import logging, inspect print inspect.getmodulename(logging.__file__), inspect.getmodulename(__file__) I hoped that this would print logging __main__ or at least logging inspect_test but instead it prints __init__ None which isn't very helpful, although it could technically be considered correct (I can go to the 'logging' directory and type "import __init__" but nobody ever does) I guess I was hoping to get the names of the modules as they appear in sys.modules. Maybe there is some other way to do that? I can naturally write my own method to do this but wondered if I'm missing something here. Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: inspect.getmodulename giving unexpected results
> > Unfortunately, it doesn't seem to do that in some cases. Consider the > > following code: > > It does behave as documented: it does not find package names, or investigate > sys.modules Possibly, although for me "logging" is exposed as a module, and ought to behave as one when requesting the module name. The fact that it's a package internally doesn't seem relevant to me in this context. It's weird to me that when I do import os, logging treating these things as normal modules I can get the right name for "os" but get "__init__" when requesting "logging". Externally, these things appear to be the same. I can accept from the docs that if I did "getmodulename" on "logging.config" I would get "config" back. > > > ## file 'inspect_test' > > rename this file to "inspect_test.py", and the filename will make sense to > inspect.getmodulename, which should *then* return "inspect_test" instead of > None. I realise that, but of course I can't do that in the real code. It's a long-established convention to drop the ".py" from executable programs on UNIX. Regards, Geoff -- http://mail.python.org/mailman/listinfo/python-list
A way to get setup.py to create links instead of copy
Hi all, One of the things I've always loved about Python (having come from compiled languages) was the lack of an extra step between changing my code and running it. On my current project, however, I find that I have to install my Python code with setup.py before it will run. Being used to not having this step, I easily forget to run setup.py install before they will run, and then spend time wondering why my changes didn't work. So I went into the target directory and replaced the copies with symbolic links back to the original code. This felt like a hack but does mean I don't have to install the whole time any more. I wonder if there is some standard way to deal with this situation? Regards, Geoff Bache -- http://mail.python.org/mailman/listinfo/python-list
Re: A way to get setup.py to create links instead of copy
On Nov 9, 6:33 pm, Steve Holden wrote: > On 11/9/2010 4:18 AM, Geoff Bache wrote: > > > Hi all, > > > One of the things I've always loved about Python (having come from > > compiled languages) was the lack of an extra step between changing my > > code and running it. > > > On my current project, however, I find that I have to install my > > Python code with setup.py before it will run. Being used to not having > > this step, I easily forget to run setup.py install before they will > > run, and then spend time wondering why my changes didn't work. > > > So I went into the target directory and replaced the copies with > > symbolic links back to the original code. This felt like a hack but > > does mean I don't have to install the whole time any more. > > > I wonder if there is some standard way to deal with this situation? > > Yes, there is. It involves (to a first approximation) understanding the > role of sys.path, and its relationship with the PYTHONPATH environment > variable. In brief, this tells the interpreter where to look for modules > when it is importing. > > Installation with setup.py is normally reserved for a fairly permanent > insertion of the code into your Python distribution, which might have > added it for all users. If this is an issue you may want to undo the > installation manually by removing its additions to your Python's > Lib/site-packages directory. > Hi Steve, I think I understand this stuff fairly well. I have my own Python installation (using virtualenv) so I'm not affecting anyone else. The problem is that my program runs from the command line, and is tested via other programs that call it as a subprocess, and don't expect to have to invoke Python explicitly. It does not work with the default Python on my system, which is a too old version, so relies on the "#!" line at the top to be correct. As I cannot hardcode this in my source tree, it seems difficult to make it runnable from the source tree. Incidentally, I'm interested that you say setup.py is normally reserved for a permanent installation. I've run into many Python programs, including quite well known ones, which did not appear to be runnable directly from the download or source tree, but required running setup.py before they would work. Regards, Geoff -- http://mail.python.org/mailman/listinfo/python-list
