Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
On Fri, Feb 21, 2020 at 6:56 PM Geoff Bache wrote:
>
> I'm not sure I understand what you mean by "stick something" there. What did
> you have in mind? I can't just put anything there or the importing will fail
> anyhow.
> Also, setting the module back into there afterwards isn't possible unless
> I've already imported it, and as mentioned I can't do that in practice.
>
The best object to put in there would be something that prints out all
__getattr__ calls. Something like this:
import sys
_this_module = sys.modules[__name__]
print("** Importing %s, module ID is %s **" % (__name__, id(_this_module)))
class Marker:
def __getattr__(self, attr):
print("** Sentinel imported, looking for %s.%s **" % (__name__, attr))
sys.modules[__name__] = Marker()
...
# rest of module goes here
...
sys.modules[__name__] = _this_module
del _this_module
Untested, might need some tweaks. The idea is that, if anything
imports the module while it's still running, it should either print
out a second "importing" line with a different ID, or it'll get the
sentinel object, and any use of it will trigger the message.
(Feel free to use the logging module instead of print, same difference.)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
On Fri, Feb 21, 2020 at 8:58 PM Geoff Bache wrote: > > But, as I say, I don't have the module loaded beforehand, so caching it from > sys.modules and restoring it afterwards won't be possible. Those lines go into the actual file you're importing. > Basically I'm looking for hints as to information about known problems and > whether this kind of thing is likely to work anyway, and also hints for > debugging - as in extracting more information about possible causes rather > than documenting the symptoms further. I think I already have a fairly good > overview of the symptoms. > Yeah, changing the actual module that's being messed with might help with that. I am kinda clutching at straws though. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
But, as I say, I don't have the module loaded beforehand, so caching it
from sys.modules and restoring it afterwards won't be possible.
Just investigating which calls are made doesn't seem likely to provide new
information. I already know which call is made, it's the one that produces
the stacktrace...
Basically I'm looking for hints as to information about known problems and
whether this kind of thing is likely to work anyway, and also hints for
debugging - as in extracting more information about possible causes rather
than documenting the symptoms further. I think I already have a fairly good
overview of the symptoms.
/Geoff
On Fri, Feb 21, 2020 at 9:11 AM Chris Angelico wrote:
> On Fri, Feb 21, 2020 at 6:56 PM Geoff Bache wrote:
> >
> > I'm not sure I understand what you mean by "stick something" there. What
> did you have in mind? I can't just put anything there or the importing will
> fail anyhow.
> > Also, setting the module back into there afterwards isn't possible
> unless I've already imported it, and as mentioned I can't do that in
> practice.
> >
>
> The best object to put in there would be something that prints out all
> __getattr__ calls. Something like this:
>
> import sys
> _this_module = sys.modules[__name__]
> print("** Importing %s, module ID is %s **" % (__name__, id(_this_module)))
> class Marker:
> def __getattr__(self, attr):
> print("** Sentinel imported, looking for %s.%s **" % (__name__,
> attr))
> sys.modules[__name__] = Marker()
>
> ...
> # rest of module goes here
> ...
>
> sys.modules[__name__] = _this_module
> del _this_module
>
>
> Untested, might need some tweaks. The idea is that, if anything
> imports the module while it's still running, it should either print
> out a second "importing" line with a different ID, or it'll get the
> sentinel object, and any use of it will trigger the message.
>
> (Feel free to use the logging module instead of print, same difference.)
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
I'm not sure I understand what you mean by "stick something" there. What did you have in mind? I can't just put anything there or the importing will fail anyhow. Also, setting the module back into there afterwards isn't possible unless I've already imported it, and as mentioned I can't do that in practice. /Geoff On Thu, Feb 20, 2020 at 6:59 PM Chris Angelico wrote: > On Fri, Feb 21, 2020 at 3:06 AM Geoff Bache wrote: > > > > Hi Chris, > > > > Yes, I've tried both of these things already. I can confirm there are > multiple calls, and that pre-importing the module fixes it. But > pre-importing it is not a solution in practice. > > > > Cool, good to know. > > Crazy idea: What would happen if you stick something temporarily into > sys.modules["b"], then when you're done importing, set the module back > into there? Might not help, but would be interesting to try, and might > show a bit more of what's going on. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
pip issue
Hi, all
Went to setup path and got an error. attempted to update pip and got the
same error... The error being thrown is -
Traceback (most recent call last):
File "/home/jack/.local/bin/pip", line 11, in
sys.exit(main())
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/cli/main.py",
line 73, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/commands/__init__.py",
line 96, in create_command
module = importlib.import_module(module_path)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in
import_module
__import__(name)
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/commands/install.py",
line 24, in
from pip._internal.cli.req_command import RequirementCommand
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/cli/req_command.py",
line 15, in
from pip._internal.index.package_finder import PackageFinder
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/index/package_finder.py",
line 21, in
from pip._internal.index.collector import parse_links
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_internal/index/collector.py",
line 12, in
from pip._vendor import html5lib, requests
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py",
line 97, in
from pip._vendor.urllib3.contrib import pyopenssl
File
"/home/jack/.local/lib/python2.7/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py",
line 46, in
import OpenSSL.SSL
File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8,
in
from OpenSSL import rand, crypto, SSL
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 118, in
SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
Any input, as always, is appreciated.
Regards
--
https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio question
On 21/02/20 7:59 pm, Frank Millman wrote: My first attempt was to create a background task for each session which runs for the life-time of the session, and 'awaits' its queue. It works, but I was concerned about having a lot a background tasks active at the same time. The whole point of asyncio is to make tasks very lightweight, so you can use as many of them as is convenient without worries. One task per client sounds like the right thing to do here. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Idiom for partial failures
Yes, the API has to support partial failures across all six supported languages. Not all operations need to be atomic and there is considerable efficiency in having multiple operations sent in a single request. Thanks, -David On Thu, Feb 20, 2020 at 6:07 PM Rob Gaddi wrote: > On 2/20/20 9:30 AM, David Wihl wrote: > > > > (first post) > > > > I'm working on the Python client library [0]for the Google Ads API [1]. > In some cases, we can start a request with a partial failure [2] flag = > True. This means that the request may contain say 1000 operations. If any > of the operations fail, the request will return with a success status > without an exception. Then the developer has to iterate through the list of > operation return statuses to determine which specific ones failed (example > [3]). > > > > I believe that it would be more idiomatic in Python (and other languages > like Ruby) to throw an exception when one of these partial errors occur. > That way there would be the same control flow if a major or minor error > occurred. > > > > The team is asking me for other examples or justification of this being > idiomatic of Python. Can you recommend any examples or related best > practices? > > > > [0] https://github.com/googleads/google-ads-python > > > > [1] https://developers.google.com/google-ads/api/docs/start > > > > [2] > https://developers.google.com/google-ads/api/docs/best-practices/partial-failures > > > > [3] > https://github.com/googleads/google-ads-python/blob/master/examples/error_handling/handle_partial_failure.py > > > > Thanks! > > -David > > > > Potentially stupid question, does your library to the API have to support > partial failures? Sure Google Ads does, but does it really add value to > do so? > And then you've got to decide what to DO with those partially failed > results. > > You could, as a library, just say "No, this library never sets the > partial_failure field. We only support atomic transactions that fully > succeed > or fully fail and raise an exception." Makes your life a lot easier; so > unless > it makes your customer's lives a lot harder... > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Should PyImport_ImportModule be threadsafe when importing from zipfiles?
OK, I see what you mean with the import backup change now, though it seems like it changes the semantics of Python's import mechanism a bit... But I still don't really get what information we'd gain from doing this. What call is being made and is failing is pretty obvious from the stacktrace. Thanks for your replies, anyway. /Geoff On Fri, Feb 21, 2020 at 12:25 PM Chris Angelico wrote: > On Fri, Feb 21, 2020 at 8:58 PM Geoff Bache wrote: > > > > But, as I say, I don't have the module loaded beforehand, so caching it > from sys.modules and restoring it afterwards won't be possible. > > Those lines go into the actual file you're importing. > > > Basically I'm looking for hints as to information about known problems > and whether this kind of thing is likely to work anyway, and also hints for > debugging - as in extracting more information about possible causes rather > than documenting the symptoms further. I think I already have a fairly good > overview of the symptoms. > > > > Yeah, changing the actual module that's being messed with might help > with that. I am kinda clutching at straws though. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Idiom for partial failures
On 2/21/20 9:18 AM, David Wihl wrote: Yes, the API has to support partial failures across all six supported languages. Not all operations need to be atomic and there is considerable efficiency in having multiple operations sent in a single request. Thanks, -David On Thu, Feb 20, 2020 at 6:07 PM Rob Gaddi wrote: On 2/20/20 9:30 AM, David Wihl wrote: (first post) I'm working on the Python client library [0]for the Google Ads API [1]. In some cases, we can start a request with a partial failure [2] flag = True. This means that the request may contain say 1000 operations. If any of the operations fail, the request will return with a success status without an exception. Then the developer has to iterate through the list of operation return statuses to determine which specific ones failed (example [3]). I believe that it would be more idiomatic in Python (and other languages like Ruby) to throw an exception when one of these partial errors occur. That way there would be the same control flow if a major or minor error occurred. The team is asking me for other examples or justification of this being idiomatic of Python. Can you recommend any examples or related best practices? [0] https://github.com/googleads/google-ads-python [1] https://developers.google.com/google-ads/api/docs/start [2] https://developers.google.com/google-ads/api/docs/best-practices/partial-failures [3] https://github.com/googleads/google-ads-python/blob/master/examples/error_handling/handle_partial_failure.py Thanks! -David Potentially stupid question, does your library to the API have to support partial failures? Sure Google Ads does, but does it really add value to do so? And then you've got to decide what to DO with those partially failed results. You could, as a library, just say "No, this library never sets the partial_failure field. We only support atomic transactions that fully succeed or fully fail and raise an exception." Makes your life a lot easier; so unless it makes your customer's lives a lot harder... -- https://mail.python.org/mailman/listinfo/python-list In that case, I think Richard Damon's answer smells rightest. A partial failure is a partial success and should not trigger a mandatory flow-control change, which is what an exception is. If there is any information in the successes, then I'd expect an operation like this to return me a list corresponding 1-to-1 with the list of operations such that I could: results = do_the_thing() for op, result in zip(operations, results): if isinstance(result, GoogleApiError): deal_with_error(op, result) If successes have nothing to add to the party, then I'd expect the operation to return a list containing only the errors, and an empty list means no errors. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio question
On 2020-02-21 11:13 PM, Greg Ewing wrote: On 21/02/20 7:59 pm, Frank Millman wrote: My first attempt was to create a background task for each session which runs for the life-time of the session, and 'awaits' its queue. It works, but I was concerned about having a lot a background tasks active at the same time. The whole point of asyncio is to make tasks very lightweight, so you can use as many of them as is convenient without worries. One task per client sounds like the right thing to do here. Perfect. Thanks so much. Frank -- https://mail.python.org/mailman/listinfo/python-list
