[Python-Dev] Re: Thread argument for exc_info and public API
On Fri, Dec 20 2019, Victor Stinner wrote: Hi Victor, > If we add the following function, does it solve your use case? > > void > _PyErr_GetExcInfo(PyThreadState *tstate, > PyObject **p_type, PyObject **p_value, PyObject > **p_traceback) Yes, it would. >> In order to retrieve the exception from *any* PyThreadState, the caller >> has to use _PyErr_GetTopmostException which takes a PyThreadState as >> argument — though that function is private and therefore not documented >> or usable in an external module (in theory at least). > > What if this function is exported as a private function? Usually, > functions in Python header files are prefixed by PyAPI_FUNC(), but for > an unknown reason, this one is not: > > _PyErr_StackItem *_PyErr_GetTopmostException(PyThreadState *tstate); > > Maybe it's because the author didn't want to expose the private > _PyErr_StackItem structure? This is my guess too. It'd make sense since this was struct was introduced around 3.7 and those values where just stored inside PyThreadState before. I guess nobody wanted to commit to an API. I don't think exposing _PyErr_StackItem is worth it. It's just a placeholder for the usual exception triplet (tp, value, tb). >> Should we make _PyErr_GetTopmostException public, or implement something >> different to retrieve the top most exception from a PyThreadState? > > IMHO a _PyErr_GetExcInfo() function taking a tstate parameter is a > better idea than exposing the private _PyErr_StackItem structure. I agree. > Private functions *can* be used for debuggers, but we don't provide > any warranty that the function is not going to disappear. Such > functions have been moved to the internal API for example. The > internal C API *can* be used, but you have to opt-in so you know that > you get unstable APIs :-) Alright! I didn't know that you could use private functions. I'm totally in the debugger case so it's fine with me. :) I'll come up with a patch that exposes _PyErr_GetExcInfo() as you suggested. Thanks Victor! -- Julien Danjou // Free Software hacker // https://julien.danjou.info ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NQMQVVWMURHR7VFVCD43F2LC2JZZEX3F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should set objects maintain insertion order too?
On Thu, Dec 19, 2019 at 9:57 PM Tim Peters wrote: > > It's not obvious to me that insertion order is even the most obvious or > > most commonly relevant sort order. I'm sure it is for Larry's program, > but > > often a work queue might want some other order. Very often queues > > might instead, for example, have a priority number assigned to them. > > Indeed, and it makes me more cautious that claims for the _usefulness_ > (rather than just consistency) of an ordered set are missing an XY > problem. The only "natural" value of insertion-order ordering for a > dynamic ordered set is that it supports FIFO queues. Well, that's one > thing that a deque already excels at, but much more obviously, > flexibly, and space- and time- efficiently. I agree with Tim and David. Furthermore, I'd like to point out that the core built-in types are frequently archetypes or foundational paradigms for many derived concepts in 3rd-party libraries. Despite the fact that they have implementations, they also form a basis set of interfaces which are part of the lexicon of "Pythonic-ness". Something like a "set" is a very very common and useful base interface for programming. If we were to cloud or confound the (relatively clean) semantics around sets & set algebra by introducing ordinality, that same cloudiness will spread throughout the ecosystem. Unlike dict()s, which are a more computer science-y data structure with lots of details, sets are simple things with mathematical algebraic properties. These allow the user to compactly specify *what* they want done, without delving into the details of *how*. In my experience, this is always preferable, for ease of use, ease of learning, and correctness. Adding orderedness as a concern now imposes an additional cognitive burden onto the coder because they must be concerned with *how*. Ultimately, even if we establish that there is major utility in insertion-ordered sets, unordered sets also clearly have utility as well (as a more relaxed interface, with fewer promises). So in that case, which do we want as the default? I think about it like this: In a future time when sets maintain insertion order by default, every time I see library docs that require casting sets via collections.unordered_set() will feel like sand in the semantic vaseline. I would much rather have things the other way around. I don't really have a horse in this race, and I'm sure the dev community will arrive at a good resolution to this issue. However, I would just encourage folks to think less about "utility of the implementation", and more about "economy of semantics". As Larry pointed out earlier, ordered dicts posed a problem for MicroPython. This actually isn't a minor issue -- in the long run, forked semantics are a Bad Thing for the language (even for a language that is incorrectly eponymized with snakes!). -Peter ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EDWT5S3YIQ5UUEFDEWZHTV6YZJ3BNNKL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should set objects maintain insertion order too?
Got it. Thanks On Fri, Dec 20, 2019, 2:34 AM Inada Naoki wrote: > On Fri, Dec 20, 2019 at 4:15 PM Wes Turner wrote: > > > > How slow and space-inefficient would it be to just implement the set > methods on top of dict? > > Speed: Dict doesn't cache the position of the first item. Calling > next(iter(D)) repeatedly is O(N) in worst case. > Space: It waste 8bytes per member. > > > > > Do dicts lose insertion order when a key is deleted? AFAIU, OrderedDict > do not lose insertion order on delete. > > Dict keeps insertion order after deletion too. > > >Would this limit the utility of an ordered set as a queue? What set > methods does a queue need to have? > > I want O(1) D.popleft(). (The name is borrowed from deque. popfirst() > would be better maybe). > > -- > Inada Naoki > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UHNHQGJL7J4UUL44F57KYPQMO4NQRZPO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should set objects maintain insertion order too?
[Wes Turner ] >> How slow and space-inefficient would it be to just implement the set methods >> on top of dict? [Inada Naoki ] > Speed: Dict doesn't cache the position of the first item. Calling > next(iter(D)) repeatedly is O(N) in worst case. > ... See also Raymond's (only) message in this thread. We would also lose low-level speed optimizations specific to the current set implementation. And we would need to define what "insertion order" _means_ for operators (union, intersection, symmetric difference) that build a new set out of other sets without a user-level "insert" in sight. However that's defined, it may constrain possible implementations in ways that are inherently slower (for example, may require the implementation to iterate over the larger operand where they currently iterate over the smaller). And the current set implementation (like the older dict implementation) never needs to "rebuild the table from scratch" unless the cardinality of the set keeps growing. As Raymond telegraphically hinted, the current dict implementation can, at times, in the presence of deletions, require rebuilding the table from scratch even if the dict's maximum size remains bounded. That last can't be seen directly from Python code (rebuilding the table is invisible at the Python level). Here's a short example: d = dict.fromkeys(range(3)) while True: del d[0] d[0] = None Run that with a breakpoint set on dictobject.c's `dictresize()` function. You'll see that it rebuilds the table from scratch every 3rd time through the loop. In effect, for the purpose of deciding whether it needs to rebuild, the current dict implementation sees no difference between adding a new element and deleting an existing element Deleting leaves behind "holes" that periodically have to be squashed out of existence so that insertion order can be maintained in a dead simple way upon future additions. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BVCDKT5ULY324RZATMCEFGAMYOBJFHIZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2019-12-13 - 2019-12-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7200 (+26) closed 43662 (+45) total 50862 (+71) Open issues with patches: 2822 Issues opened (46) == #39041: Support GitHub Actions in CI https://bugs.python.org/issue39041 opened by steve.dower #39042: Use the runtime's main thread ID in the threading module. https://bugs.python.org/issue39042 opened by eric.snow #39046: collections.abc.Reversible should not be a subclass of Hashabl https://bugs.python.org/issue39046 opened by Zac Hatfield-Dodds #39047: TestTemporaryDirectory.test_flags fails on FreeBSD/ZFS https://bugs.python.org/issue39047 opened by attilajeges #39048: Reorder the __aenter__ and __aexit__ method checks for the asy https://bugs.python.org/issue39048 opened by maggyero #39050: The "Help" button in IDLE's config dialog does not work https://bugs.python.org/issue39050 opened by ZackerySpytz #39052: import error when in python -m pdb debug mode https://bugs.python.org/issue39052 opened by chengyang #39055: base64.b64decode() with validate=True does not raise for a tra https://bugs.python.org/issue39055 opened by serhiy.storchaka #39056: Issues with handling the -W option https://bugs.python.org/issue39056 opened by serhiy.storchaka #39057: Issues with urllib.request.proxy_bypass_environment https://bugs.python.org/issue39057 opened by serhiy.storchaka #39058: argparse should preserve argument ordering in Namespace https://bugs.python.org/issue39058 opened by rhettinger #39060: asyncio.Task.print_stack doesn't print the full stack https://bugs.python.org/issue39060 opened by amit7itz #39061: Garbage Collection makes some object live for very long https://bugs.python.org/issue39061 opened by mistasse #39062: ValueError in TarFile.getmembers https://bugs.python.org/issue39062 opened by jvoisin #39064: ValueError in zipfile.ZipFile https://bugs.python.org/issue39064 opened by jvoisin #39065: OSError in TarFile.getmembers() https://bugs.python.org/issue39065 opened by jvoisin #39067: EOFError in tarfile.open https://bugs.python.org/issue39067 opened by jvoisin #39068: Base 85 encoding initialization race condition https://bugs.python.org/issue39068 opened by drmonkeysee #39070: Uninstalling 3.8.0 fails but it says it succeeds.. https://bugs.python.org/issue39070 opened by tuijatuulia #39071: email.parser.BytesParser - parse and parsebytes work not equiv https://bugs.python.org/issue39071 opened by mkaiser #39072: Azure Pipelines: git clone failed with: OpenSSL SSL_read: Conn https://bugs.python.org/issue39072 opened by vstinner #39073: email incorrect handling of crlf in Address objects. https://bugs.python.org/issue39073 opened by jap #39074: Threading memory leak in _shutdown_locks for non-daemon thread https://bugs.python.org/issue39074 opened by krypticus #39075: types.SimpleNamespace should preserve attribute ordering (?) https://bugs.python.org/issue39075 opened by eric.snow #39076: Use types.SimpleNamespace for argparse.Namespace https://bugs.python.org/issue39076 opened by eric.snow #39078: __function.__defaults__ breaks for __init__ of dataclasses wit https://bugs.python.org/issue39078 opened by RunOrVeith #39082: AsyncMock is unable to correctly patch static or class methods https://bugs.python.org/issue39082 opened by czardoz #39085: Improve docs for await expression https://bugs.python.org/issue39085 opened by aeros #39087: [C API] No efficient C API to get UTF-8 string from unicode ob https://bugs.python.org/issue39087 opened by inada.naoki #39088: test_concurrent_futures crashed with python.core core dump on https://bugs.python.org/issue39088 opened by vstinner #39089: Update IDLE's credits https://bugs.python.org/issue39089 opened by taleinat #39090: Document various options for getting the absolute path from pa https://bugs.python.org/issue39090 opened by brett.cannon #39091: CPython Segfault in 5 lines of code https://bugs.python.org/issue39091 opened by skrause #39092: Csv sniffer doesn't attempt to determine and set escape charac https://bugs.python.org/issue39092 opened by evan.whitfield #39093: tkinter objects garbage collected from non-tkinter thread caus https://bugs.python.org/issue39093 opened by obserience #39096: Description of "Format Specification Mini-Language" not accura https://bugs.python.org/issue39096 opened by mamrhein #39098: OSError: handle is closed in ProcessPoolExecutor on shutdown( https://bugs.python.org/issue39098 opened by patbuxton #39100: email.policy.SMTP throws AttributeError on invalid header https://bugs.python.org/issue39100 opened by elenril #39101: IsolatedAsyncioTestCase freezes when exception is raised https://bugs.python.org/issue39101 opened by fornellas #39102: Increase Enum performance https://bugs.python.
[Python-Dev] Re: Should set objects maintain insertion order too?
On Sat, Dec 21, 2019 at 3:17 AM Tim Peters wrote: > > [Wes Turner ] > >> How slow and space-inefficient would it be to just implement the set > >> methods > >> on top of dict? > > [Inada Naoki ] > > Speed: Dict doesn't cache the position of the first item. Calling > > next(iter(D)) repeatedly is O(N) in worst case. > > ... > > See also Raymond's (only) message in this thread. We would also lose > low-level speed optimizations specific to the current set > implementation. I read this thread, and I understand all of them. I just meant the performance of the next(iter(D)) is the most critical part when you implement orderdset on top of the current dict and use it as a queue. This code should be O(N), but it's O(N^2) if q is implemented on top of the dict. while q: item = q.popleft() Sorry for the confusion. > > And the current set implementation (like the older dict > implementation) never needs to "rebuild the table from scratch" unless > the cardinality of the set keeps growing. It is a bit misleading. If "the cardinality of the set" means len(S), set requires the rebuild in low frequency if the its items are random. Anyway, it is a smaller problem than next(iter(D)) because of the amortized cost is O(1). Current dict is not optimized for queue usage. Regards, -- Inada Naoki ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DFJQOW225OSRPFDHBJ3UBYRRMZ52AXDH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should set objects maintain insertion order too?
[Inada Naoki ] > I just meant the performance of the next(iter(D)) is the most critical part > when you implement orderdset on top of the current dict and use it as a queue. Which is a good point. I added a lot more, though, because Wes didn't even mention queues in his question: [Wes Turner ] How slow and space-inefficient would it be to just implement the set methods on top of dict? I can't guess what he was _really_ asking about, so now he's got lots of answers ;-) > This code should be O(N), but it's O(N^2) if q is implemented on top > of the dict. > >while q: >item = q.popleft() > > Sorry for the confusion. To flesh this out, the FIFO queue entries are all together in a contiguous vector, with no "holes" in the middle. Pushing a new item appends "to the right" (higher index in the vector), while popping an item leaves "a hole" at the left. But there are no holes "in the middle" in this case. So the first real entry is at a higher index with every pop, but iteration starts at index 0 every time. The more pops there are, the more holes iteration has to skip over, one at a time, to find the first real entry remaining. Until pushing a new item would fall off the right end of the vector. Then the table is rebuilt from scratch, moving all the real entries to form a contiguous block starting at index 0 again. >> And the current set implementation (like the older dict >> implementation) never needs to "rebuild the table from scratch" unless >> the cardinality of the set keeps growing. > It is a bit misleading. If "the cardinality of the set" means len(S), Yes. > set requires the rebuild in low frequency if the its items are random. > > Anyway, it is a smaller problem than next(iter(D)) because of the > amortized cost is O(1). For the specific case of FIFO queues, sure. In general, though, "O(1) period". is more desirable than "amortized O(1)". This is especially true when sets get very large. > Current dict is not optimized for queue usage. Nor should it be:-) That's what collections.deque is for, pushes and pops, at both ends, always time O(1) period. No holes, no internal rearrangements, and O(1) "wasted" space. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5VDU52JX2GFM6546W6FCFKXIODDAQKP4/ Code of Conduct: http://python.org/psf/codeofconduct/
