[Python-Dev] When val=b'', but val == b'' returns False - bytes initialization
Hello, I am not sure if this is expected behaviour, or a bug. In a C extension module, if I create and return an empty bytes object like this: val = PyBytes_FromStringAndSize (NULL, 20); Py_SIZE(val) = 0; Then from the Python interpreter's perspective: isinstance(val, bytes) returns True print(val) returns b'' print(repr(val)) returns b'' BUT val == b'' returns False. On the other hand, initializing the underlying memory: val = PyBytes_FromStringAndSize (NULL, 20); PyBytes_AS_STRING (val); c[0] = '\0'; Py_SIZE(val) = 0; Then, from the Python interpreter, val == b'' returns True, as expected. So, my question is: is this the expected behaviour, or a bug? I was slightly surprised to have to initialize the storage. On the other hand, I can perhaps also see it might be expected, since the docs do say that PyBytes_FromStringAndSize will not initialize the underlying storage. Please cc me on any replies - am not subscribed to the list. Many thanks, Jonathan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] When val=b'', but val == b'' returns False - bytes initialization
On Wed, 27 Dec 2017 14:19:16 + Jonathan Underwood wrote: > Hello, > > I am not sure if this is expected behaviour, or a bug. > > In a C extension module, if I create and return an empty bytes object like > this: > > val = PyBytes_FromStringAndSize (NULL, 20); > Py_SIZE(val) = 0; I wouldn't call it "expected", but bytes objects are supposed to be NULL-terminated internally, so your code technically creates an invalid bytes object. The NULL-terminated constraint may be relied on by some code, for example when the string gets passed to a third-party C function. Perhaps that should be mentioned in the C API docs. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Documenting types outside of typing
In his review of PR#4911, Antoine points to the documentation of two type definitions in importlib.resources, Package and Resource. https://github.com/python/cpython/pull/4911/files#diff-2a479c407f7177f3d7cb876f244e47bcR804 One question is what markup to use for type definitions. I’m using class:: because that’s what’s used in typing and there doesn’t seem to be any better alternative. More to the point, Antoine questions whether these two types should be documented at all: https://github.com/python/cpython/pull/4911#discussion_r158801065 "What I mean is that a class is supposed to specify concrete behaviour, but being a type, Package doesn't have any methods or attributes of its own. So I don't see the point of mentioning it in the docs.” I suggest that they are worth documenting because they help to organize the discussion about what API is expected from the arguments to the functions, without having to duplicate that information in every function description. I also think that since you’ll see those types in the code, they are worth documenting. I don’t think you *lose* anything by including their documentation. But Antoine makes a good point that we probably don’t have a lot of precedence here, so suggests we discuss it on python-dev to come up with some useful conventions. I haven’t kept up on the dataclasses discussion, but given that types are important in that API too, have the same issues come up there and if so, how are they being handled? Cheers, -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] When val=b'', but val == b'' returns False - bytes initialization
On Wed, 27 Dec 2017 17:28:41 +0100 Antoine Pitrou wrote: > On Wed, 27 Dec 2017 14:19:16 + > Jonathan Underwood wrote: > > Hello, > > > > I am not sure if this is expected behaviour, or a bug. > > > > In a C extension module, if I create and return an empty bytes object like > > this: > > > > val = PyBytes_FromStringAndSize (NULL, 20); > > Py_SIZE(val) = 0; > > I wouldn't call it "expected", but bytes objects are supposed to be > NULL-terminated internally, so your code technically creates an invalid > bytes object. The NULL-terminated constraint may be relied on by some > code, for example when the string gets passed to a third-party C > function. Note this is really happening because you're allocating a 20-long bytes object and then shortening it to 0 bytes. PyBytes_FromStringAndSize() already stored a NULL byte in the 21st place. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting functools.singledispatch with classes.
Okay, if there is no further feedback, I will work on a singledispatchmethod decorator like partialmethod. For the future perhaps, would it not be possible to tell that the passed argument is a descriptor/function and dispatch to the correct implementation, thus not needing two functions for essentially the same thing? It seems more straightforward to make the implementation a bit more complex to provide a single, simple API to users. Cheers, Ethan On Tue, Dec 26, 2017 at 3:29 PM, Ivan Levkivskyi wrote: > On 26 December 2017 at 01:41, Nick Coghlan wrote: > >> On 25 December 2017 at 12:32, Ethan Smith wrote: >> > So at the moment, I don't think it is possible to implement >> singledispatch >> > on classmethod or staticmethod decorated functions. >> >> I've posted this to the PR, but adding it here as well: I think this >> is a situation very similar to the case with functools.partialmethod, >> where you're going to need to write a separate >> functools.singledispatchmethod class that's aware of the descriptor >> protocol, rather than trying to add the functionality directly to >> functools.singledispatch. >> > > I agree with Nick here. Adding a separate decorator looks like the right > approach, > especially taking into account the precedent of @partialmethod. > > -- > Ivan > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documenting types outside of typing
FWIW the same problem was discussed a year ago when documenting typing. At that time the discussion was not conclusive, so that some types use class:: directive while other use data:: directive. At that time Guido was in favour of data:: and now in view of PEP 560 many types in typing will stop being class objects, and will be just (compact) objects. Therefore, my understanding is that all special forms like Union, Any, ClassVar, etc. will use data:: in the docs. Concerning the question whether it makes to document types, I think it makes sense if it is a publicly available type (or type alias) that will be useful to annotate user code. -- Ivan On 27 December 2017 at 17:41, Barry Warsaw wrote: > In his review of PR#4911, Antoine points to the documentation of two type > definitions in importlib.resources, Package and Resource. > > https://github.com/python/cpython/pull/4911/files#diff- > 2a479c407f7177f3d7cb876f244e47bcR804 > > One question is what markup to use for type definitions. I’m using > class:: because that’s what’s used in typing and there doesn’t seem to be > any better alternative. > > More to the point, Antoine questions whether these two types should be > documented at all: > > https://github.com/python/cpython/pull/4911#discussion_r158801065 > > "What I mean is that a class is supposed to specify concrete behaviour, > but being a type, Package doesn't have any methods or attributes of its > own. So I don't see the point of mentioning it in the docs.” > > I suggest that they are worth documenting because they help to organize > the discussion about what API is expected from the arguments to the > functions, without having to duplicate that information in every function > description. I also think that since you’ll see those types in the code, > they are worth documenting. I don’t think you *lose* anything by including > their documentation. > > But Antoine makes a good point that we probably don’t have a lot of > precedence here, so suggests we discuss it on python-dev to come up with > some useful conventions. I haven’t kept up on the dataclasses discussion, > but given that types are important in that API too, have the same issues > come up there and if so, how are they being handled? > > Cheers, > -Barry > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > levkivskyi%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 567 v2
This is a second version of PEP 567.
A few things have changed:
1. I now have a reference implementation:
https://github.com/python/cpython/pull/5027
2. The C API was updated to match the implementation.
3. The get_context() function was renamed to copy_context() to better
reflect what it is really doing.
4. Few clarifications/edits here and there to address earlier feedback.
Yury
PEP: 567
Title: Context Variables
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Dec-2017
Python-Version: 3.7
Post-History: 12-Dec-2017, 28-Dec-2017
Abstract
This PEP proposes a new ``contextvars`` module and a set of new
CPython C APIs to support context variables. This concept is
similar to thread-local storage (TLS), but, unlike TLS, it also allows
correctly keeping track of values per asynchronous task, e.g.
``asyncio.Task``.
This proposal is a simplified version of :pep:`550`. The key
difference is that this PEP is concerned only with solving the case
for asynchronous tasks, not for generators. There are no proposed
modifications to any built-in types or to the interpreter.
This proposal is not strictly related to Python Context Managers.
Although it does provide a mechanism that can be used by Context
Managers to store their state.
Rationale
=
Thread-local variables are insufficient for asynchronous tasks that
execute concurrently in the same OS thread. Any context manager that
saves and restores a context value using ``threading.local()`` will
have its context values bleed to other code unexpectedly when used
in async/await code.
A few examples where having a working context local storage for
asynchronous code is desirable:
* Context managers like ``decimal`` contexts and ``numpy.errstate``.
* Request-related data, such as security tokens and request
data in web applications, language context for ``gettext``, etc.
* Profiling, tracing, and logging in large code bases.
Introduction
The PEP proposes a new mechanism for managing context variables.
The key classes involved in this mechanism are ``contextvars.Context``
and ``contextvars.ContextVar``. The PEP also proposes some policies
for using the mechanism around asynchronous tasks.
The proposed mechanism for accessing context variables uses the
``ContextVar`` class. A module (such as ``decimal``) that wishes to
store a context variable should:
* declare a module-global variable holding a ``ContextVar`` to
serve as a key;
* access the current value via the ``get()`` method on the
key variable;
* modify the current value via the ``set()`` method on the
key variable.
The notion of "current value" deserves special consideration:
different asynchronous tasks that exist and execute concurrently
may have different values for the same key. This idea is well-known
from thread-local storage but in this case the locality of the value is
not necessarily bound to a thread. Instead, there is the notion of the
"current ``Context``" which is stored in thread-local storage, and
is accessed via ``contextvars.copy_context()`` function.
Manipulation of the current ``Context`` is the responsibility of the
task framework, e.g. asyncio.
A ``Context`` is conceptually a read-only mapping, implemented using
an immutable dictionary. The ``ContextVar.get()`` method does a
lookup in the current ``Context`` with ``self`` as a key, raising a
``LookupError`` or returning a default value specified in
the constructor.
The ``ContextVar.set(value)`` method clones the current ``Context``,
assigns the ``value`` to it with ``self`` as a key, and sets the
new ``Context`` as the new current ``Context``.
Specification
=
A new standard library module ``contextvars`` is added with the
following APIs:
1. ``copy_context() -> Context`` function is used to get a copy of
the current ``Context`` object for the current OS thread.
2. ``ContextVar`` class to declare and access context variables.
3. ``Context`` class encapsulates context state. Every OS thread
stores a reference to its current ``Context`` instance.
It is not possible to control that reference manually.
Instead, the ``Context.run(callable, *args, **kwargs)`` method is
used to run Python code in another context.
contextvars.ContextVar
--
The ``ContextVar`` class has the following constructor signature:
``ContextVar(name, *, default=_NO_DEFAULT)``. The ``name`` parameter
is used only for introspection and debug purposes, and is exposed
as a read-only ``ContextVar.name`` attribute. The ``default``
parameter is optional. Example::
# Declare a context variable 'var' with the default value 42.
var = ContextVar('var', default=42)
(The ``_NO_DEFAULT`` is an internal sentinel object used to
detect if the default value was provided.)
``ContextVar.get()`` returns a value for context variable from the
current ``Context``::
# Get t
