Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Victor Stinner]

Le jeu. 11 avr. 2019 à 17:33, Serhiy Storchaka  a écrit :
> If reducing the Python memory footprint is an argument for disabling
> Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> the debug build.

The "serialno" field of debug memory allocators is documented as: "an
excellent way to set a breakpoint on the next run, to capture the
instant at which this block was passed out." I'm debugging crashes and
memory leaks in CPython for 10 years, and I simply never had to use
"serialno". I wrote https://bugs.python.org/issue36611 to remove the
serialno field of debug hooks on Python memory allocators: it reduces
the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running
test_asyncio).

Python is used on devices with low memory (ex: 256 MiB for the whole
system). Allowing developers to use a debug build on such devices seem
to be a legit rationale for such change. The debug build is very
useful to identify bugs in C extensions.


> On other hand, since using the debug allocator doesn't cause problems
> with compatibility, it may be possible to use similar technique for the
> objects double list. Although this is not easy because of objects placed
> at static memory.

I'm not sure of what you means by "objects placed at static memory":
the double linked list of all Python objects is created at runtime.
_ob_next and _ob_prev are initialized statically to NULL.

I would be interested if Py_TRACE_REFS could be reimplemented in a
more dynamic fashion. Even if it would still require a debug build, it
would be nice to be able to "opt-in" for this feature (have it
disabled by default, again, to reduce the overhead and reduce the
memory footprint), as tracemalloc which plugs itself into memory
allocators to attach traces to memory blocks.

Except Guido who wrote "I recall finding memory leaks using this.
(E.g. I remember a leak in Zope due to a cache that was never pruned.)
But presumably gc.get_objects() would have been sufficient. (IIRC it
didn't exist at the time.)", at this point, nobody said that they use
Py_TRACE_REFS. So I'm not sure that it's worth it to invest time on a
feature if nobody uses it?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Victor Stinner]

Le ven. 12 avr. 2019 à 12:57, Victor Stinner  a écrit :
> I wrote https://bugs.python.org/issue36611 to remove the serialno field
> of debug hooks on Python memory allocators: it reduces
> the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running
> test_asyncio).

I measured the memory footprint when I combine my two changes:

* disable Py_TRACE_REFS: https://bugs.python.org/issue36465
* disable/remove serialno field: https://bugs.python.org/issue36611

python3 -m test test_asyncio, without => with the change: 34,038.0 kB
=> 30,612.2 kB (-3,425.8 kiB, -10%)

A reduction of 3.4 MiB on 34.0 MiB is quite significant, no?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-582 and multiple Python installations

[Victor Stinner]

Hi,

Le mar. 2 avr. 2019 à 17:20, Calvin Spealman  a écrit :
> While the PEP does show the version number as part of the path to the actual 
> packages, implying support for multiple versions, this doesn't seem to be 
> spelled out in the actual text. Presumably __pypackages__/3.8/ might sit 
> beside __pypackages__/3.9/, etc. to keep future versions capable of 
> installing packages for each version, the way virtualenv today is bound to 
> one version of Python.
>
>
> I'd like to raise a potential edge case that might be a problem, and likely 
> an increasingly common one: users with multiple installations of the *same* 
> version of Python.

Hum, I don't know if it's relevant to support multiple Python binaries
of the same Python version, but just in case, let me share my
experience with that in the pyperformance project.

The pyperformance project uses virtual environment for two binaries of
the exact Python version (and usually the same path!): one unpatched
"reference" and one "patched" binary, to experiment an optimization. I
needed a way to build a short text identifier to still be able to get
a "cached" virtual environment per Python binary. I wrote a short code
to generate the identifier using:

* pyperformance version
* requirements.txt
* sys.executable
* sys.version
* sys.version_info
* sys.implementation.name of platform.python_implementation()

The script builds a long string using these info, hash it with SHA1
and take first 12 characters of the hexadecimal format of the hash.

Script:
---
import hashlib
import platform
import sys

performance_version = sys.argv[1]
requirements = sys.argv[2]

data = performance_version + sys.executable + sys.version

pyver= sys.version_info

if hasattr(sys, 'implementation'):
# PEP 421, Python 3.3
implementation = sys.implementation.name
else:
implementation = platform.python_implementation()
implementation = implementation.lower()

if not isinstance(data, bytes):
data = data.encode('utf-8')
with open(requirements, 'rb') as fp:
data += fp.read()
sha1 = hashlib.sha1(data).hexdigest()

name = ('%s%s.%s-%s'
% (implementation, pyver.major, pyver.minor, sha1[:12]))
print(name)
---

Examples:

$ touch requirements.txt # empty file
$ python3.7 x.py version requirements.txt
cpython3.7-502d35b8e005
$ python3.6 x.py version requirements.txt
cpython3.6-7f4febbec0be

$ python3 x.py version requirements.txt
cpython3.7-59ab636dfacb
$ file /usr/bin/python3
/usr/bin/python3: symbolic link to python3.7

Hum, python3 and python3.7 produce the different hash whereas it's the
same binary. Maybe os.path.realpath() should be called on
sys.executable :-)

Victor

--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Stefan Behnel]

Serhiy Storchaka schrieb am 11.04.19 um 17:30:
> If reducing the Python memory footprint is an argument for disabling
> Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> the debug build.

I think what Victor is argueing is rather that we have better ways to debug
memory problems these days, so we might be able to get rid of a relict that
no-one is using (or should be using) anymore and that has its drawbacks
(such as a very different ABI and higher memory load).

I don't really have an opinion here, but I can at least say that I never
found a use case for Py_TRACE_REFS myself and therefore certainly wouldn't
miss it.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposal: dict.with_values(iterable)

[Inada Naoki]

Hi, all.

I propose adding new method: dict.with_values(iterable)

# Motivation

Python is used to handle data.
While dict is not efficient way to handle may records, it is still
convenient way.

When creating many dicts with same keys, dict need to
lookup internal hash table while inserting each keys.

It is costful operation.  If we can reuse existing keys of dict,
we can skip this inserting cost.

Additionally, we have "Key-Sharing Dictionary (PEP 412)".
When all keys are string, many dict can share one key.
It reduces memory consumption.

This might be usable for:

* csv.DictReader
* namedtuple._asdict()
* DB-API 2.0 implementations:  (e.g. DictCursor of mysqlclient-python)


# Draft implementation

pull request: https://github.com/python/cpython/pull/12802

with_values(self, iterable, /)
Create a new dictionary with keys from this dict and values from iterable.

When length of iterable is different from len(self), ValueError is raised.
This method does not support dict subclass.


## Memory usage (Key-Sharing dict)

>>> import sys
>>> keys = tuple("abcdefg")
>>> keys
('a', 'b', 'c', 'd', 'e', 'f', 'g')
>>> d = dict(zip(keys, range(7)))
>>> d
{'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6}
>>> sys.getsizeof(d)
360

>>> keys = dict.fromkeys("abcdefg")
>>> d = keys.with_values(range(7))
>>> d
{'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6}
>>> sys.getsizeof(d)
144

## Speed

$ ./python -m perf timeit -o zip_dict.json -s 'keys =
tuple("abcdefg"); values=[*range(7)]' 'dict(zip(keys, values))'

$ ./python -m perf timeit -o with_values.json -s 'keys =
dict.fromkeys("abcdefg"); values=[*range(7)]'
'keys.with_values(values)'

$ ./python -m perf compare_to zip_dict.json with_values.json
Mean +- std dev: [zip_dict] 935 ns +- 9 ns -> [with_values] 109 ns +-
2 ns: 8.59x faster (-88%)


How do you think?
Any comments are appreciated.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: dict.with_values(iterable)

[Victor Stinner]

Nice optimization! I have questions on the proposed API.

> with_values(self, iterable, /)
> Create a new dictionary with keys from this dict and values from iterable.
>
> When length of iterable is different from len(self), ValueError is raised.
> This method does not support dict subclass.

In short, mydict.with_values(values) behaves as
dict(zip(mydict.keys(), values)), but is more efficient?

The method rely on the fact that dict is preserving key insertion order, right?


Le ven. 12 avr. 2019 à 15:47, Inada Naoki  a écrit :
> This might be usable for:
>
> * csv.DictReader
> * namedtuple._asdict()
> * DB-API 2.0 implementations:  (e.g. DictCursor of mysqlclient-python)

I guess that a new dict constructor taken keys and values like
dict.from_keys_and_values(keys, values) would work, but would not
benefit from the dict key-sharing optimization?

Would it be possible to implement the key-sharing optimization using a
dict.from_keys_and_values(mydict.keys(), values) method: detect that
keys are owned by a dict, and so create a new dict linked to the keys
dict? A dict view contains a reference to the iterated dict
(dictiterobject.di_dict).

I'm fine with dict.with_values() API, but I'm asking if it could be
written differently.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Guido van Rossum]

On Fri, Apr 12, 2019 at 5:51 AM Stefan Behnel  wrote:

> Serhiy Storchaka schrieb am 11.04.19 um 17:30:
> > If reducing the Python memory footprint is an argument for disabling
> > Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> > the debug build.
>
> I think what Victor is argueing is rather that we have better ways to debug
> memory problems these days, so we might be able to get rid of a relict that
> no-one is using (or should be using) anymore and that has its drawbacks
> (such as a very different ABI and higher memory load).
>
> I don't really have an opinion here, but I can at least say that I never
> found a use case for Py_TRACE_REFS myself and therefore certainly wouldn't
> miss it.
>

I have a feeling that at some point someone might want to use this to debug
some leak (presumably caused by C code) beyond what gc.get_objects() can
report. But I agree that it isn't useful to the vast majority of users of a
regular debug build. So let's leave it off by default even in debug builds.
But let's not delete the macros.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Miro Hrončok]

Hello.

Based on discussions in [1], Petr Viktorin and me have drafted a new update [2] 
to the PEP 394 (The "python" Command on Unix-Like Systems).


The update gives distributors the opportunity to decide where does the "python" 
command lead to, whether it is present etc.


Please, see the PR [2] for the suggested changes.

[1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html
[2]: https://github.com/python/peps/pull/989

Thanks,
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Petr Viktorin]

On 4/12/19 4:53 PM, Miro Hrončok wrote:

Hello.

Based on discussions in [1], Petr Viktorin and me have drafted a new 
update [2] to the PEP 394 (The "python" Command on Unix-Like Systems).


The update gives distributors the opportunity to decide where does the 
"python" command lead to, whether it is present etc.


Please, see the PR [2] for the suggested changes.

[1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html
[2]: https://github.com/python/peps/pull/989


The text is available at 
https://github.com/hroncok/peps/blob/pep394-2019/pep-0394.txt


As a summary, I'll paste the rationale sections here:

History of this PEP
===

In 2011, the majority of distributions
aliased the ``python`` command to Python 2, but some started switching it to
Python 3 ([5]_). As some of the former distributions did not provide a
``python2`` command by default, there was previously no way for Python 2 
code

(or any code that invokes the Python 2 interpreter directly rather than via
``sys.executable``) to reliably run on all Unix-like systems without
modification, as the ``python`` command would invoke the wrong interpreter
version on some systems, and the ``python2`` command would fail completely
on others. This PEP originally provided a very simple mechanism
to restore cross-platform support, with minimal additional work required
on the part of distribution maintainers. Simplified, the recommendation was:

1. The ``python`` command was preferred for code compatible with both
   Python 2 and 3 (since it was available on all systems, even those that
   already aliased it to Python 3).
2. The ``python`` command should always invoke Python 2 (to prevent
   hard-to-diagnose errors when Python 2 code is run on Python 3).
3. The ``python2`` and ``python3`` commands should be available to specify
   the version explicitly.

However, these recommendations implicitly assumed that Python 2 would 
always be
available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 
404),

distributions are making Python 2 optional or removing entirely.
This means either removing the ``python`` command or switching it to invoke
Python 3, invalidating respectively the first or second recommendation.
Also, some distributors decided that their users are better served by
ignoring the PEP's recommendations, making the PEP's supposedly
cross-platform recommendations on ``python`` and ``python2`` in shebangs
increasingly unreliable.


Current Rationale
=

As of 2019, nearly all new systems include Python 3 and the ``python3``
command. This makes the ``python3`` command the best general choice for
code that can run on either Python 3.x or 2.x, even though it is not
available everywhere.

The recommendation is skewed toward current and future systems, leaving
behind “*old systems*” (like RHEL 6 or default Python installed on macOS).
On these systems, Python software is rarely updated and any recommendations
this PEP makes would likely be ignored.

Also, since distributors often ignored recommendations the PEP gave
regarding the ``python`` command (for what they saw as legitimate special
needs), this PEP now gives them broad control over the command.
Correspondingly, users are advised to not use the ``python`` command
in cross-platform code.
Instead, this PEP specifies the expected behavior of the ``python3`` and
``python2`` commands, which is not controversial.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Guido van Rossum]

I think this is reasonable. Thanks for making the rationale clear!

On Fri, Apr 12, 2019 at 8:02 AM Petr Viktorin  wrote:

> On 4/12/19 4:53 PM, Miro Hrončok wrote:
> > Hello.
> >
> > Based on discussions in [1], Petr Viktorin and me have drafted a new
> > update [2] to the PEP 394 (The "python" Command on Unix-Like Systems).
> >
> > The update gives distributors the opportunity to decide where does the
> > "python" command lead to, whether it is present etc.
> >
> > Please, see the PR [2] for the suggested changes.
> >
> > [1]:
> https://mail.python.org/pipermail/python-dev/2019-February/156272.html
> > [2]: https://github.com/python/peps/pull/989
>
> The text is available at
> https://github.com/hroncok/peps/blob/pep394-2019/pep-0394.txt
>
> As a summary, I'll paste the rationale sections here:
>
> History of this PEP
> ===
>
> In 2011, the majority of distributions
> aliased the ``python`` command to Python 2, but some started switching it
> to
> Python 3 ([5]_). As some of the former distributions did not provide a
> ``python2`` command by default, there was previously no way for Python 2
> code
> (or any code that invokes the Python 2 interpreter directly rather than via
> ``sys.executable``) to reliably run on all Unix-like systems without
> modification, as the ``python`` command would invoke the wrong interpreter
> version on some systems, and the ``python2`` command would fail completely
> on others. This PEP originally provided a very simple mechanism
> to restore cross-platform support, with minimal additional work required
> on the part of distribution maintainers. Simplified, the recommendation
> was:
>
> 1. The ``python`` command was preferred for code compatible with both
> Python 2 and 3 (since it was available on all systems, even those that
> already aliased it to Python 3).
> 2. The ``python`` command should always invoke Python 2 (to prevent
> hard-to-diagnose errors when Python 2 code is run on Python 3).
> 3. The ``python2`` and ``python3`` commands should be available to specify
> the version explicitly.
>
> However, these recommendations implicitly assumed that Python 2 would
> always be
> available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP
> 404),
> distributions are making Python 2 optional or removing entirely.
> This means either removing the ``python`` command or switching it to invoke
> Python 3, invalidating respectively the first or second recommendation.
> Also, some distributors decided that their users are better served by
> ignoring the PEP's recommendations, making the PEP's supposedly
> cross-platform recommendations on ``python`` and ``python2`` in shebangs
> increasingly unreliable.
>
>
> Current Rationale
> =
>
> As of 2019, nearly all new systems include Python 3 and the ``python3``
> command. This makes the ``python3`` command the best general choice for
> code that can run on either Python 3.x or 2.x, even though it is not
> available everywhere.
>
> The recommendation is skewed toward current and future systems, leaving
> behind “*old systems*” (like RHEL 6 or default Python installed on macOS).
> On these systems, Python software is rarely updated and any recommendations
> this PEP makes would likely be ignored.
>
> Also, since distributors often ignored recommendations the PEP gave
> regarding the ``python`` command (for what they saw as legitimate special
> needs), this PEP now gives them broad control over the command.
> Correspondingly, users are advised to not use the ``python`` command
> in cross-platform code.
> Instead, this PEP specifies the expected behavior of the ``python3`` and
> ``python2`` commands, which is not controversial.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Serhiy Storchaka]

12.04.19 17:40, Guido van Rossum пише:
So let's leave it off 
by default even in debug builds. But let's not delete the macros.


Maybe switch it on (together with other disabled by default options) on 
some fast buildbot?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: dict.with_values(iterable)

[Serhiy Storchaka]

12.04.19 16:44, Inada Naoki пише:

When creating many dicts with same keys, dict need to
lookup internal hash table while inserting each keys.

It is costful operation.  If we can reuse existing keys of dict,
we can skip this inserting cost.

Additionally, we have "Key-Sharing Dictionary (PEP 412)".
When all keys are string, many dict can share one key.
It reduces memory consumption.


It looks contrary to simplification made in Python 3 when we get rid of 
some more efficient lists in favor of more general iterators.


If this is a common case we can add an invisible optimization for 
dict(zip(keys, values)), especially if keys is a key-sharing dictionary. 
This will benefit all users without the need to rewrite the code to use 
the new special method.


The interface of dict is already overloaded. It contains many methods 
which most users use rarely (and therefore which are not kept in the 
working set of memory).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: dict.with_values(iterable)

[Inada Naoki]

On Fri, Apr 12, 2019 at 11:31 PM Victor Stinner  wrote:
>
> Nice optimization! I have questions on the proposed API.
>
> > with_values(self, iterable, /)
> > Create a new dictionary with keys from this dict and values from 
> > iterable.
> >
> > When length of iterable is different from len(self), ValueError is 
> > raised.
> > This method does not support dict subclass.
>
> In short, mydict.with_values(values) behaves as
> dict(zip(mydict.keys(), values)), but is more efficient?

Yes. But unlike zip, keys() and values must have exactly same length.

>
> The method rely on the fact that dict is preserving key insertion order, 
> right?
>

Yes.

> > This might be usable for:
> >
> > * csv.DictReader
> > * namedtuple._asdict()
> > * DB-API 2.0 implementations:  (e.g. DictCursor of mysqlclient-python)
>
> I guess that a new dict constructor taken keys and values like
> dict.from_keys_and_values(keys, values) would work, but would not
> benefit from the dict key-sharing optimization?
>

I don't like more overloading.
And this API is specialized to build multiple dicts, not one dict.
So I want to have dedicated API for it.

> Would it be possible to implement the key-sharing optimization using a
> dict.from_keys_and_values(mydict.keys(), values) method: detect that
> keys are owned by a dict, and so create a new dict linked to the keys
> dict? A dict view contains a reference to the iterated dict
> (dictiterobject.di_dict).

I think it is possible.
>
> I'm fine with dict.with_values() API, but I'm asking if it could be
> written differently.
>
> Victor

I implemented it as instance method of dict
because it may modify the dict internally (at first invocation).

-- 
Inada Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: dict.with_values(iterable)

[Inada Naoki]

On Sat, Apr 13, 2019 at 12:38 AM Serhiy Storchaka  wrote:
>
> It looks contrary to simplification made in Python 3 when we get rid of
> some more efficient lists in favor of more general iterators.
>

Yes.  This is API for special use case creates many dict having
same keys, like csv.DictReader.
It is not good design for general purpose.

strings module has strings.Template class.  But I don't want to add
dicts module.
Maybe, collections.DictBuilder may be another option.  e.g.

>>> from collections import DictBuilder
>>> builder = DictBuilder(tuple("abc"))
>>> builder.build(range(3))
{"a": 0, "b": 1, "c": 2}


> If this is a common case we can add an invisible optimization for
> dict(zip(keys, values)), especially if keys is a key-sharing dictionary.
> This will benefit all users without the need to rewrite the code to use
> the new special method.

But this optimization may slow down when creating one dict...

>
> The interface of dict is already overloaded. It contains many methods
> which most users use rarely (and therefore which are not kept in the
> working set of memory).

Yes.

-- 
Inada Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Removing PID check from signal handler

[Jeroen Demeyer]

The signal handler (that receives signals from the OS) in Python starts 
with a check


if (getpid() == main_pid)

Looking at the comments, the intent was to do a check for the main 
*thread* but this is checking the *process* id. So this condition is 
basically always true. Therefore, I suggest to remove it in 
https://bugs.python.org/issue36601


If you have any objections or comments, I suggest to post them to that bpo.


Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: dict.with_values(iterable)

[Brett Cannon]

On Fri, Apr 12, 2019 at 8:35 AM Serhiy Storchaka 
wrote:

> 12.04.19 16:44, Inada Naoki пише:
> > When creating many dicts with same keys, dict need to
> > lookup internal hash table while inserting each keys.
> >
> > It is costful operation.  If we can reuse existing keys of dict,
> > we can skip this inserting cost.
> >
> > Additionally, we have "Key-Sharing Dictionary (PEP 412)".
> > When all keys are string, many dict can share one key.
> > It reduces memory consumption.
>
> It looks contrary to simplification made in Python 3 when we get rid of
> some more efficient lists in favor of more general iterators.
>
> If this is a common case


I think that "if" is my big sticking point. I don't think I've ever had a
need for this and the zip() solution was what I originally thought of when
I realized what the method was meant to do (which wasn't obvious to me
initially).

This doesn't strike me as needing an optimization through a dedicated
method.

-Brett


> we can add an invisible optimization for
> dict(zip(keys, values)), especially if keys is a key-sharing dictionary.
> This will benefit all users without the need to rewrite the code to use
> the new special method.
>
> The interface of dict is already overloaded. It contains many methods
> which most users use rarely (and therefore which are not kept in the
> working set of memory).
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing PID check from signal handler

[Steve Dower]

On 12Apr.2019 0919, Jeroen Demeyer wrote:
> The signal handler (that receives signals from the OS) in Python starts
> with a check
> 
>     if (getpid() == main_pid)
> 
> Looking at the comments, the intent was to do a check for the main
> *thread* but this is checking the *process* id. So this condition is
> basically always true. Therefore, I suggest to remove it in
> https://bugs.python.org/issue36601
> 
> If you have any objections or comments, I suggest to post them to that bpo.

To add a little more context, the check was added about 25 years ago as
a "hack" for some reason that we can't figure out anymore.

So if you are a historian of ancient operating systems and know of one
that might have raised signal handlers in a different process from the
one where it was registered, we'd love to hear from you.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2019-04-05 - 2019-04-12)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7063 ( +7)
  closed 41307 (+76)
  total  48370 (+83)

Open issues with patches: 2815 


Issues opened (60)
==

#25160: Stop using deprecated imp module; imp should now emit a real D
https://bugs.python.org/issue25160  reopened by ncoghlan

#35934: Add socket.create_server() utility function
https://bugs.python.org/issue35934  reopened by vstinner

#36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var 
https://bugs.python.org/issue36235  reopened by vstinner

#36537: except statement block incorrectly assumes end of scope(?).
https://bugs.python.org/issue36537  opened by Saim Raza

#36538: _thread.interrupt_main() no longer interrupts Lock.wait
https://bugs.python.org/issue36538  opened by gregory.p.smith

#36540: PEP 570: Python Positional-Only Parameters
https://bugs.python.org/issue36540  opened by pablogsal

#36541: Make lib2to3 grammar more closely match Python
https://bugs.python.org/issue36541  opened by thatch

#36542: Allow to overwrite the signature for Python functions
https://bugs.python.org/issue36542  opened by serhiy.storchaka

#36543: Remove old-deprecated ElementTree features (part 2)
https://bugs.python.org/issue36543  opened by serhiy.storchaka

#36545: Python 3.5 OOM during test_socket on make
https://bugs.python.org/issue36545  opened by dekken

#36546: Add quantiles() to the statistics module
https://bugs.python.org/issue36546  opened by rhettinger

#36548: Make the repr of re flags more readable
https://bugs.python.org/issue36548  opened by serhiy.storchaka

#36550: Avoid creating AttributeError exceptions in the debugger
https://bugs.python.org/issue36550  opened by blueyed

#36551: Optimize list comprehensions with preallocate size and protect
https://bugs.python.org/issue36551  opened by anthony shaw

#36552: Replace OverflowError with ValueError when calculating length 
https://bugs.python.org/issue36552  opened by anthony shaw

#36553: inspect.is_decorator_call(frame)
https://bugs.python.org/issue36553  opened by smarie

#36556: Trashcan causing duplicated __del__ calls
https://bugs.python.org/issue36556  opened by jdemeyer

#36557: Python (Launcher)3.7.3 CMDLine install/uninstall
https://bugs.python.org/issue36557  opened by mattcher_h

#36558: Change time.mktime() return type from float to int?
https://bugs.python.org/issue36558  opened by vstinner

#36560: test_functools leaks randomly 1 memory block
https://bugs.python.org/issue36560  opened by vstinner

#36563: pdbrc is read twice if current directory is the home directory
https://bugs.python.org/issue36563  opened by blueyed

#36564: Infinite loop with short maximum line lengths in EmailPolicy
https://bugs.python.org/issue36564  opened by p-ganssle

#36567: DOC: manpage directive doesn't create hyperlink
https://bugs.python.org/issue36567  opened by cheryl.sabella

#36568: Typo in socket.CAN_RAW_FD_FRAMES library documentation
https://bugs.python.org/issue36568  opened by Carl Cerecke

#36569: @staticmethod seems to work with setUpClass, but docs say it s
https://bugs.python.org/issue36569  opened by Peter de Blanc

#36572: python-snappy install issue during Crossbar install with Pytho
https://bugs.python.org/issue36572  opened by telatoa

#36573: zipfile zipfile.BadZipFile: Bad CRC-32 for file '11_02_2019.pd
https://bugs.python.org/issue36573  opened by Jozef Cernak

#36576: Some test_ssl and test_asyncio tests fail with OpenSSL 1.1.1 o
https://bugs.python.org/issue36576  opened by vstinner

#36580: unittest.mock does not understand dataclasses
https://bugs.python.org/issue36580  opened by John Parejko2

#36581: __dir__ on unittest.mock not safe for all spec types
https://bugs.python.org/issue36581  opened by Dylan Semler

#36582: collections.UserString encode method returns a string
https://bugs.python.org/issue36582  opened by trey

#36583: Do not swallow exceptions in the _ssl module
https://bugs.python.org/issue36583  opened by serhiy.storchaka

#36585: test_posix.py fails due to unsupported RWF_HIPRI
https://bugs.python.org/issue36585  opened by jdemeyer

#36586: multiprocessing.Queue.close doesn't behave as documented
https://bugs.python.org/issue36586  opened by graingert

#36587: race in logging code when fork()
https://bugs.python.org/issue36587  opened by cagney

#36589: Incorrect error handling in curses.update_lines_cols()
https://bugs.python.org/issue36589  opened by ZackerySpytz

#36590: Add Bluetooth RFCOMM Support for Windows
https://bugs.python.org/issue36590  opened by topnotcher

#36593: Trace function interferes with MagicMock isinstance?
https://bugs.python.org/issue36593  opened by nedbat

#36594: Undefined behavior due to incorrect usage of %p in format stri
https://bugs.python.org/issue36594  opened by ZackerySpytz

#36595: IDLE: Add search 

Re: [Python-Dev] Update PEP 394: Distributions can choose what does python command mean

[Barry Warsaw]

Thanks for the update.  I have made one small suggestion on the PR for 
clarification, but otherwise the changes LGTM.

-Barry

> On Apr 12, 2019, at 07:53, Miro Hrončok  wrote:
> 
> Hello.
> 
> Based on discussions in [1], Petr Viktorin and me have drafted a new update 
> [2] to the PEP 394 (The "python" Command on Unix-Like Systems).
> 
> The update gives distributors the opportunity to decide where does the 
> "python" command lead to, whether it is present etc.
> 
> Please, see the PR [2] for the suggested changes.
> 
> [1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html
> [2]: https://github.com/python/peps/pull/989
> 
> Thanks,
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/barry%40python.org



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing PID check from signal handler

[Ivan Pozdeev via Python-Dev]

On 12.04.2019 21:05, Steve Dower wrote:

On 12Apr.2019 0919, Jeroen Demeyer wrote:

The signal handler (that receives signals from the OS) in Python starts
with a check

     if (getpid() == main_pid)

Looking at the comments, the intent was to do a check for the main
*thread* but this is checking the *process* id. So this condition is
basically always true. Therefore, I suggest to remove it in
https://bugs.python.org/issue36601

If you have any objections or comments, I suggest to post them to that bpo.

To add a little more context, the check was added about 25 years ago as
a "hack" for some reason that we can't figure out anymore.

So if you are a historian of ancient operating systems and know of one
that might have raised signal handlers in a different process from the
one where it was registered, we'd love to hear from you.


According to 
https://www.linuxquestions.org/questions/programming-9/the-return-value-of-getpid-called-from-main-thread-and-new-thread-r-identical-624399/ ,

threads used to have different PIDs in the 2.4 Linux kernel.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru


--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Greg Ewing]

Victor Stinner wrote:

Python is used on devices with low memory (ex: 256 MiB for the whole
system). Allowing developers to use a debug build on such devices seem
to be a legit rationale for such change.


Rather than removing features altogether, maybe the debug
build could be split into a number of separate features
that can be enabled individually?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Greg Ewing]

Victor Stinner wrote:

I'm not sure of what you means by "objects placed at static memory":
the double linked list of all Python objects is created at runtime.
_ob_next and _ob_prev are initialized statically to NULL.


The trick of allocating extra memory in front of the object
would be harder to pull off for statically allocated objects,
although probably not impossible.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Victor Stinner]

Le sam. 13 avr. 2019 à 00:38, Greg Ewing  a écrit :
> Rather than removing features altogether, maybe the debug
> build could be split into a number of separate features
> that can be enabled individually?

I don't propose to *remove* a feature, but just to *disable* it *by
default* (when Python is compiled in debug mode):

"[WIP] bpo-36465: Py_DEBUG no longer implies Py_TRACE_REFS #12615"
https://github.com/python/cpython/pull/12615/files

In short, my change just removes:

/* Py_DEBUG implies Py_TRACE_REFS. */
#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS)
#define Py_TRACE_REFS
#endif

The feature will still be accessible if you compile Python with
Py_TRACE_REFS defined.

In practice, I understood that the debug build of Python is not known
by all core developers, and it seems like it's mostly used by core
developers. Maybe it's even only used by core developers? It's hard to
say. If it's only used by core developers, I hope that all core devs
know to compile Python :-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Nathaniel Smith]

On Thu, Apr 11, 2019 at 8:26 AM Steve Dower  wrote:
>
> On 10Apr2019 1917, Nathaniel Smith wrote:
> > It sounds like --with-pydebug has accumulated a big grab bag of
> > unrelated features, mostly stuff that was useful at some point for
> > some CPython dev trying to debug CPython itself? It's clearly not
> > designed with end users as the primary audience, given that no-one
> > knows what it actually does and that it makes third-party extensions
> > really awkward to run. If that's right then I think Victor's plan of
> > to sort through what it's actually doing makes a lot of sense,
> > especially if we can remove the ABI breaking stuff, since that causes
> > a disproportionate amount of trouble.
>
> Does it really cause a "disproportionate" amount of trouble? It's
> definitely not meant for anyone who isn't working on C code, whether in
> CPython, an extension or a host application. If you want to use
> third-party extensions and are not able to rebuild them, that's a very
> good sign that you probably shouldn't be on the debug build at all.

Well, here's what I mean by "disproportionate". Some of the costs of
the ABI divergence are:

- The first time I had to debug a C extension, I wasted a bunch of
time trying to figure out how I was supposed to use Debian's
'python-dbg' package (the --with-pydebug build), before eventually
figuring out that it was a red herring and what I actually wanted was
the -dbgsym package (their equivalent of MSVC's /Zi /DEBUG files).

- The extension loading machinery has extra code and complexity to
track the two different ABIs. The package ecosystem does too, e.g.
distutils needs to name extensions appropriately, and we need special
wheel tags, and pip needs code to handle these tags:

https://github.com/pypa/pip/blob/54b6a91405adc79cdb8a2954e9614d6860799ccb/src/pip/_internal/pep425tags.py#L106-L109

- If you want some of the features of --with-pydebug that don't change
the ABI, then you still have to rebuild third-party extensions to get
at them, and that's a significant hassle. (I could do it if I had to,
but my time has value.)

- Everyone who uses ctypes to access a PyObject* has to include some
extra hacks to handle the difference between the regular and debug
ABIs. There are a few different versions that get copy/pasted around
as folklore, and they're all pretty obscure. For example:

https://github.com/pallets/jinja/blob/fd89fed7456e755e33ba70674c41be5ab222e193/jinja2/debug.py#L317-L334

https://github.com/johndpope/sims4-ai-engine/blob/865212e841c716dc4364e0dba286f02af8d716e8/core/framewrapper.py#L12-L41

https://github.com/python-trio/trio/blob/862ced04e1f19287e098380ed8a0635004c36dd1/trio/_core/_multierror.py#L282
  And then if you want to test this code, it means you have to add a
--with-pydebug build to your CI infrastructure...

I don't know how many people use Py_TRACE_REFS, but if we can't find
anyone on python-dev who uses it then it must be pretty rare. If
dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
all the stuff above, then that seems like a pretty good trade! But
maybe the Windows C runtime issue will foil this...

> >> The reason we ship debug Python binaries is because debug builds use a
> >> different C Runtime, so if you do a debug build of an extension module
> >> you're working on it won't actually work with a non-debug build of CPython.
> >
> > ...But this is an important point. I'd forgotten that MSVC has a habit
> > of changing the entire C runtime when you turn on the compiler's
> > debugging mode.
>
> Technically they are separate options, but most project files are
> configured such that *their* Debug/Release switch affects both the
> compiler options (optimization) and the linker options (C runtime linkage).

So how do other projects handle this? I guess historically the main
target audience for Visual Studio was folks building monolithic apps,
where you can just rebuild everything with whatever options you want,
and compared to that Python extensions are messier. But Python isn't
the only project in this boat. Do ruby, nodejs, R, etc., all provide
separate debug builds with incompatible ABIs on Windows, and propagate
that information throughout their module/package ecosystem?

-n

--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Steve Dower]

On 12Apr.2019 1643, Nathaniel Smith wrote:
> On Thu, Apr 11, 2019 at 8:26 AM Steve Dower  wrote:
>>
>> On 10Apr2019 1917, Nathaniel Smith wrote:
> I don't know how many people use Py_TRACE_REFS, but if we can't find
> anyone on python-dev who uses it then it must be pretty rare. If
> dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> all the stuff above, then that seems like a pretty good trade! But
> maybe the Windows C runtime issue will foil this...

The very first question I asked was whether this would let us converge
the ABIs, and the answer was "no".

Otherwise I'd have said go for it, despite the C runtime issues.

 The reason we ship debug Python binaries is because debug builds use a
 different C Runtime, so if you do a debug build of an extension module
 you're working on it won't actually work with a non-debug build of CPython.
>>>
>>> ...But this is an important point. I'd forgotten that MSVC has a habit
>>> of changing the entire C runtime when you turn on the compiler's
>>> debugging mode.
>>
>> Technically they are separate options, but most project files are
>> configured such that *their* Debug/Release switch affects both the
>> compiler options (optimization) and the linker options (C runtime linkage).
> 
> So how do other projects handle this? I guess historically the main
> target audience for Visual Studio was folks building monolithic apps,
> where you can just rebuild everything with whatever options you want,
> and compared to that Python extensions are messier. But Python isn't
> the only project in this boat. Do ruby, nodejs, R, etc., all provide
> separate debug builds with incompatible ABIs on Windows, and propagate
> that information throughout their module/package ecosystem?

Mostly I hear complaints about those languages *not* providing any help
here. Python is renowned for having significantly better Windows support
than any of them, so they're the wrong comparison to make in my opinion.
Arguing that we should regress because other languages haven't caught up
to us yet makes no sense.

The tools that are better than Python typically don't ship debug builds
either, unless you specifically request them. But they also don't leak
their implementation details all over the place. If we had a better C
API, we wouldn't have users who needed to match ABIs.

For the most part, disabling optimizations in your own extension but
using the non-debug ABI is sufficient, and if you're having to deal with
other people's packages then maybe you don't have any choice (though I
do know of people who have built debug versions of numpy before - turns
out Windows developers are often just as capable as non-Windows
developers when it comes to building things ;) ). And yes, they could
also build CPython from source as well to get the debug ABI, or get the
debug symbols, but I saw enough need that I decided it was worth the
effort to just solve that problem. 250k downloads a month is enough to
justify it for me.

Not to bring the packaging discussions to another venue, but maybe this
is yet another area we need to stop pretending that we're able to solve
every single problem with just the tools we already have available?
People who want debug builds of packages can build them themselves, even
numpy and scipy, they don't need us to preemptively do all their work
for them. But we can (and should) help short-cut unnecessary effort or
research by providing helpful tools and instruction.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Nathaniel Smith]

On Fri, Apr 12, 2019 at 5:05 PM Steve Dower  wrote:
>
> On 12Apr.2019 1643, Nathaniel Smith wrote:
> > On Thu, Apr 11, 2019 at 8:26 AM Steve Dower  wrote:
> >>
> >> On 10Apr2019 1917, Nathaniel Smith wrote:
> > I don't know how many people use Py_TRACE_REFS, but if we can't find
> > anyone on python-dev who uses it then it must be pretty rare. If
> > dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> > all the stuff above, then that seems like a pretty good trade! But
> > maybe the Windows C runtime issue will foil this...
>
> The very first question I asked was whether this would let us converge
> the ABIs, and the answer was "no".
>
> Otherwise I'd have said go for it, despite the C runtime issues.

I don't see that in the thread... just Victor saying he isn't sure
whether there might be other ABI incompatibilities lurking that he
hasn't found yet. Did I miss something?

I'm mostly interested in this because of the possibility of converging
the ABIs. If you think that the C runtime thing isn't a blocker for
that, then that's useful information. Though obviously we still need
to figure out whether there are any other blockers :-).

>  The reason we ship debug Python binaries is because debug builds use a
>  different C Runtime, so if you do a debug build of an extension module
>  you're working on it won't actually work with a non-debug build of 
>  CPython.
> >>>
> >>> ...But this is an important point. I'd forgotten that MSVC has a habit
> >>> of changing the entire C runtime when you turn on the compiler's
> >>> debugging mode.
> >>
> >> Technically they are separate options, but most project files are
> >> configured such that *their* Debug/Release switch affects both the
> >> compiler options (optimization) and the linker options (C runtime linkage).
> >
> > So how do other projects handle this? I guess historically the main
> > target audience for Visual Studio was folks building monolithic apps,
> > where you can just rebuild everything with whatever options you want,
> > and compared to that Python extensions are messier. But Python isn't
> > the only project in this boat. Do ruby, nodejs, R, etc., all provide
> > separate debug builds with incompatible ABIs on Windows, and propagate
> > that information throughout their module/package ecosystem?
>
> Mostly I hear complaints about those languages *not* providing any help
> here. Python is renowned for having significantly better Windows support
> than any of them, so they're the wrong comparison to make in my opinion.
> Arguing that we should regress because other languages haven't caught up
> to us yet makes no sense.
>
> The tools that are better than Python typically don't ship debug builds
> either, unless you specifically request them. But they also don't leak
> their implementation details all over the place. If we had a better C
> API, we wouldn't have users who needed to match ABIs.

Do you happen to have a list of places where the C API leaks details
of the underlying CRT?

(I'm mostly curious because whenever I've looked my conclusion was
essentially: "Well... I don't see any places that are *definitely*
broken, so maybe mixing CRTs is fine? but I have zero confidence that
I caught everything, so probably better to play it safe?". At least on
py3 – I know the py2 C API was definitely broken if you mixed CRTs,
because of the exposed FILE*.)

> For the most part, disabling optimizations in your own extension but
> using the non-debug ABI is sufficient, and if you're having to deal with
> other people's packages then maybe you don't have any choice (though I
> do know of people who have built debug versions of numpy before - turns
> out Windows developers are often just as capable as non-Windows
> developers when it comes to building things ;)

I'm not sure why you think I was implying otherwise? I'm sorry if you
thought I was attacking your users or something. I did say that I
thought most users downloading the debug builds were probably confused
about what they were actually getting, but I didn't mean because they
were stupid Windows users, I meant because the debug builds are so
confusing that even folks on the Python core team are confused about
what they're actually getting.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Victor Stinner]

>> The very first question I asked was whether this would let us converge
>> the ABIs, and the answer was "no".

The answer is yes and it's my primary goal.

See my first email: "This change makes the debug build ABI closer to the
release build ABI".

To be honest, I am now lost in this long thread :-) I don't recall why I
started to argue so much about the memory footprint, it's not really the
main point here.

Victor


>> Otherwise I'd have said go for it, despite the C runtime issues.
>
> I don't see that in the thread... just Victor saying he isn't sure
> whether there might be other ABI incompatibilities lurking that he
> hasn't found yet. Did I miss something?
>
> I'm mostly interested in this because of the possibility of converging
> the ABIs. If you think that the C runtime thing isn't a blocker for
> that, then that's useful information. Though obviously we still need
> to figure out whether there are any other blockers :-).
>
>>  The reason we ship debug Python binaries is because debug builds
use a
>>  different C Runtime, so if you do a debug build of an extension
module
>>  you're working on it won't actually work with a non-debug build of
CPython.
>> >>>
>> >>> ...But this is an important point. I'd forgotten that MSVC has a
habit
>> >>> of changing the entire C runtime when you turn on the compiler's
>> >>> debugging mode.
>> >>
>> >> Technically they are separate options, but most project files are
>> >> configured such that *their* Debug/Release switch affects both the
>> >> compiler options (optimization) and the linker options (C runtime
linkage).
>> >
>> > So how do other projects handle this? I guess historically the main
>> > target audience for Visual Studio was folks building monolithic apps,
>> > where you can just rebuild everything with whatever options you want,
>> > and compared to that Python extensions are messier. But Python isn't
>> > the only project in this boat. Do ruby, nodejs, R, etc., all provide
>> > separate debug builds with incompatible ABIs on Windows, and propagate
>> > that information throughout their module/package ecosystem?
>>
>> Mostly I hear complaints about those languages *not* providing any help
>> here. Python is renowned for having significantly better Windows support
>> than any of them, so they're the wrong comparison to make in my opinion.
>> Arguing that we should regress because other languages haven't caught up
>> to us yet makes no sense.
>>
>> The tools that are better than Python typically don't ship debug builds
>> either, unless you specifically request them. But they also don't leak
>> their implementation details all over the place. If we had a better C
>> API, we wouldn't have users who needed to match ABIs.
>
> Do you happen to have a list of places where the C API leaks details
> of the underlying CRT?
>
> (I'm mostly curious because whenever I've looked my conclusion was
> essentially: "Well... I don't see any places that are *definitely*
> broken, so maybe mixing CRTs is fine? but I have zero confidence that
> I caught everything, so probably better to play it safe?". At least on
> py3 – I know the py2 C API was definitely broken if you mixed CRTs,
> because of the exposed FILE*.)
>
>> For the most part, disabling optimizations in your own extension but
>> using the non-debug ABI is sufficient, and if you're having to deal with
>> other people's packages then maybe you don't have any choice (though I
>> do know of people who have built debug versions of numpy before - turns
>> out Windows developers are often just as capable as non-Windows
>> developers when it comes to building things ;)
>
> I'm not sure why you think I was implying otherwise? I'm sorry if you
> thought I was attacking your users or something. I did say that I
> thought most users downloading the debug builds were probably confused
> about what they were actually getting, but I didn't mean because they
> were stupid Windows users, I meant because the debug builds are so
> confusing that even folks on the Python core team are confused about
> what they're actually getting.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>

-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com