Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
Marco Sulla wrote at 2021-12-29 08:08 +0100: >On Wed, 29 Dec 2021 at 00:03, Dieter Maurer wrote: >> Why do you not derive from `dict` and override its mutating methods >> (to raise a type error after initialization is complete)? > >I've done this for the pure py version, for speed. But in this way, >frozendict results to be a subclass of MutableMapping. `MutableMapping` is a so called abstract base class (--> `abc`). It uses the `__subclass_check__` (and `__instance_check__`) of `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. Those can be customized by overriding `MutableMapping.__subclasshook__` to ensure that your `frozendict` class (and their subclasses) are not considered subclasses of `MutableMapping`. There is a PEP (I forgot its number, but searching for any of the special method names above should locate it) describing the `issubclass/isinstance` extension logik. -- https://mail.python.org/mailman/listinfo/python-list
Re: Option for venv to upgrade pip automatically?
Cool, thanks! On Wed, 29 Dec 2021 at 07:10, Inada Naoki wrote: > > You can use --upgrade-deps option. My alias is: > > alias mkvenv='python3 -m venv --upgrade-deps --prompt . venv' > > On Wed, Dec 29, 2021 at 4:55 AM Marco Sulla > wrote: > > > > I think it's very boring that, after creating a venv, you have > > immediately to do every time: > > > > pip install -U pip > > > > Can't venv have an option for doing this automatically or, better, a > > config file where you can put commands that will be launched every > > time after you create a venv? > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > > -- > Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On Wed, 29 Dec 2021 at 09:12, Dieter Maurer wrote: > > Marco Sulla wrote at 2021-12-29 08:08 +0100: > >On Wed, 29 Dec 2021 at 00:03, Dieter Maurer wrote: > >> Why do you not derive from `dict` and override its mutating methods > >> (to raise a type error after initialization is complete)? > > > >I've done this for the pure py version, for speed. But in this way, > >frozendict results to be a subclass of MutableMapping. > > `MutableMapping` is a so called abstract base class (--> `abc`). > > It uses the `__subclass_check__` (and `__instance_check__`) of > `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. > Those can be customized by overriding `MutableMapping.__subclasshook__` > to ensure that your `frozendict` class (and their subclasses) > are not considered subclasses of `MutableMapping`. Emh. Too hacky for me too, sorry :D -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On second thought, I think I'll do this for the pure py version. But I will definitely not do this for the C extension, since it's anyway strange that an immutable mapping inherits from a mutable one! I've done it in the pure py version only for a matter of speed. On Wed, 29 Dec 2021 at 09:24, Marco Sulla wrote: > > On Wed, 29 Dec 2021 at 09:12, Dieter Maurer wrote: > > > > Marco Sulla wrote at 2021-12-29 08:08 +0100: > > >On Wed, 29 Dec 2021 at 00:03, Dieter Maurer wrote: > > >> Why do you not derive from `dict` and override its mutating methods > > >> (to raise a type error after initialization is complete)? > > > > > >I've done this for the pure py version, for speed. But in this way, > > >frozendict results to be a subclass of MutableMapping. > > > > `MutableMapping` is a so called abstract base class (--> `abc`). > > > > It uses the `__subclass_check__` (and `__instance_check__`) of > > `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. > > Those can be customized by overriding `MutableMapping.__subclasshook__` > > to ensure that your `frozendict` class (and their subclasses) > > are not considered subclasses of `MutableMapping`. > > Emh. Too hacky for me too, sorry :D -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
Marco Sulla wrote at 2021-12-29 09:29 +0100:
>On second thought, I think I'll do this for the pure py version. But I
>will definitely not do this for the C extension
Are you sure you need to implement your type in C at all?
I made a small `timeit` test:
```
>>> class cd(dict): pass
...
>>> timeit("d[1]", "d={1:1}", globals=globals())
0.0247416019765
>>> timeit("d[1]", "d=cd({1:1})", globals=globals())
0.08281239100051607
```
This means that for the above trivial case, access is 3.5 times slower
(the difference is smaller for more complex cases when hashing
becomes more expensive) but it is still only 83 ns per access.
Thus, if your application is not highly dominated by dict accesses,
the overall difference will not be large.
--
https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On Wed, Dec 29, 2021 at 8:07 PM Dieter Maurer wrote:
>
> Marco Sulla wrote at 2021-12-29 09:29 +0100:
> >On second thought, I think I'll do this for the pure py version. But I
> >will definitely not do this for the C extension
>
> Are you sure you need to implement your type in C at all?
>
> I made a small `timeit` test:
> ```
> >>> class cd(dict): pass
> ...
> >>> timeit("d[1]", "d={1:1}", globals=globals())
> 0.0247416019765
> >>> timeit("d[1]", "d=cd({1:1})", globals=globals())
> 0.08281239100051607
> ```
> This means that for the above trivial case, access is 3.5 times slower
> (the difference is smaller for more complex cases when hashing
> becomes more expensive) but it is still only 83 ns per access.
> Thus, if your application is not highly dominated by dict accesses,
> the overall difference will not be large.
You forgot slots. The difference can be even smaller.
>>> class cd(dict): __slots__ = ()
...
>>> timeit("d[1]", "d={1:1}", globals=globals())
0.02511977031826973
>>> timeit("d[1]", "d=cd({1:1})", globals=globals())
0.0333079993724823
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On Wed, 29 Dec 2021 at 10:06, Dieter Maurer wrote: > > Are you sure you need to implement your type in C at all? It's already implemented, and, in some cases, is faster than dict: https://github.com/Marco-Sulla/python-frozendict#benchmarks PS: I'm doing a refactoring that speeds up creation even further, making it almost as fast as dict. -- https://mail.python.org/mailman/listinfo/python-list
How to implement freelists in dict 3.10 for previous versions?
I noticed that now freelists in dict use _Py_dict_state. I suppose this is done for thread safety. I would implement it also for a C extension that uses CPython < 3.10. How can I achieve this? -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On Wed, 29 Dec 2021 at 09:12, Dieter Maurer wrote: > `MutableMapping` is a so called abstract base class (--> `abc`). > > It uses the `__subclass_check__` (and `__instance_check__`) of > `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. > Those can be customized by overriding `MutableMapping.__subclasshook__` > to ensure that your `frozendict` class (and their subclasses) > are not considered subclasses of `MutableMapping`. It does not work: $ python Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18) [GCC 10.1.1 20200718] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import frozendict >>> frozendict.c_ext False >>> from frozendict import frozendict as fd >>> from collections.abc import MutableMapping as Mm >>> issubclass(fd, Mm) True >>> @classmethod ... def _my_subclasshook(klass, subclass): ... if subclass == fd: ... return False ... return NotImplemented ... >>> @classmethod ... def _my_subclasshook(klass, subclass): ... print(subclass) ... if subclass == fd: ... return False ... return NotImplemented ... >>> Mm.__subclasshook__ = _my_subclasshook >>> issubclass(fd, Mm) True >>> issubclass(tuple, Mm) False >>> -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
Marco Sulla wrote at 2021-12-29 11:59 +0100: >On Wed, 29 Dec 2021 at 09:12, Dieter Maurer wrote: >> `MutableMapping` is a so called abstract base class (--> `abc`). >> >> It uses the `__subclass_check__` (and `__instance_check__`) of >> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. >> Those can be customized by overriding `MutableMapping.__subclasshook__` >> to ensure that your `frozendict` class (and their subclasses) >> are not considered subclasses of `MutableMapping`. > >It does not work: > ... issubclass(fd, Mm) >True There is a cache involved. The `issubclass` above, brings your `fd` in the `Mn`'s subclass cache. > ... Mm.__subclasshook__ = _my_subclasshook issubclass(fd, Mm) >True See above. -- https://mail.python.org/mailman/listinfo/python-list
Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
On Wed, 29 Dec 2021 at 12:11, Dieter Maurer wrote: > > Marco Sulla wrote at 2021-12-29 11:59 +0100: > >On Wed, 29 Dec 2021 at 09:12, Dieter Maurer wrote: > >> `MutableMapping` is a so called abstract base class (--> `abc`). > >> > >> It uses the `__subclass_check__` (and `__instance_check__`) of > >> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`. > >> Those can be customized by overriding `MutableMapping.__subclasshook__` > >> to ensure that your `frozendict` class (and their subclasses) > >> are not considered subclasses of `MutableMapping`. > > > >It does not work: > > ... > issubclass(fd, Mm) > >True > > There is a cache involved. The `issubclass` above, > brings your `fd` in the `Mn`'s subclass cache. It works, thank you! I had to put it before Mapping.register(frozendict) -- https://mail.python.org/mailman/listinfo/python-list
Re: recover pickled data: pickle data was truncated
> You have lost the data in that case. But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data. -- https://mail.python.org/mailman/listinfo/python-list
Re: recover pickled data: pickle data was truncated
On Thu, Dec 30, 2021 at 4:32 AM iMath wrote: > > > You have lost the data in that case. > > But I found the size of the file of the shelve data didn't change much, so I > guess the data are still in it , I just wonder any way to recover my data. Unless two conflicting versions got interleaved, in which case I strongly advise you NOT to try unpickling it. If you really feel like delving into it, try manually decoding the pickle stream, but be very careful. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
RE: recover pickled data: pickle data was truncated
I am not an expert on the topic but my first reaction is it depends on how the data is corrupted and we do not know that. So I am addressing a more general concept here. Some algorithms break if a single byte or even bit changes and nothing beyond that point makes sense. Many encryption techniques are like that and adding or deleting a byte might throw things off completely. But if your problem is that two processes or threads wrote interleaved and yet resulted in an output of a similar size, then, yes, in some cases some of the data could be retrieved, albeit be fragmentary and unreliable. If they both included say a data structure with names and phone numbers, it is possible you get two partial or complete copies and maybe retrieve a phone number you can try and see if it works. But the tax authorities might not react favorably to your recovery of a business expense if it is possible the currency amount was corrupted and perhaps a few zeroes were appended at the end. For some mission-critical purposes, I am sure people have come up with many ideas including perhaps making multiple copies before an exit spread across multiple disks and sites or reading the file back in and checking it. But corruption can happen for many reasons including at the level of the disk it is written to. -Original Message- From: Python-list On Behalf Of iMath Sent: Wednesday, December 29, 2021 10:51 AM To: [email protected] Subject: Re: recover pickled data: pickle data was truncated > You have lost the data in that case. But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: recover pickled data: pickle data was truncated
On Wed, 29 Dec 2021 at 18:33, iMath wrote: > But I found the size of the file of the shelve data didn't change much, so I > guess the data are still in it , I just wonder any way to recover my data. I agree with Barry, Chris and Avi. IMHO your data is lost. Unpickling it by hand is a harsh work and maybe unreliable. Is there any reason you can't simply add a semaphore to avoid writing at the same time and re-run the code and regenerate the data? -- https://mail.python.org/mailman/listinfo/python-list
Re: How to implement freelists in dict 3.10 for previous versions?
On Wed, Dec 29, 2021 at 7:25 PM Marco Sulla wrote: > > I noticed that now freelists in dict use _Py_dict_state. I suppose > this is done for thread safety. > Some core-dev are working on per-interpreter GIL. But it is not done yet. So you don't need to follow it soon. Your extension module will work well in Python 3.11. > I would implement it also for a C extension that uses CPython < 3.10. > How can I achieve this? See PyModule_GetState() to have per-interpreter module state instead of static variables. https://docs.python.org/3/c-api/module.html#c.PyModule_GetState -- Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
