[Python-Dev] Increasing the C-optimized pickle extensibility

2019-04-26 Thread Pierre Glaser
Hi All, We (Antoine Pitrou, Olivier Grisel and myself) spent some efforts recently on enabling pickle extensions to extend the C-optimized Pickler instead of the pure Python one. Pickle extensions have a crucial role in many distributed computing libraries: cloudpickle (https://github.com/cl

Re: [Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data

2018-11-05 Thread Pierre Glaser
Hi All, As part of our scikit-learn development and our effort to provide better parallelism for python, we rely heavily on dynamic classes and functions pickling. For this usage we use cloudpickle, but it suffers from performance issues due to its pure python implementation. After long discuss

Re: [Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data

2018-11-05 Thread Pierre Glaser
Hi All, As part of our scikit-learn development and our effort to provide better parallelism for python, we rely heavily on dynamic classes and functions pickling. For this usage we use cloudpickle, but it suffers from performance issues due to its pure python implementation. After long discuss