Call julia from Python: which package?
Hi, I have a Python program that uses Tkinter for its GUI. It's rather slow so I hope to replace many or all of the non-GUI parts by Julia code. Has anybody experience with this? Any packages you can recommend? I found three alternatives: * https://pyjulia.readthedocs.io/en/latest/usage.html# * https://pypi.org/project/juliacall/ * https://github.com/JuliaPy/PyCall.jl Thanks in advance! Albert-Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam
wrote:
> Hi,
>
> I have a Python program that uses Tkinter for its GUI. It's rather slow so
> I hope to replace many or all of the non-GUI parts by Julia code. Has
> anybody experience with this? Any packages you can recommend? I found three
> alternatives:
>
> * https://pyjulia.readthedocs.io/en/latest/usage.html#
> * https://pypi.org/project/juliacall/
> * https://github.com/JuliaPy/PyCall.jl
>
> Thanks in advance!
>
I have zero Julia experience.
I thought I would share this though:
https://stromberg.dnsalias.org/~strombrg/speeding-python/
Even if you go the Julia route, it's probably still best to profile your
Python code to identify the slow ("hot") spots, and rewrite only them.
--
https://mail.python.org/mailman/listinfo/python-list
Problem Installing Pipenv
I have Python 3.9 installed in Windows 10 on my Laptop and have installed pipenv v-2021.11.5. When I invoke the python shell command, it fails with thee following errors: C:\Users\SPM\Apps\Spyder-5.2.1>pipenv shell Traceback (most recent call last): File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python39\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Python39\Scripts\pipenv.exe\__main__.py", line 7, in File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 1062, in main rv = self.invoke(ctx) File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke return __callback(*args, **kwargs) File "C:\Python39\lib\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func return ctx.invoke(f, obj, *args, **kwargs) File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke return __callback(*args, **kwargs) File "C:\Python39\lib\site-packages\pipenv\cli\command.py", line 419, in shell do_shell( File "C:\Python39\lib\site-packages\pipenv\core.py", line 2309, in do_shell shell = choose_shell() TypeError: choose_shell() missing 1 required positional argument: 'project' Stephen P. Molnar, Ph.D. (614)312-7528 Skype: smolnar1 -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
Additionally I'd like to ask, if you e.g. used pandas and numpy before
you make the effort of rewriting everything.
Yes python is slower than compiled languages, but especially with numpy
you can make use of compiled code underneath.
Of course this could mean rewriting some existing code but staying in
python.
If you have done all of that or there are other impediments, then of
course no one will hold you back.
Am 17.12.21 um 16:12 schrieb Dan Stromberg:
> On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam
> wrote:
>
>> Hi,
>>
>> I have a Python program that uses Tkinter for its GUI. It's rather slow so
>> I hope to replace many or all of the non-GUI parts by Julia code. Has
>> anybody experience with this? Any packages you can recommend? I found three
>> alternatives:
>>
>> * https://pyjulia.readthedocs.io/en/latest/usage.html#
>> * https://pypi.org/project/juliacall/
>> * https://github.com/JuliaPy/PyCall.jl
>>
>> Thanks in advance!
>>
> I have zero Julia experience.
>
> I thought I would share this though:
> https://stromberg.dnsalias.org/~strombrg/speeding-python/
>
> Even if you go the Julia route, it's probably still best to profile your
> Python code to identify the slow ("hot") spots, and rewrite only them.
--
punkt.de GmbH
Lars Liedtke
.infrastructure
Kaiserallee 13a
76133 Karlsruhe
Tel. +49 721 9109 500
https://infrastructure.punkt.de
[email protected]
AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein
--
https://mail.python.org/mailman/listinfo/python-list
A problem with itertools.groupby
Python 3.9.9
Hello
I have some troubles with groupby from itertools
from itertools import groupby
for k, grp in groupby("aahfffddnnb"):
print(k, list(grp))
print(k, list(grp))
a ['a', 'a']
a []
h ['h']
h []
f ['f', 'f', 'f']
f []
d ['d', 'd']
d []
s ['s', 's', 's', 's']
s []
n ['n', 'n']
n []
b ['b']
b []
It works as expected.
itertools._grouper objects are probably iterators
so they provide their datas only once. OK
but:
li = [grp for k, grp in groupby("aahfffddnnb")]
list(li[0])
[]
list(li[1])
[]
It seems empty ... I don't understand why, this is
the first read of an iterator, it should provide its
data.
This one works:
["".join(grp) for k, grp in groupby("aahfffddnnb")]
['aa', 'h', 'fff', 'dd', '', 'nn', 'b']
regards
--
https://mail.python.org/mailman/listinfo/python-list
Re: A problem with itertools.groupby
but:
li = [grp for k, grp in groupby("aahfffddnnb")]
list(li[0])
[]
list(li[1])
[]
It seems empty ... I don't understand why, this is
the first read of an iterator, it should provide its
data.
The group-iterators are connected. Each group-iterator is a wrapper
around the original iterator
with an extra termination condition. So in order to start the next
group-iterator the previous
group-iterator is exhausted, because the original iterator has to be
ready to produce values
for the next group-iterator.
--
Antoon Pardon.
--
https://mail.python.org/mailman/listinfo/python-list
Re: A problem with itertools.groupby
On Sat, Dec 18, 2021 at 2:32 AM ast wrote:
>
> Python 3.9.9
>
> Hello
>
> I have some troubles with groupby from itertools
>
> from itertools import groupby
>
> li = [grp for k, grp in groupby("aahfffddnnb")]
> list(li[0])
>
> []
>
> list(li[1])
>
> []
>
> It seems empty ... I don't understand why, this is
> the first read of an iterator, it should provide its
> data.
>
https://docs.python.org/3/library/itertools.html#itertools.groupby
Check the explanatory third paragraph :)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: A problem with itertools.groupby
On Fri, 17 Dec 2021 09:25:03 +0100, ast wrote:
[snip]
>
> but:
>
> li = [grp for k, grp in groupby("aahfffddnnb")]
> list(li[0])
>
> []
>
> list(li[1])
>
> []
>
> It seems empty ... I don't understand why, this is
> the first read of an iterator, it should provide its
> data.
Baffling. Here's a shorter and less readable illustration:
>>> list(groupby("aabbb"))
[('a', ),
('b', )]
>>> list(groupby("aabbb"))[0]
('a', )
>>> list(list(groupby("aabbb"))[0][1])
[]
--
To email me, substitute nowhere->runbox, invalid->com.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
On Fri, 17 Dec 2021 at 15:04, Albert-Jan Roskam wrote: > > Hi, > > I have a Python program that uses Tkinter for its GUI. It's rather slow so I > hope to replace many or all of the non-GUI parts by Julia code. Has anybody > experience with this? Any packages you can recommend? I found three > alternatives: > > * https://pyjulia.readthedocs.io/en/latest/usage.html# > * https://pypi.org/project/juliacall/ > * https://github.com/JuliaPy/PyCall.jl Like others here I don't have any particular advice to offer for the actual question you have asked. Probably you would get better answers on a Julia mailing list. Before you consider rewriting any of your Python code in Julia I suggest trying it out to see if you can actually demonstrate a proof of concept that it can even be faster for your particular problem. Also like others here I suggest that you explore the options for speeding things up within Python. Putting together a Frankenstein application that's half in Julia and half in Python might be worth it but that approach has significant costs so make sure you are clear that there are going to be benefits before you go down that road. I do have a very small amount of experience using Julia. As a maintainer of SymPy I was very interested by Symbolics.jl which was created in an attempt to be faster than SymPy. I spent some time going through tutorials for Julia and making toy programs and then I tried out Symbolics.jl. I also had various discussions with those involved in Symbolics.jl on Julia mailing lists and GitHub. Based on my experience with Julia the immediate conclusion I have that is relevant for you is this: Do *not* presume that writing your code in Julia rather than Python will make it faster. There is a lot of hype about Julia and its proselytisers will claim that it "runs at the speed of C". That might be sort of true for certain specific kinds of code or particular kinds of problems but it is certainly not true in general that Julia runs at the speed of C. In fact I have seen plenty of basic cases where Julia is slower than Python. The other thing that I have seen plenty of in discussions about performance in both Python and Julia is poor use of benchmark results. The big culprit is timeit and its equivalent btime in Julia. When I tried to discuss the speed of operations in Julia it was suggested that I should use btime but there are so many different levels of caching in Julia (at least in Symbolics.jl) that the btime results are meaningless. It seems to be common practice in the Julia community to refer to btime results just as it is in Python to refer to timeit results. I think that there is a specific class of problems where Julia is significantly faster than (CPython) which in practice is mostly around intensive non-vectorisable floating point calculations. Note that these are the same kind of things that can be made a lot faster if you use PyPy or Numba or any of the related options that can be done without rewriting your existing Python code in another language. In Julia just as in PyPy/Numba etc it's important for the speed gain that you can actually reduce your operations down to low-level machine types like float64, int32 etc. I tried writing a matrix multiplication routine in Julia to see how it would compare with Python. I used a straight-forward dumb implementation in both languages. I wanted to compare with something like this which is pure Python: https://github.com/sympy/sympy/blob/88ed7abb488da615b007dd2ed5404312caef473c/sympy/polys/matrices/dense.py#L85-L91 Note that in that particular application it isn't possible to use something like int32 because it's part of a symbolic library that performs exact calculations that can easily overflow any standard machine types e.g. the determinant of a 20x20 matrix of integers between 0 and 100 easily overflows the range of a 64 bit integer: >>> from sympy import * >>> det(randMatrix(20, 20)) 1389363438512397826059139369892638115688 When I timed the result in Julia and in Python I found that the Julia code was slower than the Python code. Of course I don't know how to optimise Julia code so I asked one of my colleagues who does (and who likes to proselytise about Julia). He pointed me to here where the creator of Julia says "BigInts are currently pretty slow in Julia": https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast. I should make clear here that I used the gmpy2 library in Python for the basic integer operations which is a wrapper around the same gmp library that is used by Julia. That means that the basic integer operations were being done by the same gmp library (a C library) in each case. The timing differences between Python and Julia are purely about overhead around usage of the same underlying C library. The Julia code was 2x slower than Python in this task. By comparison flint is a library that act
Re: Call julia from Python: which package?
On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin wrote: > When I timed the result in Julia and in Python I found that the Julia > code was slower than the Python code. Of course I don't know how to > optimise Julia code so I asked one of my colleagues who does (and who > likes to proselytise about Julia). He pointed me to here where the > creator of Julia says "BigInts are currently pretty slow in Julia": > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast. > I should make clear here that I used the gmpy2 library in Python for > the basic integer operations which is a wrapper around the same gmp > library that is used by Julia. That means that the basic integer > operations were being done by the same gmp library (a C library) in > each case. The timing differences between Python and Julia are purely > about overhead around usage of the same underlying C library. Point of note: "Python actually uses a hybrid approach where small integer values are represented inline and only when values get too large are they represented as BigInts" might be sorta-kinda true for Python 2, but it's not true for Python 3. In all versions of CPython to date, all integers are objects, they're not "represented inline" (there *are* some languages that do this intrinsically, and I think that PyPy can sometimes unbox integers, but CPython never will); and the performance advantage of Py2's machine-sized integers clearly wasn't worth recreating in Py3. (It would be possible to implement it in Py3 as an optimization, while still having the same 'int' type for both, but nobody's done it.) So if Python's integers are faster than Julia's, it's not because there's any sort of hybrid approach, it's simply because CPython is more heavily optimized for that sort of work. (That said: I have no idea whether a StackOverflow answer from 2016 is still applicable.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
On Fri, 17 Dec 2021 at 22:40, Chris Angelico wrote: > > On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin > wrote: > > When I timed the result in Julia and in Python I found that the Julia > > code was slower than the Python code. Of course I don't know how to > > optimise Julia code so I asked one of my colleagues who does (and who > > likes to proselytise about Julia). He pointed me to here where the > > creator of Julia says "BigInts are currently pretty slow in Julia": > > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast. > > I should make clear here that I used the gmpy2 library in Python for > > the basic integer operations which is a wrapper around the same gmp > > library that is used by Julia. That means that the basic integer > > operations were being done by the same gmp library (a C library) in > > each case. The timing differences between Python and Julia are purely > > about overhead around usage of the same underlying C library. > > Point of note: "Python actually uses a hybrid approach where small > integer values are represented inline and only when values get too > large are they represented as BigInts" might be sorta-kinda true for > Python 2, but it's not true for Python 3. In all versions of CPython > to date, all integers are objects, they're not "represented inline" > (there *are* some languages that do this intrinsically, and I think > that PyPy can sometimes unbox integers, but CPython never will); and > the performance advantage of Py2's machine-sized integers clearly > wasn't worth recreating in Py3. (It would be possible to implement it > in Py3 as an optimization, while still having the same 'int' type for > both, but nobody's done it.) > > So if Python's integers are faster than Julia's, it's not because > there's any sort of hybrid approach, it's simply because CPython is > more heavily optimized for that sort of work. > > (That said: I have no idea whether a StackOverflow answer from 2016 is > still applicable.) To be clear: I wasn't using Python's int type. I used the gmpy2.mpz type which is precisely the same mpz from the same gmp library that Julia uses. I'm told by my colleague that Julia has a lot of overhead when using "heap types" which is possibly the cause of the problem. The other library I referenced (flint) does have a super-optimised approach for handling small integers natively as machine types at the same time as supporting large integers and even includes hand-written assembly for the most basic operations. That library is significantly faster than Python's int or gmpy2/gmp. In any case what it shows is that Julia does not have reduced overhead *in general* (i.e. for all situations) as compared to Python. I expected that but I also thought that this particular example would be within the range of things where Julia out-performed Python and that turned out not to be the case. -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
On Sat, Dec 18, 2021 at 10:01 AM Oscar Benjamin wrote: > > On Fri, 17 Dec 2021 at 22:40, Chris Angelico wrote: > > > > On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin > > wrote: > > > When I timed the result in Julia and in Python I found that the Julia > > > code was slower than the Python code. Of course I don't know how to > > > optimise Julia code so I asked one of my colleagues who does (and who > > > likes to proselytise about Julia). He pointed me to here where the > > > creator of Julia says "BigInts are currently pretty slow in Julia": > > > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast. > > > I should make clear here that I used the gmpy2 library in Python for > > > the basic integer operations which is a wrapper around the same gmp > > > library that is used by Julia. That means that the basic integer > > > operations were being done by the same gmp library (a C library) in > > > each case. The timing differences between Python and Julia are purely > > > about overhead around usage of the same underlying C library. > > > > Point of note: "Python actually uses a hybrid approach where small > > integer values are represented inline and only when values get too > > large are they represented as BigInts" might be sorta-kinda true for > > Python 2, but it's not true for Python 3. In all versions of CPython > > to date, all integers are objects, they're not "represented inline" > > (there *are* some languages that do this intrinsically, and I think > > that PyPy can sometimes unbox integers, but CPython never will); and > > the performance advantage of Py2's machine-sized integers clearly > > wasn't worth recreating in Py3. (It would be possible to implement it > > in Py3 as an optimization, while still having the same 'int' type for > > both, but nobody's done it.) > > > > So if Python's integers are faster than Julia's, it's not because > > there's any sort of hybrid approach, it's simply because CPython is > > more heavily optimized for that sort of work. > > > > (That said: I have no idea whether a StackOverflow answer from 2016 is > > still applicable.) > > To be clear: I wasn't using Python's int type. I used the gmpy2.mpz > type which is precisely the same mpz from the same gmp library that > Julia uses. I'm told by my colleague that Julia has a lot of overhead > when using "heap types" which is possibly the cause of the problem. Ah, interesting. What's the advantage of using mpz instead of Python's builtin int? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
On Fri, 17 Dec 2021 at 23:11, Chris Angelico wrote: > > On Sat, Dec 18, 2021 at 10:01 AM Oscar Benjamin > wrote: > > > > On Fri, 17 Dec 2021 at 22:40, Chris Angelico wrote: > > > > > > On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin > > > wrote: > > > > When I timed the result in Julia and in Python I found that the Julia > > > > code was slower than the Python code. Of course I don't know how to > > > > optimise Julia code so I asked one of my colleagues who does (and who > > > > likes to proselytise about Julia). He pointed me to here where the > > > > creator of Julia says "BigInts are currently pretty slow in Julia": > > > > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast. > > > > I should make clear here that I used the gmpy2 library in Python for > > > > the basic integer operations which is a wrapper around the same gmp > > > > library that is used by Julia. That means that the basic integer > > > > operations were being done by the same gmp library (a C library) in > > > > each case. The timing differences between Python and Julia are purely > > > > about overhead around usage of the same underlying C library. > > > > > > Point of note: "Python actually uses a hybrid approach where small > > > integer values are represented inline and only when values get too > > > large are they represented as BigInts" might be sorta-kinda true for > > > Python 2, but it's not true for Python 3. In all versions of CPython > > > to date, all integers are objects, they're not "represented inline" > > > (there *are* some languages that do this intrinsically, and I think > > > that PyPy can sometimes unbox integers, but CPython never will); and > > > the performance advantage of Py2's machine-sized integers clearly > > > wasn't worth recreating in Py3. (It would be possible to implement it > > > in Py3 as an optimization, while still having the same 'int' type for > > > both, but nobody's done it.) > > > > > > So if Python's integers are faster than Julia's, it's not because > > > there's any sort of hybrid approach, it's simply because CPython is > > > more heavily optimized for that sort of work. > > > > > > (That said: I have no idea whether a StackOverflow answer from 2016 is > > > still applicable.) > > > > To be clear: I wasn't using Python's int type. I used the gmpy2.mpz > > type which is precisely the same mpz from the same gmp library that > > Julia uses. I'm told by my colleague that Julia has a lot of overhead > > when using "heap types" which is possibly the cause of the problem. > > Ah, interesting. What's the advantage of using mpz instead of Python's > builtin int? In this particular context the advantage was to give parity to the two languages I was profiling but in general gmpy2/flint have faster large integer arithmetic than Python's int type: In [13]: from gmpy2 import mpz In [14]: nums_int = [3**i for i in range(1000)] In [15]: nums_mpz = [mpz(3)**i for i in range(1000)] In [16]: def prod(nums): ...: result = nums[0] ...: for num in nums[1:]: ...: result *= num ...: return result ...: In [17]: %time ok = prod(nums_int) CPU times: user 384 ms, sys: 12 ms, total: 396 ms Wall time: 398 ms In [18]: %time ok = prod(nums_mpz) CPU times: user 124 ms, sys: 0 ns, total: 124 ms Wall time: 125 ms That's somewhat significant but the big difference for SymPy in using gmpy2 (as an optional dependency) is the mpq rational type which is significantly faster for small rational numbers: In [19]: from gmpy2 import mpq In [20]: from fractions import Fraction In [21]: nums_mpq = [mpq(i, 3) for i in range(1000)] In [22]: nums_frac = [Fraction(i, 3) for i in range(1000)] In [23]: %time ok = sum(nums_mpq) CPU times: user 0 ns, sys: 0 ns, total: 0 ns Wall time: 633 µs In [24]: %time ok = sum(nums_frac) CPU times: user 8 ms, sys: 0 ns, total: 8 ms Wall time: 10.2 ms For some slow operations SymPy is about 30x faster when gmpy2 is installed. -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
