Re: pytest segfault, not with -v
Ok, I created the script: https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py The problem is it does _not_ crash, while a get a segfault using pytest with python 3.9 on MacOS 10.15 Maybe it's because I'm using eval / exec in the script? On Sat, 20 Nov 2021 at 18:40, Marco Sulla wrote: > > Indeed I have introduced a command line parameter in my bench.py > script that simply specifies the number of times the benchmarks are > performed. This way I have a sort of segfault checker. > > But I don't bench any part of the library. I suppose I have to create > a separate script that does a simple loop for all the cases, and > remove the optional parameter from bench. How boring. > PS: is there a way to monitor the Python consumed memory inside Python > itself? In this way I could also trap memory leaks. > > On Sat, 20 Nov 2021 at 01:46, MRAB wrote: > > > > On 2021-11-19 23:44, Marco Sulla wrote: > > > On Fri, 19 Nov 2021 at 20:38, MRAB wrote: > > >> > > >> On 2021-11-19 17:48, Marco Sulla wrote: > > >> > I have a battery of tests done with pytest. My tests break with a > > >> > segfault if I run them normally. If I run them using pytest -v, the > > >> > segfault does not happen. > > >> > > > >> > What could cause this quantical phenomenon? > > >> > > > >> Are you testing an extension that you're compiling? That kind of problem > > >> can occur if there's an uninitialised variable or incorrect reference > > >> counting (Py_INCREF/Py_DECREF). > > > > > > Ok, I know. But why can't it be reproduced if I do pytest -v? This way > > > I don't know which test fails. > > > Furthermore I noticed that if I remove the __pycache__ dir of tests, > > > pytest does not crash, until I re-ran it with the __pycache__ dir > > > present. > > > This way is very hard for me to understand what caused the segfault. > > > I'm starting to think pytest is not good for testing C extensions. > > > > > If there are too few Py_INCREF or too many Py_DECREF, it'll free the > > object too soon, and whether or when that will cause a segfault will > > depend on whatever other code is running. That's the nature of the > > beast: it's unpredictable! > > > > You could try running each of the tests in a loop to see which one > > causes a segfault. (Trying several in a loop will let you narrow it down > > more quickly.) > > > > pytest et al. are good for testing behaviour, but not for narrowing down > > segfaults. > > -- > > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: pytest segfault, not with -v
Marco Sulla wrote at 2021-12-18 14:10 +0100: >Ok, I created the script: > >https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py > >The problem is it does _not_ crash, while a get a segfault using >pytest with python 3.9 on MacOS 10.15 > >Maybe it's because I'm using eval / exec in the script? Segfaults can result from C stack overflow which in turn can be caused in special cases by too deeply nested function calls (usually, Python's "maximal recursion depth exceeded" prevents this before a C stack overflow). Otherwise, whatever you do in Python (this includes "eval/exec") should not cause a segfault. The cause for it likely comes from a memory management bug in some C implemented part of your application. Note that memory management bugs may not show deterministic behavior. Minor changes (such as "with/without -v") can significantly change the outcome. -- https://mail.python.org/mailman/listinfo/python-list
Re: ANN: Dogelog, Invitation to the Moon!
Dear All, We are happy to announce a new edition of the Dogelog Player: - Binary Release: The new version 0.9.7 of the Dogelog player now masters DCG. We have decided to upload the transpiled and compacted editions for the JavaScript and Python platforms to www.xlog.ch under the heading "Products". - Quelltexte: As an alternative to GitHub, we have now activated pages.xlog.ch. In addition to the source texts of the transpiler and the built-ins of the Dogelog player, this website also contains the manuals and the source texts of the tutorials. Both JavaScript and Python source code can be found. - Blogging: As an alternative to Gist, we are now reporting on medium.com/@janburse_2989 about the developments in the Dogelog player. We also use twitter.com/dogelogch and www.facebook.com/groups/dogelog for short messages. Have Fun! #StaySafe Jan Burse, 18.12.2021 Mostowski Collapse schrieb: Dear All, The website http://www.xlog.ch/ is now open for newsletter registration. What are the plans for the next 10 years (sic!): - Past: 100% Java Prolog We had the Jekejeke Suite consisting of Runtime, Minlog and Debugger. The Runtime was initially 100% Java, including things like consult and top-level. - Present: 100% Prolog Prolog This year in 2021 we managed to deliver a new breed of Prolog, with the Dogelog Player we demonstrated a Prolog system which had most of it written in Prolog itself. The Dogelog Player is available for JavaScript and Python. - Future: The goal is to produce a Dogelog Suite consisting of Runtime, Minlog and Debugger, all based on the new 100% Prolog approach of the Dogelog Player. It is planned that Dogelog Suite will again cover Java, but we could also try novel targets such as Steel Bank Common Lisp (SBCL), etc.. Disclaimer: It might take some time till the new website http://www.xlog.ch/ shows some binaries, since we have removed us from GitHub. We are working on it. Have Fun! #StaySafe Jan Burse, 11.12.2021 -- https://mail.python.org/mailman/listinfo/python-list
Re: pytest segfault, not with -v
Emh, maybe I was not clear. I created a C extension and it segfaults. So I created that script to see where it segfaults. But the script does not segfault. My doubt is: is that because I'm using eval and exec in the script? On Sat, 18 Dec 2021 at 18:33, Dieter Maurer wrote: > > Marco Sulla wrote at 2021-12-18 14:10 +0100: > >Ok, I created the script: > > > >https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py > > > >The problem is it does _not_ crash, while a get a segfault using > >pytest with python 3.9 on MacOS 10.15 > > > >Maybe it's because I'm using eval / exec in the script? > > Segfaults can result from C stack overflow which in turn can > be caused in special cases by too deeply nested function calls > (usually, Python's "maximal recursion depth exceeded" prevents > this before a C stack overflow). > > Otherwise, whatever you do in Python (this includes "eval/exec") > should not cause a segfault. The cause for it likely comes from > a memory management bug in some C implemented part of your > application. > > Note that memory management bugs may not show deterministic > behavior. Minor changes (such as "with/without -v") > can significantly change the outcome. -- https://mail.python.org/mailman/listinfo/python-list
Re: pytest segfault, not with -v
> On 18 Dec 2021, at 20:05, Marco Sulla wrote: > > Emh, maybe I was not clear. I created a C extension and it segfaults. > So I created that script to see where it segfaults. But the script > does not segfault. My doubt is: is that because I'm using eval and > exec in the script? You may struggle to duplicate the pattern of memory allocations and accesses that lead to the segv. Assuming you are on a unix, turn on coredumps and run the code until it segfaults. Then use gdb on the core file. Barry > >> On Sat, 18 Dec 2021 at 18:33, Dieter Maurer wrote: >> >> Marco Sulla wrote at 2021-12-18 14:10 +0100: >>> Ok, I created the script: >>> >>> https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py >>> >>> The problem is it does _not_ crash, while a get a segfault using >>> pytest with python 3.9 on MacOS 10.15 >>> >>> Maybe it's because I'm using eval / exec in the script? >> >> Segfaults can result from C stack overflow which in turn can >> be caused in special cases by too deeply nested function calls >> (usually, Python's "maximal recursion depth exceeded" prevents >> this before a C stack overflow). >> >> Otherwise, whatever you do in Python (this includes "eval/exec") >> should not cause a segfault. The cause for it likely comes from >> a memory management bug in some C implemented part of your >> application. >> >> Note that memory management bugs may not show deterministic >> behavior. Minor changes (such as "with/without -v") >> can significantly change the outcome. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Call julia from Python: which package?
I played with Julia a few months ago. I was doing some data-science
stuff with Pandas and the performance was terrible so I decided to give
Julia a try.
My plan was to do the slow part in Julia and call it from Python.
I tried juliacall (if I don't remember wrong) but I couldn't set up it.
It wasn't as smooth as it was advertised (but hey! you may have a better
luck than me).
The other thing that I couldn't figure out is *how the data is shared
between Python and Julia*. Basically numpy/pandas structures cannot be
used by Julia own libraries as they are, they need to be copied at least
and this can be a real performance hit.
I desisted the idea but you may still consider this as an real option.
Just validate how much data you need to share (in my cases where quite
large datasets, hence the issue).
Having said that, is Julia a real option? May be.
In my case the processing that I needed was quite basic and Julia did
a really good job.
But I felt that the library is too fragmented. In Python you can relay
on Pandas/numpy for processing and matplotlib/seaborn for plotting and
you will 99% covered.
In Julia I need DataFrames.jl, Parquet.jl, CategoricalArrays.jl,
StatsBase.jl, Statistics.jl and Peaks.jl
And I'm not including any plotting stuff.
This fragmentation is not just "inconvenient" because you need to
install more packages but it is more difficult to code.
The API is not consistent so you need to be careful and double check
that what you are calling is really doing what you think.
About the performance, Julia is not magic. It depends on how well it was
coded each package.
In my case I had a good experience except with Parquet.jl which it
didn't understand how to handle categories in the dataset and ended up
loading a lot of duplicated strings and blew up the memory a few times.
I suggest you to write down what you need to speed up and see if it is
implemented in Julia (do a proof of concept). Only then consider to do
the switch.
Good luck and share your results!
Martin.
On Fri, Dec 17, 2021 at 07:12:22AM -0800, Dan Stromberg wrote:
On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam
wrote:
Hi,
I have a Python program that uses Tkinter for its GUI. It's rather slow so
I hope to replace many or all of the non-GUI parts by Julia code. Has
anybody experience with this? Any packages you can recommend? I found three
alternatives:
* https://pyjulia.readthedocs.io/en/latest/usage.html#
* https://pypi.org/project/juliacall/
* https://github.com/JuliaPy/PyCall.jl
Thanks in advance!
I have zero Julia experience.
I thought I would share this though:
https://stromberg.dnsalias.org/~strombrg/speeding-python/
Even if you go the Julia route, it's probably still best to profile your
Python code to identify the slow ("hot") spots, and rewrite only them.
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Custom designed alarm clock
I have designed a simple alarm using Python. It has about 10 limes. How do I convert it to an app that I can on my android Moto E power 2021 phone? Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Custom designed alarm clock
Suggested reading: https://pypi.org/project/python-for-android/ https://play.google.com/store/apps/details?id=org.qpython.qpy3 https://www.androidauthority.com/an-introduction-to-python-on-android-759685/ https://data-flair.training/blogs/android-app-using-python/ On Sat, 2021-12-18 at 18:36 -0500, Steve wrote: > > I have designed a simple alarm using Python. It has about 10 limes. > How do I convert it to an app that I can on my android Moto E power > 2021 > phone? > > Steve > -- https://mail.python.org/mailman/listinfo/python-list
RE: Custom designed alarm clock
Most of what is in that tutorial is old-hat for me. I have been coding Python for 10 years now and have a program with over 3000 lines of code. However, I am reminded that Kivy has to be reinstalled on my system and now I see that using Linux is the way to go. Buildozer? I also have to get the s14a library… Oh well, nothing is as simple as it first seems. It may take a while but I am willing to give it a go. Steve George Melly remarked to Mike Jagger on how lined his face was for one so young. Jagger replied “They’re laughter lines George” to which Melly countered: “Mick, nothing’s that f**king funny!”. From: Paul Bryan Sent: Saturday, December 18, 2021 7:50 PM To: Steve ; 'Python Main' Subject: Re: Custom designed alarm clock Suggested reading: https://pypi.org/project/python-for-android/ https://play.google.com/store/apps/details?id=org.qpython.qpy3 https://www.androidauthority.com/an-introduction-to-python-on-android-759685/ https://data-flair.training/blogs/android-app-using-python/ On Sat, 2021-12-18 at 18:36 -0500, Steve wrote: I have designed a simple alarm using Python. It has about 10 lines. How do I convert it to an app that I can on my android Moto E power 2021 phone? Steve -- https://mail.python.org/mailman/listinfo/python-list
