c bindings with non-python thread callback while python exits
I have a C++ library (luckily with source code, so I know what's going on) to which I wrote c bindings. The library internally starts an event pthread which generates callbacks back into python. The c binding for processing these callbacks look like this (simplified for convenience): PyGILState_STATE gstate; gstate = PyGILState_Ensure(); PyObject_CallObject(cb, arglist); PyGILState_Release(gstate); This works perfectly while the python program using this c binding is running. However, when the program terminates normally and there happens to be one last callback underway, the c bindings hang. When viewed with GDB, I can see that the callback thread callstack looks like this: futex_abstimed_wait_cancelable() _pthread_cond_wait_common() __pthread_cond_timedwait() PyCOND_TIMEDWAIT() take_gil() PyEval_RestoreThread() PyGILState_Ensure() c_bindings_callback() cpp_library_callback() and the main thread callstack, which is where the python interpreter was running, looks like this __pthread_clockjoin_ex () std::thread::join() cpp_library_static_singleton_class_destructor () __run_exit_handlers () __GI_exit () __libc_start_main () _start () in other words, the program hangs because the main thread is waiting for the event pthread to join, but that thread is stuck in a callback waiting for the GIL. What is the right way to prevent this problem from happening? Thank you in advance, Paul. P.S. I am running on Linux: ubuntu 18.04 with python 3.6.9, also reproduced with python 3.7.5, as well as ubuntu 20.04 with python 3.8.5 -- https://mail.python.org/mailman/listinfo/python-list
Re: help
Hi help me to repair this erorr: Warning: translation file 'git_en_US' could not be loaded. Using default. and download python39.dll On 1/26/21, Maziar Ghasemi wrote: > Hi > help me to repair this erorr: > Warning: translation file 'git_en_US' could not be loaded. > Using default. > and download python39.dll > -- > mazyar ghasemi > MS student of Razi university > -- mazyar ghasemi MS student of Razi university -- https://mail.python.org/mailman/listinfo/python-list
Re: help
On 1/26/2021 4:01 AM, Maziar Ghasemi wrote: Hi help me to repair this erorr: Warning: translation file 'git_en_US' could not be loaded. Using default. and download python39.dll Please do not repost, especially the same day. On 1/26/21, Maziar Ghasemi wrote: Hi help me to repair this erorr: Warning: translation file 'git_en_US' could not be loaded. Using default. and download python39.dll You ran some python software that wants a missing translation file. Ask the help forum for that software. If you ran Eric IDE, perhaps https://tracker.die-offenbachs.homelinux.org/eric/issue293 will help. I found this by searching the web (google) for 'git_en_us'. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: help
On 1/26/21 9:05 AM, Terry Reedy wrote: On 1/26/2021 4:01 AM, Maziar Ghasemi wrote: ... Please do not repost, especially the same day. Maziar Ghasemi appears to be a new user, and had to sign up to the Python List mailing list. Are you seeing his request twice because you use gmane? -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: c bindings with non-python thread callback while python exits
> On 26 Jan 2021, at 14:35, Paul Grinberg wrote: > > I have a C++ library (luckily with source code, so I know what's going on) to > which I wrote c bindings. The library internally starts an event pthread > which generates callbacks back into python. The c binding for processing > these callbacks look like this (simplified for convenience): > > PyGILState_STATE gstate; > gstate = PyGILState_Ensure(); > PyObject_CallObject(cb, arglist); > PyGILState_Release(gstate); > > This works perfectly while the python program using this c binding is > running. However, when the program terminates normally and there happens to > be one last callback underway, the c bindings hang. When viewed with GDB, I > can see that the callback thread callstack looks like this: > > futex_abstimed_wait_cancelable() > _pthread_cond_wait_common() > __pthread_cond_timedwait() > PyCOND_TIMEDWAIT() > take_gil() > PyEval_RestoreThread() > PyGILState_Ensure() > c_bindings_callback() > cpp_library_callback() > > and the main thread callstack, which is where the python interpreter was > running, looks like this > > __pthread_clockjoin_ex () > std::thread::join() > cpp_library_static_singleton_class_destructor () > __run_exit_handlers () > __GI_exit () > __libc_start_main () > _start () > > in other words, the program hangs because the main thread is waiting for the > event pthread to join, but that thread is stuck in a callback waiting for the > GIL. > > What is the right way to prevent this problem from happening? As you say you have dead locked. My guess is that you have to shutdown your extension before you quit python. You might find pycxx interesting if you are wrapping C++ code for use with python. http://cxx.sourceforge.net/ Barry > Thank you in advance, > Paul. > > P.S. I am running on Linux: ubuntu 18.04 with python 3.6.9, also reproduced > with python 3.7.5, as well as ubuntu 20.04 with python 3.8.5 > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
Hello everyone, I'm a long time Matlab and R user working on data science. How do you troubleshooting/debugging in Python? I ran into this impossible situation to debug: class person: def __init__(self, id, created_at, name, attend_date, distance): """Create a new `person`. """ self._id = id self.created_at = created_at self.name = name self.attend_date = attend_date self.distance = distance @classmethod def get_person(self, employee): """Find and return a person by. """ return person(employee['created_at'], employee['id'], employee['name'], employee['attend_date'], employee['distance'] ) I got an error message saying id was 'str', but expecting 'int'. In R, I use the interactive IDE with console. Wherever the error interrupts the code, I just evaluate that line in the console. Very convenient! If this error was in R, I would have tried: > self._id = 123 But, I can't do that here! What do I do? Do I HAVE TO instantiate an object first? It's not convenient if I have 10 of these objects around. I need to instantiate 10 objects. I know hardcore computer scientists would tell me about Python debugger. R also has one, but nobody ever uses it. I don't find them user-friendly! Thanks a lot, Mike -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 2021-01-26, C W wrote: > How do you troubleshooting/debugging in Python? Mostly I read exception trace and read the code and think about it. If that doesn't work, I add some print() or syslog() calls. If I really get stuck, I try to write as small a program as possible that demonstrates the problem. > I know hardcore computer scientists would tell me about Python debugger. I've been writing Python for 20+ years. I've never tried a debugger. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
CW> How do you troubleshooting/debugging in Python? GE> Mostly I read exception trace and read the code and think about it. GE> If that doesn't work, I add some print() or syslog() calls. CW> I know hardcore computer scientists would tell me about Python debugger. GE> I've been writing Python for 20+ years. I've never tried a debugger. Agree with Grant on these points. I certainly use gdb to debug C code (like the interpreter), but for Python code, tracebacks and print statements pretty much take care of things. I know pdb has been around for a long while and many people like IDEs, but I've not generally found them all that useful. I'm stuck in the 90s I guess. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 26Jan2021 14:00, C W wrote: >I'm a long time Matlab and R user working on data science. How do you >troubleshooting/debugging in Python? > >I ran into this impossible situation to debug: >class person: >def __init__(self, id, created_at, name, attend_date, distance): >"""Create a new `person`. >""" Your mailer has removed all the indenting. See if it has a mode for code. Often just making sure all the code is itself indented will do, eg: some indented code here >I got an error message saying id was 'str', but expecting 'int'. There should have been a complete traceback with that error, showing which line triggered the exception and how it was called. Here, we don't know from your description what happened or where. >If this error was in R, I would have tried: >> self._id = 123 Well, provided "self" already existed, that would work in Python but tell you almost nothing, because any variable may reference a value of any type. (Python variables are not typed, but the values _are_.) >But, I can't do that here! What do I do? Do I HAVE TO instantiate an >object >first? Well, "self" is the instantiaited object. The __init__ function does more setup (completing what you might call "normal" instantiation). >It's not convenient if I have 10 of these objects around. I need to >instantiate 10 objects. Your environment isn't clear to me. >I know hardcore computer scientists would tell me about Python debugger. R >also has one, but nobody ever uses it. I don't find them user-friendly! Me either. Like grant, i fall into the "look at the stack trace and think a bit, then put in print() calls or similar to show me whether things are what I expect them to be when the code runs". I actually have a Computer Science degree, but I think there are definition those who like debuggers and those who tend not to. I'm in the latter camp. Preferences like that affact how you _choose_ to debug. Usually I have the code in a file in one window and a command line in another, and go: python my_code_file.py ... to test when debugging. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
> On Jan 26, 2021, at 2:00 PM, C W wrote: > > Hello everyone, > > I'm a long time Matlab and R user working on data science. How do you > troubleshooting/debugging in Python? > Another approach is to run the code in an IDE. I happen to use Wing, but that is a coincidence. But almost ANY IDE will let you set a break point, then single-step through your code starting at the break point and examine the values of your variables at each step. Sometimes this is an awfully big hammer for what is a head-slapping mistake. But it has never failed me. Good luck, Bill -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On Jan 26, 2021, at 18:16, Grant Edwards wrote: > >> How do you troubleshooting/debugging in Python? > > Mostly I read exception trace and read the code and think about it. > > If that doesn't work, I add some print() or syslog() calls. > > If I really get stuck, I try to write as small a program as possible > that demonstrates the problem. I do the first two, but if I get really stuck, I use the pudb debugger (https://pypi.org/project/pudb/). Using that, I can see all the locals, jump to any point in the stack and see the locals there, or shell into ipython if I need to run some quick code. For me, this is much faster than trying to write an additional program that is close enough to the problem code to be useful. -- Ed Leafe -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
Thanks for your replies. My apologies for the poor indent. I'm rewriting the code below. class NEODatabase: def __init__(self, id, created_at, name, attend_date, distance): self._id = id self.created_at = created_at self.name = name self.attend_date = attend_date self.distance = distance @classmethod def get_person(self, employee): return PERSONDatabase(employee['created_at'], employee['id'], employee['name'], employee['attend_date'], employee['distance']) I have a naive question. How do I use traceback or trace the stack? In particular, I'm using VS Code with Python interactive console. Say, I want to print the value of employee['name']. Can I do it? My understanding is that these classes are just "skeletons". I must create an instance, assign values, then test? Thanks so much, Mike On Tue, Jan 26, 2021 at 9:55 PM Ed Leafe wrote: > On Jan 26, 2021, at 18:16, Grant Edwards > wrote: > > > >> How do you troubleshooting/debugging in Python? > > > > Mostly I read exception trace and read the code and think about it. > > > > If that doesn't work, I add some print() or syslog() calls. > > > > If I really get stuck, I try to write as small a program as possible > > that demonstrates the problem. > > I do the first two, but if I get really stuck, I use the pudb debugger ( > https://pypi.org/project/pudb/). > > Using that, I can see all the locals, jump to any point in the stack and > see the locals there, or shell into ipython if I need to run some quick > code. For me, this is much faster than trying to write an additional > program that is close enough to the problem code to be useful. > > -- Ed Leafe > > > > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 2021-01-27, Cameron Simpson wrote: > Me either. Like grant, i fall into the "look at the stack trace and > think a bit, then put in print() calls or similar to show me whether > things are what I expect them to be when the code runs". > > I actually have a Computer Science degree, but I think there are > definition those who like debuggers and those who tend not to. Me too (MS in CSci), but I can't remember the last time I used a debugger. It may help that I "grow" my apps. I don't sit down, write a 3000 line program, and then try to debug the program. That just doesn't work. It doesn't matter how much UML you can point to if a couple of your basic assumptions turned out to be wrong. Write small funtions, and add small chunks of code, and test as you go to make sure things work the way you thought they were going to. If it used to work, you've only added 5 lines of code, and now it doesn't work, it's usually not hard to figure out what's wrong. And don't be afraid to refactor stuff when your original approach starts to smell -- if you just keep going it will only get worse. Source control can also be a lifesaver if you get really confused. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On Tue, Jan 26, 2021 at 4:01 PM C W wrote: > Hello everyone, > > I'm a long time Matlab and R user working on data science. How do you > troubleshooting/debugging in Python? > I frequently read tracebacks and think about what's up in the code. I also often add print functions or logging - empiricism often beats theorizing when the problems are weird. And once in a while I will use pudb - it's probably a pretty good fit for a vim user like me, both being curses-based. pdb is sad. There are other debuggers for Python: https://wiki.python.org/moin/PythonDebuggingTools -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On Tue, Jan 26, 2021 at 8:13 PM Dan Stromberg wrote: > > On Tue, Jan 26, 2021 at 4:01 PM C W wrote: > >> Hello everyone, >> >> I'm a long time Matlab and R user working on data science. How do you >> troubleshooting/debugging in Python? >> > > I frequently read tracebacks and think about what's up in the code. > > I also often add print functions or logging - empiricism often beats > theorizing when the problems are weird. > > And once in a while I will use pudb - it's probably a pretty good fit for > a vim user like me, both being curses-based. pdb is sad. There are other > debuggers for Python: > https://wiki.python.org/moin/PythonDebuggingTools > BTW, there's a new tool I haven't tried yet, that sounds pretty interesting, especially for newcomers: It's called "friendly traceback" and can be found at https://pypi.org/project/friendly-traceback/ Oh, and of course google search is a terrific tool for deciphering error messages. -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 1/26/21 8:37 PM, C W wrote: > I have a naive question. How do I use traceback or trace the stack? In > particular, I'm using VS Code with Python interactive console. Show us the traceback here and we can help you interpret it. Copy and paste it from the VS Code console. > Say, I want to print the value of employee['name']. Can I do it? Yes I would think so. > My understanding is that these classes are just "skeletons". I must > create an instance, assign values, then test? Can't you just do something like this? class NEODatabase: def __init__(self, id, created_at, name, attend_date, distance): self._id = id self.created_at = created_at self.name = name self.attend_date = attend_date self.distance = distance @classmethod def get_person(self, employee): print (employee['name']) return PERSONDatabase(employee['created_at'], employee['id'], employee['name'], employee['attend_date'], employee['distance']) -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 1/26/21 8:30 PM, Grant Edwards wrote: > Me too (MS in CSci), but I can't remember the last time I used a > debugger. I use a debugger frequency in C++, and sometimes C. Even running a debugger on an attached device like an Arduino is sometimes very useful. Good debuggers let you do things like conditional breakpoints and watch expressions. Helpful for tracking down hard-to-find and less frequent conditions. Sure you could print stuff out, which I do, but sometimes it's more convenient to do it with the debugger. -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
Hi Michael, Here's the code again, class should be called PERSONDatabase, misspelled earlier: class PERSONDatabase: def __init__(self, id, created_at, name, attend_date, distance): self._id = id self.created_at = created_at self.name= name self.attend_date = attend_date self.distance = distance @classmethod def get_person(self, employee): return PERSONDatabase(employee['created_at'], employee['id'], employee['name'], employee['attend_date'], employee['distance']) The PERSONDatabase class is called from main. This is the trace back I got from the VS code: Traceback (most recent call last): File "/Users/Mike/Documents/Mike/main.py", line 95, in main() File "/Users/Mike/Documents/Mike/main.py", line 86, in main args = get_feed() File "/Users/Mike/DocumentsMike/main.py", line 32, in get_feed result = [PERSONatabase.get_person(raw_person) for raw_neo in raw_objects] File "/Users/Mike/Documents/Mike/main.py", line 32, in result = [NEODatabase.get_person(raw_person) for raw_neo in raw_objects] File "/Users/Mike/Documents/Mike/database.py", line 24, in get_person return PERSONDatabase(person['created_at'], KeyError: 'created_at' Thank you very much! On Wed, Jan 27, 2021 at 12:10 AM Michael Torrie wrote: > On 1/26/21 8:37 PM, C W wrote: > > I have a naive question. How do I use traceback or trace the stack? In > > particular, I'm using VS Code with Python interactive console. > > Show us the traceback here and we can help you interpret it. Copy and > paste it from the VS Code console. > > > Say, I want to print the value of employee['name']. Can I do it? > > Yes I would think so. > > > My understanding is that these classes are just "skeletons". I must > > create an instance, assign values, then test? > > Can't you just do something like this? > > class NEODatabase: > def __init__(self, id, created_at, name, attend_date, distance): > self._id = id > self.created_at = created_at > self.name = name > self.attend_date = attend_date > self.distance = distance > > @classmethod > def get_person(self, employee): > print (employee['name']) > > return PERSONDatabase(employee['created_at'], > employee['id'], > employee['name'], > employee['attend_date'], > employee['distance']) > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
To debug python code I use spyder from the anaconda distribution Am Mittwoch, 27. Januar 2021 schrieb C W : > Hi Michael, > Here's the code again, class should be called PERSONDatabase, misspelled > earlier: > class PERSONDatabase: >def __init__(self, id, created_at, name, attend_date, distance): > self._id = id > self.created_at = created_at > self.name= name > self.attend_date = attend_date > self.distance = distance > >@classmethod >def get_person(self, employee): > return PERSONDatabase(employee['created_at'], > employee['id'], > employee['name'], > employee['attend_date'], > employee['distance']) > > The PERSONDatabase class is called from main. This is the trace back I got > from the VS code: > > Traceback (most recent call last): >File "/Users/Mike/Documents/Mike/main.py", line 95, in > main() >File "/Users/Mike/Documents/Mike/main.py", line 86, in main > args = get_feed() >File "/Users/Mike/DocumentsMike/main.py", line 32, in get_feed > result = [PERSONatabase.get_person(raw_person) for raw_neo in > raw_objects] >File "/Users/Mike/Documents/Mike/main.py", line 32, in > result = [NEODatabase.get_person(raw_person) for raw_neo in > raw_objects] >File "/Users/Mike/Documents/Mike/database.py", line 24, in get_person > return PERSONDatabase(person['created_at'], > KeyError: 'created_at' > > Thank you very much! > > On Wed, Jan 27, 2021 at 12:10 AM Michael Torrie wrote: > > > On 1/26/21 8:37 PM, C W wrote: > > > I have a naive question. How do I use traceback or trace the stack? In > > > particular, I'm using VS Code with Python interactive console. > > > > Show us the traceback here and we can help you interpret it. Copy and > > paste it from the VS Code console. > > > > > Say, I want to print the value of employee['name']. Can I do it? > > > > Yes I would think so. > > > > > My understanding is that these classes are just "skeletons". I must > > > create an instance, assign values, then test? > > > > Can't you just do something like this? > > > > class NEODatabase: > > def __init__(self, id, created_at, name, attend_date, distance): > > self._id = id > > self.created_at = created_at > > self.name = name > > self.attend_date = attend_date > > self.distance = distance > > > > @classmethod > > def get_person(self, employee): > > print (employee['name']) > > > > return PERSONDatabase(employee['created_at'], > > employee['id'], > > employee['name'], > > employee['attend_date'], > > employee['distance']) > > > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- Regards, Joseph Pareti - Artificial Intelligence consultant Joseph Pareti's AI Consulting Services https://www.joepareti54-ai.com/ cell +49 1520 1600 209 cell +39 339 797 0644 -- https://mail.python.org/mailman/listinfo/python-list
Re: How do you debug in Python? Coming from a Matlab and R user. I'm already aware of pdb.
On 27Jan2021 00:19, C W wrote: >Here's the code again, class should be called PERSONDatabase, >misspelled >earlier: >class PERSONDatabase: > def __init__(self, id, created_at, name, attend_date, distance): > self._id = id > self.created_at = created_at > self.name= name > self.attend_date = attend_date > self.distance = distance Here's you're setting attributes (which is a very normal thing to do). > @classmethod > def get_person(self, employee): > return PERSONDatabase(employee['created_at'], >employee['id'], >employee['name'], >employee['attend_date'], >employee['distance']) I think this "employee" is called "person" in the code the traceback came from. It is better when these two things match. >The PERSONDatabase class is called from main. This is the trace back I got >from the VS code: > >Traceback (most recent call last): > File "/Users/Mike/Documents/Mike/main.py", line 95, in > main() > File "/Users/Mike/Documents/Mike/main.py", line 86, in main > args = get_feed() > File "/Users/Mike/DocumentsMike/main.py", line 32, in get_feed > result = [PERSONatabase.get_person(raw_person) for raw_neo in >raw_objects] > File "/Users/Mike/Documents/Mike/main.py", line 32, in > result = [NEODatabase.get_person(raw_person) for raw_neo in >raw_objects] > File "/Users/Mike/Documents/Mike/database.py", line 24, in get_person > return PERSONDatabase(person['created_at'], >KeyError: 'created_at' Here's you're trying to index another object using a string, which seems to resemble the .created_at attribute in your PERSONDatabase object. I would presume from this that the "person" object at the bottom of the traceback is the "raw_person" called above it. But I do not see raw_person defined anywhere. Are you sure you didn't mean to pass "raw_neo" instead of "raw_person"? That would be more normal, since you're iterating over "raw_objects". Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
