Re: Python extension module with callbacks into Python
> On 27 Oct 2020, at 19:21, Paul Grinberg wrote:
>
> As full disclosure, I posted this question on StackOverflow as well, but it
> looks like questions with [Python] [Extension-Module] tags are not frequently
> answered. The link to my question there is
> https://stackoverflow.com/questions/64559322/python-extension-module-with-callbacks-into-python
>
> I am running into unpredictable behavior with my Python extension module that
> wraps around a C++ library that starts a new pthread and, after doing some
> work, generates callbacks back into the caller. I've greatly simplified this
> to a simplistic example which still demonstrates this problem. The following
> will sometimes generate a Fatal Python error: PyEval_SaveThread: NULL tstate,
> usually rather quickly. Sometimes it SIGSEGV on tupledealoc. Occasionally
> this deadlocks. I am at a loss why. Does anyone have any ideas?
You did not say what OS and python version you are working with.
Why do you need to call PyEval_ThreadsInitialized()? its deprecated.
PyInitialize sets up threads unconditionally since 3.7.
When I have been in a similar situation I have used the debugger to
find out what the CPYTHON code is expecting and worked backwards from there.
Reading the python sources helps a lot.
Barry
>
> The relevant code for this is below, between the sections
>
> === python program ===
>
> import mymod
> from time import sleep
> from random import randrange
>
> def my_cb1(s):
>print("Python cb %s" % (s));
>
> for x in range(1,1000):
>num_cb = randrange(5) + 1
>print("Starting %d" % mymod.doit(my_cb1, "myid" + str(x), num_cb))
>
> while True:
>sleep(1)
>
> === Extension Module ===
>
> #include
>
> #define PY_SSIZE_T_CLEAN
> #include
> #include
>
> #include
> #include
> #include
> #include
> #include
> #include
>
> static std::map cb_map;
> static std::mutex map_mtx;
>
> struct fake_cb_info
> {
> fake_cb_info() = delete;
> fake_cb_info(const unsigned long &num_cb, const std::string &id) :
>num_cb(num_cb), id(id)
> {
> }
> const unsigned long num_cb;
> const std::string id;
> };
> static std::deque deq;
> static std::mutex deq_mtx;
>
> static bool is_worker_thread_running = false;
> static std::thread worker_thread;
>
> typedef std::function
> doit_cb_t;
> static void internal_cb(const std::string &id, const std::string &s)
> {
> std::scoped_lock lk(map_mtx);
>
> if (0 != cb_map.count(id))
> {
> PyGILState_STATE gstate;
> gstate = PyGILState_Ensure();
>
> PyObject *arglist = Py_BuildValue("(s)", s.c_str());
> PyObject *result = PyObject_CallObject(cb_map.at(id), arglist);
> Py_DECREF(arglist);
>
> if (NULL == result)
> {
> if (NULL == PyErr_Occurred())
> {
>std::cerr << "Unknown error occurred in C callback" << std::endl;
> }
> else
> {
> PyErr_Print();
> }
> }
> else
> {
>Py_DECREF(result);
> }
>
> PyGILState_Release(gstate);
> }
> else
> {
>std::cerr << "Unknown callback id " << id << std::endl;
> }
> }
>
> void static worker()
> {
> size_t x = 0;
> while(true)
> {
>std::scoped_lock lk(deq_mtx);
>if (deq.size() == 0)
>{
> usleep(1000);
> continue;
>}
>
>auto info = deq.front();
>deq.pop_front();
>for (unsigned long i=0; i{
> internal_cb(info.id, std::to_string(x++));
>}
> }
> }
>
> PyObject * _wrap_doit(void *self, PyObject *args, PyObject *kwargs)
> {
>PyObject *py_retval;
>PyThreadState *py_thread_state = NULL;
>PyObject *cb;
>const char *id = NULL;
>Py_ssize_t id_len;
>std::string id_std;
>unsigned long num_callbacks;
>const char *keywords[] = {"cb_func", "id", "num_cb", NULL};
>
>if (!PyArg_ParseTupleAndKeywords(args, kwargs, (char *) "Os#k", (char **)
> keywords, &cb, &id, &id_len, &num_callbacks))
>{
>abort();
>}
>if (!PyCallable_Check(cb))
>{
>abort();
>}
>
>id_std = std::string(id, id_len);
>
>{
> std::scoped_lock lk(map_mtx);
> if (0 == cb_map.count(id_std))
> {
>Py_INCREF(cb);
>cb_map.insert(std::make_pair(id_std, cb));
>
>// N.B. The corresponding Py_DECREF for the callback function PyObject
>// is intentionally not here. It is in another extension module method
>// that is not listed here (just trying to keep this example as small
>// and lean as possible)
> }
> else
> {
>std::cerr << "Only one callback for ID!" << std::endl;
>abort();
> }
>}
>
>if (PyEval_ThreadsInitialized ())
>{
> std::cout << "Saving thread" << std::endl;
> py_thread_state = PyEval_SaveThread();
>}
>
>{
> // Stash away the info so that we will know how many callbacks to
> // generate and sleep a bit. This is to simulate a real external library
>
Live Write to File with Code That is Reading File and Writing to Serial Port
Currently have a code that takes in a .txt file and submits commands to the
serial. Then it reads the reply from the serial port and writes it to a
hardcoded .txt file. The problem is it doesn't save it live and so if I need to
stop the code for any reason, I can't gather current data and the text file is
blank. I'm not as familiar with buffering and things like that and tried
"outputFile = open("./outputFile.txt", "a", 0)" but that gave me the error
"can't have an unbuffered text I/O” in python 3?" so I'm not sure what to do.
Here is the general layout if you would like to mess around with it:
with open(test_file) as file_test:
Lines = file_test.readlines()
for line in Lines:
#send_str is the command to send to the serial port
send_str = line
file_result.write(line + "\n") #<--- if I were to cancel out after
this it wouldn't be saved(*)
ser.write(send_str.encode('utf-8'))
time.sleep(send_pause)
reply_str = ser.readline().decode('utf-8').strip()
file_result.write("reply:" + reply_str + "\n") #<---(*)
file_result.write('\n') #<---(*)
anything helps thank you in advanced
--
https://mail.python.org/mailman/listinfo/python-list
Re: Symlinks already present
In article ,
Grant Edwards wrote:
>On 2020-09-01, Richard Damon wrote:
>
>> Remember, we are talking about a hypothetical OS that handles hardlinks
>> to directories, and defines that .. will point to the parent used to
>> come to it, NOT just having current *NIX allowing hardlinks to
>> directories with no remediation of the issues cause.
>
>I can testify from personal experience that SunOS 3/4 was in the
>latter category. After creating a hard-link to a directory, things
>like fsck and the filesystem dump backup utility got very upset and
>confused. IIRC the only way to recover was to nuke the involved
>inodes then let fsck try to pick up the pieces and put them in the
>lost+found. IIRC, I managed to recover without losing any files, but
>it wasn't a fun day.
It was a defect ("bug") in the SUNOS that this was possible.
So reread this thread.
"Let us imagine the situation that a severe, known defect was
reintroduced in linux, just for the fun of it."
>
>--
>Grant
Groetjes Albert
--
This is the first day of the end of your life.
It may not kill you, but it does make your weaker.
If you can't beat them, too bad.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python extension module with callbacks into Python
> > I am running into unpredictable behavior with my Python extension module > > that wraps around a C++ library that starts a new pthread and, after doing > > some work, generates callbacks back into the caller. I've greatly > > simplified this to a simplistic example which still demonstrates this > > problem. The following will sometimes generate a Fatal Python error: > > PyEval_SaveThread: NULL tstate, usually rather quickly. Sometimes it > > SIGSEGV on tupledealoc. Occasionally this deadlocks. I am at a loss why. > > Does anyone have any ideas? > You did not say what OS and python version you are working with. I apologize. I am running this on Ubuntu 18.04 under Python 3.6 (default for this Ubuntu's Bionic release) fully updated to the latest provided by Ubuntu's standard repos, which is 3.6.9-1~18.04ubuntu1.3. After posting this question here I continued investigating this problem and found that when I repeat the same test under Python 3.7.3 or 3.8.5 (installed via PyEnv), there are no issues. Then, I went back to the most recent 3.6 release which is 3.6.12 (also installed via PyEnv) and again saw the issue. This seems to suggest that the problem is in 3.6.x Python. > Why do you need to call PyEval_ThreadsInitialized()? its deprecated. > > PyInitialize sets up threads unconditionally since 3.7. I did not realize that PyEval_ThreadsInitialized() is deprecated. However, I was also running under 3.6. I will move away from that once I move to a newer version of Python -- https://mail.python.org/mailman/listinfo/python-list
Re: Symlinks already present
In article , Cameron Simpson wrote: > >Yeah, makes me ill. That's because these days "pwd" is usually a shell >builtin with funny semantics and a cache/sanity=check against $PWD >(which gets computed as you cd around, typically). And if has a -P >option and friends explicitly because of this hideous stuff. Would you be a fan of nash , the Never Again so complicated SHell ? The base idea that it is a Forth interpreter that has a few shell like features tucked onto it, like executing programs. And if you type pwd, sure as hell it would look through $PATH and execute the exact program you want. (You could defined `pwd' in the interpreter yourself to do something different. If you do that, probably you want it.) > >Cheers, >Cameron Simpson Groetjes Albert -- This is the first day of the end of your life. It may not kill you, but it does make your weaker. If you can't beat them, too bad. albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: Live Write to File with Code That is Reading File and Writing to Serial Port
> On Oct 28, 2020, at 5:49 AM, ktkelly_1 wrote:
>
> Currently have a code that takes in a .txt file and submits commands to the
> serial. Then it reads the reply from the serial port and writes it to a
> hardcoded .txt file. The problem is it doesn't save it live and so if I need
> to stop the code for any reason, I can't gather current data and the text
> file is blank. I'm not as familiar with buffering and things like that and
> tried "outputFile = open("./outputFile.txt", "a", 0)" but that gave me the
> error "can't have an unbuffered text I/O” in python 3?" so I'm not sure what
> to do. Here is the general layout if you would like to mess around with it:
from os import fsync
> with open(test_file) as file_test:
>Lines = file_test.readlines()
>for line in Lines:
>#send_str is the command to send to the serial port
>send_str = line
>file_result.write(line + "\n") #<--- if I were to cancel out after
> this it wouldn't be saved(*)
file_result.flush()
os.fsync()
>
>ser.write(send_str.encode('utf-8'))
>time.sleep(send_pause)
>reply_str = ser.readline().decode('utf-8').strip()
>file_result.write("reply:" + reply_str + "\n") #<---(*)
>file_result.write('\n') #<---(*)
file_result.flush()
os.fsync()
Karen
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python extension module with callbacks into Python
> On 28 Oct 2020, at 13:25, Paul Grinberg wrote: > >>> I am running into unpredictable behavior with my Python extension module >>> that wraps around a C++ library that starts a new pthread and, after doing >>> some work, generates callbacks back into the caller. I've greatly >>> simplified this to a simplistic example which still demonstrates this >>> problem. The following will sometimes generate a Fatal Python error: >>> PyEval_SaveThread: NULL tstate, usually rather quickly. Sometimes it >>> SIGSEGV on tupledealoc. Occasionally this deadlocks. I am at a loss why. >>> Does anyone have any ideas? > >> You did not say what OS and python version you are working with. > > I apologize. I am running this on Ubuntu 18.04 under Python 3.6 (default for > this Ubuntu's Bionic release) fully updated to the latest provided by > Ubuntu's standard repos, which is 3.6.9-1~18.04ubuntu1.3. After posting this > question here I continued investigating this problem and found that when I > repeat the same test under Python 3.7.3 or 3.8.5 (installed via PyEnv), there > are no issues. Then, I went back to the most recent 3.6 release which is > 3.6.12 (also installed via PyEnv) and again saw the issue. This seems to > suggest that the problem is in 3.6.x Python. > >> Why do you need to call PyEval_ThreadsInitialized()? its deprecated. >> >> PyInitialize sets up threads unconditionally since 3.7. > > I did not realize that PyEval_ThreadsInitialized() is deprecated. However, I > was also running under 3.6. I will move away from that once I move to a newer > version of Python Try calling PyEval_InitThreads() to force the python threading to be all setup. Barry > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Python extension module with callbacks into Python
> Try calling PyEval_InitThreads() to force the python threading to be all > setup. Can you please clarify where/when I should call PyEval_InitThreads()? Is this in the main python thread before any pthread callbacks are generated? If so, should this be done only once? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python extension module with callbacks into Python
> On 28 Oct 2020, at 15:22, Paul Grinberg wrote: > > >> >> Try calling PyEval_InitThreads() to force the python threading to be all >> setup. > > Can you please clarify where/when I should call PyEval_InitThreads()? Is this > in the main python thread before any pthread callbacks are generated? If so, > should this be done only once? Do it in your module init. That function is safe to be called multiple time. You could have answered the second part by reading the docs, which is what I did to check what I was telling you. Personally I do not code against the C api I use PyCXX to ease a lot of the hardship of using the C api. See http://cxx.sourceforge.net/PyCXX-Python3.html Disclaimer I am the PyCXX maintainer. I have c++ classes to handle the Gil release and acquire that I use in pysvn that you might find useful. See http://cxx.sourceforge.net/PyCXX-Python3.html and the PythonAllowThreads PythonDisallowThreads classes that to the hard workin a type safe way. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Live Write to File with Code That is Reading File and Writing to Serial Port
On 10/28/2020 8:49 AM, ktkelly_1 wrote:
Currently have a code that takes in a .txt file and submits commands to the serial. Then it reads the reply from the
serial port and writes it to a hardcoded .txt file. The problem is it doesn't save it live and so if I need to stop the
code for any reason, I can't gather current data and the text file is blank. I'm not as familiar with buffering and
things like that and tried "outputFile = open("./outputFile.txt", "a", 0)" but that gave
me the error "can't have an unbuffered text I/O” in python 3?" so I'm not sure what to do. Here is the
general layout if you would like to mess around with it:
with open(test_file) as file_test:
Lines = file_test.readlines()
for line in Lines:
#send_str is the command to send to the serial port
send_str = line
Both input and output should be controlled by the with statement. Same
for serial port? Open files are line iterables. readlines is only
needed if you need the whole file at once. Two names for the same line
is a buglet.
with open(test_file) as f_in, open(results) as f_out:
for command in f_in: # Sent to serial port
file_result.write(line + "\n") #<--- if I were to cancel out after
this it wouldn't be saved(*)
Do you really want a command at the end of the file without results? If
not, write command and result in one write after getting result.
ser.write(send_str.encode('utf-8'))
time.sleep(send_pause)
reply_str = ser.readline().decode('utf-8').strip()
file_result.write("reply:" + reply_str + "\n") #<---(*)
file_result.write('\n') #<---(*)
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
