> On 8 Dec 2021, at 17:11, Jen Kris <[email protected]> wrote:
>
> I started this post on November 29, and there have been helpful comments
> since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris
> Angelico. Thanks to all of you.
>
> I've found a solution that works for my purpose, and I said earlier that I
> would post the solution I found. If anyone has a better solution I would
> appreciate any feedback.
>
> To recap, I'm using a pair of named pipes for IPC between C and Python.
> Python runs as a child process after fork-execv. The Python program
> continues to run concurrently in a while True loop, and responds to requests
> from C at intervals, and continues to run until it receives a signal from C
> to exit. C sends signals to Python, then waits to receive data back from
> Python. My problem was that C was blocked when Python started.
>
> The solution was twofold: (1) for Python to run concurrently it must be a
> multiprocessing loop (from the multiprocessing module), and (2) Python must
> terminate its write strings with \n, or read will block in C waiting for
> something that never comes. The multiprocessing module sidesteps the GIL;
> without multiprocessing the GIL will block all other threads once Python
> starts.
>
> Originally I used epoll() on the pipes. Cameron Smith and Barry Scott
> advised against epoll, and for this case they are right. Blocking pipes work
> here, and epoll is too much overhead for watching on a single file
> descriptor.
>
> This is the Python code now:
>
> #!/usr/bin/python3
> from multiprocessing import Process
You already have feedback that multiprocessing is not required.
> import os
>
> print("Python is running")
>
> child_pid = os.getpid()
> print('child process id:', child_pid)
>
> def f(a, b):
>
> print("Python now in function f")
>
> pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
> print("File Descriptor1 Opened " + str(pr))
> pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
> print("File Descriptor2 Opened " + str(pw))
>
> while True:
>
> v = os.read(pr,64)
> print("Python read from pipe pr")
> print(v)
>
> if v == b'99':
> os.close(pr)
> os.close(pw)
> print("Python is terminating")
> os._exit(os.EX_OK)
>
> if v != "Send child PID":
> os.write(pw, b"OK message received\n")
The \n should not be required as UDS (unix domain sockets) are message based
not a stream of bytes.
> print("Python wrote back")
>
> if __name__ == '__main__':
> a = 0
> b = 0
> p = Process(target=f, args=(a, b,))
> p.start()
> p.join()
>
> The variables a and b are not currently used in the body, but they will be
> later.
>
> This is the part of the C code that communicates with Python:
>
> fifo_fd1 = open(fifo_path1, O_WRONLY);
> fifo_fd2 = open(fifo_path2, O_RDONLY);
>
> status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
> if (status_write < 0) perror("write");
You continue on after the error, exit would be better.
>
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
> if (status_read < 0) perror("read");
> printf("C received message 1 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
The length of the data read is in status_read not buf_len.
>
> status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
How much data did you put into py_msg_01 buffer?
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending
the trailing NUL (0)
of the C string.
> if (status_write < 0) perror("write");
If it failed exit until you have figured out the error handling.
>
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
> if (status_read < 0) perror("read");
> printf("C received message 2 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
The length of the data read is in status_read not buf_len.
At no point do I see in this C code the need for a \n in the data sent between
python and C.
>
> // Terminate Python multiprocessing
> printf("C is sending exit message to Python\n");
> status_write = write(fifo_fd1, py_msg_03, 2);
Would be better to not be using "2" for the length to write here.
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).
>
> printf("C is closing\n");
> close(fifo_fd1);
> close(fifo_fd2);
>
> Screen output:
>
> Python is running
> child process id: 5353
> Python now in function f
> File Descriptor1 Opened 6
> Thread created 0
> File Descriptor2 Opened 7
> Process ID: 5351
> Parent Process ID: 5351
> I am the parent
> Core joined 0
> I am the child
> Python read from pipe pr
> b'Hello to Python from C\x00\x00'
The \x00 is because the length you used is wrong.
> Python wrote back
> C received message 1 from Python
> OK message received
> Python read from pipe pr
> b'Message to Python 2\x00\x00'
The \x00 is because the length you used is wrong.
> Python wrote back
> C received message 2 from Python
> OK message received
> C is sending exit message to Python
> C is closing
> Python read from pipe pr
> b'99'
> Python is terminating
>
> Python runs on a separate thread (created with pthreads) because I want the
> flexibility of using this same basic code as a stand-alone .exe, or for a C
> extension from Python called with ctypes. If I use it as a C extension then
> I want the Python code on a separate thread because I can't have two
> instances of the Python interpreter running on one thread, and one instance
> will already be running on the main thread, albeit "suspended" by the call
> from ctypes.
>
> So that's my solution: (1) Python multiprocessing module; (2) Python strings
> written to the pipe must be terminated with \n.
>
> Thanks again to all who commented.
>
>
>
> Dec 6, 2021, 13:33 by [email protected]:
>
>
>> On 6 Dec 2021, at 21:05, Jen Kris <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Here is what I don't understand from what you said. "The child process is
>> created with a single thread—the one that called fork()." To me that
>> implies that the thread that called fork() is the same thread as the child
>> process. I guess you're talking about the distinction between logical
>> threads and physical threads.
>
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment
> of that thread,
> which means that it has all the stack variables and stack frames from the
> parent process's thread.
>
>
>> But the main issue is your suggestion that I should call fork-execv from the
>> thread that runs the main C program, not from a separate physical pthread.
>> That would certainly eliminate the overhead of creating a new pthread.
>
> Forget about physical threads, they only matter when you are performance
> tuning your code.
>
>> I am working now to finish this, and I will try your suggestion of calling
>> fork-execv from the "main" thread. When I reply back next I can give you a
>> complete picture of what I'm doing.
>>
>> Your comments, and those of Peter Holzer and Chris Angelico, are most
>> appreciated.
>
> Barry
>
>
>
--
https://mail.python.org/mailman/listinfo/python-list
