ANN: A new library for encryption and signing has been released.
A new library called pagesign (Python-age-sign) has been released on PyPI [1]. It covers similar functionality to python-gnupg, but uses the modern encryption tool age [2] and the modern signing tool minisign [3]. The initial release allows you to: * Create and manage identities (which are containers for the keys used for encryption/decryption and signing/verification). * Encrypt and decrypt files to multiple recipients. * Sign files and verify signatures. This release has been signed with my code signing key: Vinay Sajip (CODE SIGNING KEY) Fingerprint: CA74 9061 914E AC13 8E66 EADB 9147 B477 339A 9B86 Recent changes to PyPI don't show the GPG signature with the download links. An alternative download source where the signatures are available is at [4]. Documentation is available at [5]. As always, your feedback is most welcome (especially bug reports [6], patches and suggestions for improvement, or any other points). Enjoy! Cheers Vinay Sajip [1] https://pypi.org/project/pagesign/0.1.0/ [2] https://age-encryption.org/ [3] https://jedisct1.github.io/minisign/ [4] https://bitbucket.org/vinay.sajip/pagesign/downloads/ [5] https://docs.red-dove.com/pagesign/ [6] https://github.com/vsajip/pagesign/issues/new/choose -- https://mail.python.org/mailman/listinfo/python-list
Re: Python child process in while True loop blocks parent
Thanks for your comments. I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. I revised the Python program to be a multiprocessing loop using the Python multiprocessing module. That bypasses the GIL and allows Python to run concurrently with C. So far so good. Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C. Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes. I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process. I will reply back when it's finished and post the code so you can see what I have done. Thanks again. Jen Dec 4, 2021, 09:22 by [email protected]: > > >> On 1 Dec 2021, at 16:01, Jen Kris <>> [email protected]>> > wrote: >> >> Thanks for your comment re blocking. >> >> I removed pipes from the Python and C programs to see if it blocks without >> them, and it does. >> >> It looks now like the problem is not pipes. >> > > Ok. > > >> I use fork() and execv() in C to run Python in a child process, but the >> Python process blocks >> > > Use strace on the parent process to see what is happening. > You will need to use the option to follow subprocesses so that you can see > what goes on in the python process. > > See man strace and the --follow-forks and --output-separately options. > That will allow you to find the blocking system call that your code is making. > > >> because fork() does not create a new thread, so the Python global >> interpreter lock (GIL) prevents the C program from running once Python >> starts. >> > > Not sure why you think this. > > >> So the solution appears to be run Python in a separate thread, which I can >> do with pthread create. >> >> See "Thread State and the Global Interpreter Lock" >> >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>> >> and the sections below that "Non-Python created threads" and "Cautions >> about fork()." >> > > I take it you mean that in the parent you think that using pthreads will > affect python after the exec() call? > I does not. After exec() the process has one main thread create by the kernel > and a new address space as defined by the /usr/bin/python. > The only state that in inherited from the parent are open file descriptors, > the current working directory and security state like UID, GID. > > >> I'm working on that today and I hope all goes well :) >> > > You seem to be missing background information on how processes work. > Maybe "Advanced Programming in the UNIX Environment" > would be helpful? > > https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk> > > > It's a great book and covers a wide range of Unix systems programming topics. > > Have you created a small C program that just does the fork and exec of a > python program to test out your assumptions? > If not I recommend that you do. > > Barry > > > >> >> >> >> Nov 30, 2021, 11:42 by >> [email protected]>> : >> >>> >>> >>> On 29 Nov 2021, at 22:31, Jen Kris < [email protected] > wrote: Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination. >>> >>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK. >>> You should not use O_RDWR when you only need O_RDONLY access or only >>> O_WRONLY access. >>> >>> You may find >>> >>> man 2 open >>> >>> useful to understand in detail what is behind os.open(). >>> >>> Barry >>> >>> >>> >>> Nov 29, 2021, 14:12 by [email protected] : > > >> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>> >> [email protected]>> > wrote: >> >> I have a C program that forks to create a child process and uses execv >> to call a Python program. The Python program communicates with the >> parent process (in C) through a FIFO pipe monitored with epoll(). >> >> The Python child process is in a while True loop, which is intended to >> keep it running while the parent process proceeds, and perform functions >> for the C program only at intervals when the parent sends data to the >> child -- similar to a daemon process. >> >> The C process writes to its end of the pipe and the child process reads >>
Re: Python child process in while True loop blocks parent
> On 5 Dec 2021, at 17:54, Jen Kris wrote: > > > Thanks for your comments. > > I put the Python program on its own pthread, and call a small C program to > fork-execv to call the Python program as a child process. What do you mean by putting python in it’s own pthread? Are you embedding python in an other program? Barry > I revised the Python program to be a multiprocessing loop using the Python > multiprocessing module. That bypasses the GIL and allows Python to run > concurrently with C. So far so good. > > Next I will use Linux pipes, not Python multiprocessing pipes, for IPC > between Python and C. Multiprocessing pipes are (as far as I can tell) only > for commo between two Python processes. I will have the parent thread send a > signal through the pipe to the child process to exit when the parent thread > is ready to exit, then call wait() to finalize the child process. > > I will reply back when it's finished and post the code so you can see what I > have done. > > Thanks again. > > Jen > > > Dec 4, 2021, 09:22 by [email protected]: > >>> On 1 Dec 2021, at 16:01, Jen Kris wrote: >>> >>> Thanks for your comment re blocking. >>> >>> I removed pipes from the Python and C programs to see if it blocks without >>> them, and it does. >>> It looks now like the problem is not pipes. >> >> Ok. >> >> I use fork() and execv() in C to run Python in a child process, but the >> Python process blocks > > Use strace on the parent process to see what is happening. > You will need to use the option to follow subprocesses so that you can see > what goes on in the python process. > > See man strace and the --follow-forks and --output-separately options. > That will allow you to find the blocking system call that your code is making. > >> because fork() does not create a new thread, so the Python global >> interpreter lock (GIL) prevents the C program from running once Python >> starts. > > Not sure why you think this. > >> So the solution appears to be run Python in a separate thread, which I can >> do with pthread create. >> See "Thread State and the Global Interpreter Lock" >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock >> and the sections below that "Non-Python created threads" and "Cautions >> about fork()." > > I take it you mean that in the parent you think that using pthreads will > affect python after the exec() call? > I does not. After exec() the process has one main thread create by the kernel > and a new address space as defined by the /usr/bin/python. > The only state that in inherited from the parent are open file descriptors, > the current working directory and security state like UID, GID. > >> I'm working on that today and I hope all goes well :) > > You seem to be missing background information on how processes work. > Maybe "Advanced Programming in the UNIX Environment" would be helpful? > > https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk > > > It's a great book and covers a wide range of Unix systems programming topics. > > Have you created a small C program that just does the fork and exec of a > python program to test out your assumptions? > If not I recommend that you do. > > Barry > > >> >> >> >> Nov 30, 2021, 11:42 by [email protected]: >> >> >>> On 29 Nov 2021, at 22:31, Jen Kris wrote: >>> >>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl >>> set, but no event loop to handle it yet. I'm putting that in now with a >>> pipe write in Python-- as Cameron pointed out that is the likely source of >>> blocking on C. The pipes are opened as rdwr in Python because that's >>> nonblocking by default. The child will become more complex, but not in a >>> way that affects polling. And thanks for the tip about the c-string >>> termination. >>> >> >> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK. >> You should not use O_RDWR when you only need O_RDONLY access or only >> O_WRONLY access. >> >> You may find >> >> man 2 open >> >> useful to understand in detail what is behind os.open(). >> >> Barry >> >> >> >>> >>> >>> Nov 29, 2021, 14:12 by [email protected]: >>> >>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list >>> wrote: >>> >>> I have a C program that forks to create a child process and uses execv to >>> call a Python program. The Python program communicates with the parent >>> process (in C) through a FIFO pipe monitored with epoll(). >>> >>> The Python child process is in a while True loop, which is intended to keep >>> it running while the parent process proceeds, and perform functions for the >>> C program only at intervals when the parent sends data to the child -- >>> similar to a daemon process. >>> >>> The C process writes to its end of the pipe and the child process reads it, >
Re: Python child process in while True loop blocks parent
By embedding, I think you may be referring to embedding Python in a C program with the Python C API. That's not what I'm doing here -- I'm not using the Python C API. The C program creates two threads (using pthreads), one for itself and one for the child process. On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program. That way Python runs on a separate thread. The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads." https://docs.python.org/3/library/multiprocessing.html. This way I can get the Python functionality I want on call from the C program through pipes and shared memory. I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code. I hope that clarifies what I'm doing. Jen Dec 5, 2021, 15:19 by [email protected]: > > > > >> On 5 Dec 2021, at 17:54, Jen Kris wrote: >> >> >> Thanks for your comments. >> >> I put the Python program on its own pthread, and call a small C program to >> fork-execv to call the Python program as a child process. >> > > What do you mean by putting python in it’s own pthread? > Are you embedding python in an other program? > > Barry > > > >> I revised the Python program to be a multiprocessing loop using the Python >> multiprocessing module. That bypasses the GIL and allows Python to run >> concurrently with C. So far so good. >> >> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC >> between Python and C. Multiprocessing pipes are (as far as I can tell) only >> for commo between two Python processes. I will have the parent thread send >> a signal through the pipe to the child process to exit when the parent >> thread is ready to exit, then call wait() to finalize the child process. >> >> I will reply back when it's finished and post the code so you can see what I >> have done. >> >> Thanks again. >> >> Jen >> >> >> Dec 4, 2021, 09:22 by [email protected]: >> >>> >>> On 1 Dec 2021, at 16:01, Jen Kris < [email protected] > wrote: Thanks for your comment re blocking. I removed pipes from the Python and C programs to see if it blocks without them, and it does. It looks now like the problem is not pipes. >>> >>> Ok. >>> >>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks >>> >>> Use strace on the parent process to see what is happening. >>> You will need to use the option to follow subprocesses so that you can see >>> what goes on in the python process. >>> >>> See man strace and the --follow-forks and --output-separately options. >>> That will allow you to find the blocking system call that your code is >>> making. >>> >>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts. >>> >>> Not sure why you think this. >>> >>> So the solution appears to be run Python in a separate thread, which I can do with pthread create. See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()." >>> >>> I take it you mean that in the parent you think that using pthreads will >>> affect python after the exec() call? >>> I does not. After exec() the process has one main thread create by the >>> kernel and a new address space as defined by the /usr/bin/python. >>> The only state that in inherited from the parent are open file descriptors, >>> the current working directory and security state like UID, GID. >>> >>> I'm working on that today and I hope all goes well :) >>> >>> You seem to be missing background information on how processes work. >>> Maybe "Advanced Programming in the UNIX Environment" >>> would be helpful? >>> >>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>>> >>> >>> >>> It's a great book and covers a wide range of Unix systems programming >>> topics. >>> >>> Have you created a small C program that just does the fork and exec of a >>> python program to test out your assumptions? >>> If not I recommend that you do. >>> >>> Barry >>> >>> >>> Nov 30, 2021, 11:42 by [email protected] : > > > >> On 29 Nov 2021, at 22:31, Jen Kris <>> [email protected]>> > >> wrote: >> >> Thanks to you and Cameron for your replies. The C side has an epoll_ctl >> set, but no event loop to handle it yet. I'm putting that in now with a >> pipe write i
Re: Python child process in while True loop blocks parent
On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote: > The C program creates two threads (using pthreads), one for itself and > one for the child process. On creation, the second pthread is pointed > to a C program that calls fork-execv to run the Python program. That > way Python runs on a separate thread. I think you have the relationship between processes and threads backwards. A process consists of one or more threads. Fork creates a new process and therefore also a new thread. hp -- _ | Peter J. Holzer| Story must make more sense than reality. |_|_) || | | | [email protected] |-- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
python list files and folder using tkinter
Hello,
I have already posted a message some time ago for this app. Since then, I
didn't code in python or made any changes. I think before getting further with
functionnalities a few things or the whole thing need to be changed.
For exemple, it would need a button to pick folders and maybe ask if the csv
resulting file can be saved in the same folder.
And more important, files are listing ok in windows but not in linux after
running it a few times.
https://github.com/barpasc/listfiles
code extract without indentation, see source script on github
elif vvchkboxF == 1:
| # *** FOLDERS AND FILES ONLY *** |
|
| for root, dirs, files in os.walk(Lpath, topdown=False): |
|
| ### calcul taille dossier |
|
| size = 0 |
|
| for x, y, z in os.walk(root): |
|
| for i in z: |
|
| ftmp_che = x + os.sep + i |
|
| f_size += os.path.getsize(ftmp_che) |
|
| ### ecriture taille dossier |
|
| counter = root.count(os.path.sep) - counterPath |
|
| vfile_name = root |
|
| vfile_name = vfile_name + os.path.sep |
|
| vfile_name = os.path.split(os.path.dirname(vfile_name))[1] |
|
| vfile_name += os.path.sep |
|
| if counter <= f_vscale: |
|
| csv_contents += "%s;%s;%.0f;%.2f;%.2f;%.2f;%s\n" % (root, vfile_name,
f_size, f_size/1024, f_size/1048576,f_size/1073741824, "folder") |
|
|
|
|
| ### calcul +ecriture taille fichier |
|
| for f in os.listdir(Lpath): |
|
| path = os.path.join(Lpath, f) |
|
| if os.path.isfile(path): |
|
| f_size = 0 |
|
| f_size = os.path.getsize(path) |
|
| csv_contents += "%s;%s;%.0f;%.2f;%.2f;%.2f;%s\n" % (path, f, f_size,
f_size/1024, f_size/1048576,f_size/1073741824, "file") |
|
|
|
|
| fx_writeCSV_str(csv_contents) |
|
print("job adv listing files ok")
--
https://mail.python.org/mailman/listinfo/python-list
threading and multiprocessing deadlock
Hi there,
I'm a bit confused. In my scenario I a mixing threading with
multiprocessing. Threading by itself would be nice, but for GIL reasons
I need both, unfortunately. I've encountered a weird situation in which
multiprocessing Process()es which are started in a new thread don't
actually start and so they deadlock on join.
I've created a minimal example that demonstrates the issue. I'm running
on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
([GCC 10.3.0] on linux).
Here's the code:
import time
import multiprocessing
import threading
def myfnc():
print("myfnc")
def run(result_queue, callback):
result = callback()
result_queue.put(result)
def start(fnc):
def background_thread():
queue = multiprocessing.Queue()
proc = multiprocessing.Process(target = run, args = (queue,
fnc))
proc.start()
print("join?")
proc.join()
print("joined.")
result = queue.get()
threading.Thread(target = background_thread).start()
start(myfnc)
start(myfnc)
start(myfnc)
start(myfnc)
while True:
time.sleep(1)
What you'll see is that "join?" and "joined." nondeterministically does
*not* appear in pairs. For example:
join?
join?
myfnc
myfnc
join?
join?
joined.
joined.
What's worse is that when this happens and I Ctrl-C out of Python, the
started Thread is still running in the background:
$ ps ax | grep minimal
370167 pts/0S 0:00 python3 minimal.py
370175 pts/2S+ 0:00 grep minimal
Can someone figure out what is going on there?
Best,
Johannes
--
https://mail.python.org/mailman/listinfo/python-list
