Re: Python question
On 11/03/2020 04:06, Michael Torrie wrote: On 3/10/20 6:49 PM, Souvik Dutta wrote: What about moving on to a social media app completely made in pythoj for python? No thanks. I don't want to be on yet another web forum. I don't need "social media" or a "social media app." Email works exceedingly well for this sort of thing, despite Google's antics. +10 The best response to "This system breaks when I abuse it" is almost always "Well stop abusing it then." -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
reactive programming use case
hi, i am thinking about using RxPy to solve the following problem, and not yet sure if its the right tool for the job. I am working on a calculation engine, that takes a dataset (or multiple sets) and calculates new columns. so in pandas terms, it starts as DataFrame, and then i run it through a set of calcs. which all add new columns to the frame, and in the end i use those columns to check my dataset for things such as minimum % of colA / ColX etc.. i was thinking about using reactive programming to: * decouple the dataset from running all the calculations * maybe help in making it more of a observable graph * combine datasets but maybe there is a better pattern? -- https://mail.python.org/mailman/listinfo/python-list
link to venv python sees a different sys.path
I'm trying to understand why python 3.8.2 venv behaves differently when it is
executed va a link
Make the env
rptlab@everest:~/code/hg-repos
$ python38 -mvenv __py__/382v
rptlab@everest:~/code/hg-repos
make a link
$ ln -s ../__py__/382v/bin/python bin/python382v
the venv sys.path
rptlab@everest:~/code/hg-repos > $ __py__/382v/bin/python -c"import
sys;print('\n'.join(sys.path))"
/home/rptlab/LOCAL/382/lib/python38.zip
/home/rptlab/LOCAL/382/lib/python3.8
/home/rptlab/LOCAL/382/lib/python3.8/lib-dynload
/home/rptlab/code/hg-repos/__py__/382v/lib/python3.8/site-packages
rptlab@everest:~/code/hg-repos
the linked python sys.path
$ bin/python382v -c"import sys;print('\n'.join(sys.path))"
/home/rptlab/LOCAL/382/lib/python38.zip
/home/rptlab/LOCAL/382/lib/python3.8
/home/rptlab/LOCAL/382/lib/python3.8/lib-dynload
/home/rptlab/LOCAL/382/lib/python3.8/site-packages
rptlab@everest:~/code/hg-repos
$
so the linked version of the venv python sees the base python site packages and
not the expected venv site-packages.
Is there a way to make the link work properly.
This problem doesn't seem to occur with older virtualenv made environments.
--
Robin Becker
--
https://mail.python.org/mailman/listinfo/python-list
Re: link to venv python sees a different sys.path
On 2020-03-11, Robin Becker wrote: > I'm trying to understand why python 3.8.2 venv behaves differently > when it is executed va a link Because site.py contains a function called venv() which looks up the path of the executed python binary, and searches for the virtual environment relative to that path. If you're executing the python binary via a link, it will look relative to the link, not the target of the link. > so the linked version of the venv python sees the base python site > packages and not the expected venv site-packages. > > Is there a way to make the link work properly. No. Unless you count making a shell script in the place you want which does nothing except exec the python binary inside the venv, i.e.: cat >pythonish #!/bin/sh exec some/other/python "$@" ^D -- https://mail.python.org/mailman/listinfo/python-list
Re: Lock acquisition by the same thread - deadlock protection
That code I'm talking about didn't require a reentrant lock - the algorithm really wasn't reentrant. Let me clarify my point: I'm wondering why the non-reentrant lock doesn't raise an exception immediately on this erroneous situation. I thought it could be altered, or at least we could add an option to let a `threading.Lock` behave like a pthread mutex in mode `PTHREAD_MUTEX_ERRORCHECK`: Disallow double locking by same thread, disallow unlocking by another thread. However, after searching a bit more, I found a reference mentioning current behavior in a docstring defined in `_threadmodule.c`: "A lock is not owned by the thread that locked it; another thread may unlock it.". It was added in 75e9fc31d3a18068, a commit from 1998... Since it's a well-documented behavior I guess it's here to stay. At least the "unlock by another thread" part. But I question the double locking. On Tue, Mar 10, 2020 at 5:07 PM Barry Scott wrote: > > > > > On 9 Mar 2020, at 22:53, Yonatan Goldschmidt > > wrote: > > > > I recently debugged a program hang, eventually finding out it's a deadlock > > of a single thread, > > resulting from my usage of 2 libraries. One of them - call it library A - > > is reentrant & runs code in > > GC finalizers, while the other - library B - is not reentrant at all. > > Library B held one of its `threading.Lock` locks, and during this period, > > GC was invoked, running > > finalizers of library A which call back into library B, now attempting to > > take the lock again, > > locking the thread forever. > > > > Considering how relatively common this scenario might be (Python, by > > design, can preempt any user code > > to run some other user code, due to GC finalizers), I was surprised Python > > code is not protected > > from this simple type of deadlock. It makes sense that while > > `threading.RLock` allows for recursive > > locking, `threading.Lock` will prevent it - raising an exception if you > > attempt it. > > > > I might be missing something, but why isn't it the status? Why taking a > > `threading.Lock` twice from > > the same thread just hangs, instead of raising a friendly exception? > > I ran a quick search in bpo but found nothing about this topic. I also > > tried to > > search this mailing list but couldn't find how to, so I grepped a few > > random archives > > but found nothing about it. > > > > Would be happy if anyone could shed some light on it... > > threading.Lock is not reentrant and its implementation does not allow > detection of the problem > from what I recall. > > In this case the code might want to use the threading.RLock that is reentrant. > Of course there may be other issues in the code that prevent the finalizers > working if it holds the lock. > > Barry > > > -- https://mail.python.org/mailman/listinfo/python-list
ANN: PyDDF Python Spring Sprint 2020
[This announcement is in German since it targets a local user group meeting in Düsseldorf, Germany] ANKÜNDIGUNG PyDDF Python Spring Sprint 2020 in Düsseldorf Samstag, 28.03.2020, 10:00-18:00 Uhr Sonntag, 29.03.2020, 10:00-18:00 Uhr trivago N.V., Kesselstrasse 5-7, 40221 Düsseldorf Python Meeting Düsseldorf https://www.pyddf.de/sprint2020/ INFORMATION Das Python Meeting Düsseldorf (PyDDF) veranstaltet mit freundlicher Unterstützung der *trivago N.V.* ein Python Sprint Wochenende. Der Sprint findet am Wochenende 28./29.3.2020 in der trivago Niederlassung im Medienhafen Düsseldorf statt (Achtung: Nicht mehr am Karl-Arnold-Platz). * Google Maps: https://goo.gl/maps/dGM6ThfkLiJ2 Folgende Themengebiete haben wir als Anregung angedacht: * Matrix Protokoll - https://matrix.org * Aufzugssimulator * Wegeheld Natürlich kann jeder Teilnehmer weitere Themen vorschlagen. Alles weitere und die Anmeldung findet Ihr auf der Sprint Seite: https://www.pyddf.de/sprint2020/ WICHTIG: Ohne Anmeldung können wir kein Badge für den Gebäudezugang bereitstellen lassen. Eine spontane Anmeldung am Sprint Tag wird daher vermutlich nicht funktionieren. Also bitte unbedingt mit vollen Namen bis spätestens Mittwoch, 25.03., anmelden. Teilnehmer sollten sich zudem auf der PyDDF Liste anmelden, da wir uns dort koordinieren: https://www.egenix.com/mailman/listinfo/pyddf ÜBER UNS Das Python Meeting Düsseldorf (PyDDF) ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet: * https://pyddf.de/ Einen guten Überblick über die Vorträge bietet unser YouTube-Kanal, auf dem wir die Vorträge nach den Meetings veröffentlichen: * http://www.youtube.com/pyddf/ Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld,- in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf: * http://www.egenix.com/ * http://www.clark-consulting.eu/ Mit freundlichen Grüßen, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 11 2020) >>> Python Projects, Coaching and Support ...https://www.egenix.com/ >>> Python Product Development ...https://consulting.egenix.com/ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Python question
On 2020-03-11, Michael Torrie wrote: > On 3/10/20 6:49 PM, Souvik Dutta wrote: >> What about moving on to a social media app completely made in pythoj for >> python? > > No thanks. I don't want to be on yet another web forum. I don't need > "social media" or a "social media app." Email works exceedingly well > for this sort of thing, despite Google's antics. For me, "news" works even better than email, which is why I read all my email lists via the gmane NNTP server using a news client (slrn). :) -- Grant Edwards grant.b.edwardsYow! Now I'm concentrating at on a specific tank battle gmail.comtoward the end of World War II! -- https://mail.python.org/mailman/listinfo/python-list
Re: link to venv python sees a different sys.path
Robin Becker wrote at 2020-3-11 15:26 +:
>I'm trying to understand why python 3.8.2 venv behaves differently when it is
>executed va a link
>
>Make the env
>> rptlab@everest:~/code/hg-repos
>> $ python38 -mvenv __py__/382v
>> rptlab@everest:~/code/hg-repos
>
>make a link
>> $ ln -s ../__py__/382v/bin/python bin/python382v
>
>the venv sys.path
>> rptlab@everest:~/code/hg-repos > $ __py__/382v/bin/python -c"import
>> sys;print('\n'.join(sys.path))"
>>
>> /home/rptlab/LOCAL/382/lib/python38.zip
>> /home/rptlab/LOCAL/382/lib/python3.8
>> /home/rptlab/LOCAL/382/lib/python3.8/lib-dynload
>> /home/rptlab/code/hg-repos/__py__/382v/lib/python3.8/site-packages
>> rptlab@everest:~/code/hg-repos
>
>the linked python sys.path
>> $ bin/python382v -c"import sys;print('\n'.join(sys.path))"
>>
>> /home/rptlab/LOCAL/382/lib/python38.zip
>> /home/rptlab/LOCAL/382/lib/python3.8
>> /home/rptlab/LOCAL/382/lib/python3.8/lib-dynload
>> /home/rptlab/LOCAL/382/lib/python3.8/site-packages
>> rptlab@everest:~/code/hg-repos
>> $
>
>so the linked version of the venv python sees the base python site packages
>and not the expected venv site-packages.
I guess (!) that it is using the path to the interpreter program
in order to locate the venv's "site-packages" and falls back to
the system's when this is not possible.
If my guess is correct, then some links will not work.
But, you should be able to use links which find
"site-packages" via "/../lib/python/site-packages".
--
https://mail.python.org/mailman/listinfo/python-list
Re: Issue with PYSFTP - Working in WINSCP
NAND KISHORE wrote at 2020-3-10 20:20 -0500: > We have requirement where we need to get file from client path and then >upload the same to vendor directory path. I am not able to upload the file >to vendor directory path , however when I tried to use the WINSCP it worked >fine. So I thought of checking with Gurus what is wrong I am doing in my >script. Appreciate your input. >I will attach my script. Here is what I am doing. > ... >Also below is the error which I am getting > >- >ERROR:root:Error in getting the file from EBS Outbound Server >Traceback (most recent call last): > ... > File >"/d01/python3/lib64/python3.6/site-packages/paramiko/sftp_client.py", line >894, in _convert_status >raise IOError(errno.ENOENT, text) >FileNotFoundError: [Errno 2] /custom/OWO/ECE_OWO_20200303_143895.dat This error message tells you that "/customOWO/ECE_OWO_20200303_143895.dat" does not exist. Are you sure that this is the correct filename? Especially, the leading "/" looks wrong. -- https://mail.python.org/mailman/listinfo/python-list
Re: Lock acquisition by the same thread - deadlock protection
> On 11 Mar 2020, at 14:24, Yonatan wrote: > > That code I'm talking about didn't require a reentrant lock - the > algorithm really wasn't reentrant. > > Let me clarify my point: I'm wondering why the non-reentrant lock > doesn't raise an exception immediately on this > erroneous situation. > I thought it could be altered, or at least we could add an option to > let a `threading.Lock` behave like a pthread > mutex in mode `PTHREAD_MUTEX_ERRORCHECK`: Disallow double locking by > same thread, disallow unlocking > by another thread. > However, after searching a bit more, I found a reference mentioning > current behavior in a docstring defined > in `_threadmodule.c`: "A lock is not owned by the thread that locked > it; another thread may unlock it.". > It was added in 75e9fc31d3a18068, a commit from 1998... > > Since it's a well-documented behavior I guess it's here to stay. At > least the "unlock by another thread" part. > But I question the double locking. Reading the CPYTHON code I find that the type of mutex is NORMAL and man pthread_mutex_lock documents that it will deadlock if the same thread calls lock a second time. This is what you are seeing it seems. The code I found was in ./Python/thread_pthread.h Sounds to me like the library has a bug in its locking. Getting a exception would help track down the bug faster. But still sounds like a bug. It used to be the case that using the NORMAL locks was higher performance then the ERRORCHECK or RECURSIVE locks. No idea if this is still true of that it matters for cpython. Maybe changing from NORMAL to ERRORCHECK would be a benefit. Barry > > > On Tue, Mar 10, 2020 at 5:07 PM Barry Scott wrote: >> >> >> >>> On 9 Mar 2020, at 22:53, Yonatan Goldschmidt >>> wrote: >>> >>> I recently debugged a program hang, eventually finding out it's a deadlock >>> of a single thread, >>> resulting from my usage of 2 libraries. One of them - call it library A - >>> is reentrant & runs code in >>> GC finalizers, while the other - library B - is not reentrant at all. >>> Library B held one of its `threading.Lock` locks, and during this period, >>> GC was invoked, running >>> finalizers of library A which call back into library B, now attempting to >>> take the lock again, >>> locking the thread forever. >>> >>> Considering how relatively common this scenario might be (Python, by >>> design, can preempt any user code >>> to run some other user code, due to GC finalizers), I was surprised Python >>> code is not protected >>> from this simple type of deadlock. It makes sense that while >>> `threading.RLock` allows for recursive >>> locking, `threading.Lock` will prevent it - raising an exception if you >>> attempt it. >>> >>> I might be missing something, but why isn't it the status? Why taking a >>> `threading.Lock` twice from >>> the same thread just hangs, instead of raising a friendly exception? >>> I ran a quick search in bpo but found nothing about this topic. I also >>> tried to >>> search this mailing list but couldn't find how to, so I grepped a few >>> random archives >>> but found nothing about it. >>> >>> Would be happy if anyone could shed some light on it... >> >> threading.Lock is not reentrant and its implementation does not allow >> detection of the problem >> from what I recall. >> >> In this case the code might want to use the threading.RLock that is >> reentrant. >> Of course there may be other issues in the code that prevent the finalizers >> working if it holds the lock. >> >> Barry >> >> >> > -- https://mail.python.org/mailman/listinfo/python-list
Is there a Python profiler that preserves function arguments?
I'm profiling a Python function `foo()` that takes a single argument,
but that argument makes a huge difference in what the function
actually does.
Currently I'm using `cProfile`, which records every call to `foo()` as
if it was the same, preventing me from figuring out what's going on.
Is there a way to get `cProfile`, or any other Python profiler, to
preserve function call arguments, so when I look at the call stack
later (especially using a visualizer like SnakeViz) I can distinguish
between `foo('bar')` and `foo('qux')`?
P.S. arguably this is a code design issue: since `foo('bar')` and
`foo('qux')` do very different things, they should be distinctly-named
separate functions, like `foo_bar()` and `foo_qux()`. However, the
question is whether I can profile them as-is, without refactoring.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python question
On 12/03/20 3:03 AM, Rhodri James wrote: On 11/03/2020 04:06, Michael Torrie wrote: On 3/10/20 6:49 PM, Souvik Dutta wrote: What about moving on to a social media app completely made in pythoj for python? No thanks. I don't want to be on yet another web forum. I don't need "social media" or a "social media app." Email works exceedingly well for this sort of thing, despite Google's antics. +10 The best response to "This system breaks when I abuse it" is almost always "Well stop abusing it then." +1 Didn't someone once claim "do no harm"? There are two sides to every story! Rather than changing the (Discussion List) server, which affects everyone; ask those who don't like Google's tactics/behavior to change their (email) client! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
