What kind of magic do I need to get python to talk to Excel xlsm file?

[Steve]

Glutton for punishment, I am looking into designing another .py program.  I
would like to pull two columns of information from Excel but the more I look
into coding on the 'net, the more confusing it looks.  I don't understand
what I need to import or install to get the link.

Steve




Footnote:
Some mornings it just isn't worth chewing through the leather straps.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading plus multiprocessing plus cv2 error

[John O'Hagan]

On Sun, 30 Aug 2020 09:59:15 +0100
Barry Scott  wrote:

> 
> 
> > On 29 Aug 2020, at 18:01, Dennis Lee Bieber 
> > wrote:
> > 
> > On Sat, 29 Aug 2020 18:24:10 +1000, John O'Hagan
> >  declaimed the following:
> > 
> >> There's no error without the sleep(1), nor if the Process is
> >> started before the Thread, nor if two Processes are used instead,
> >> nor if two Threads are used instead. IOW the error only occurs if
> >> a Thread is started first, and a Process is started a little later.
> >> 
> >> Any ideas what might be causing the error?
> >> 
> > 
> > Under Linux, multiprocessing creates processes using
> > fork(). That means that, for some fraction of time, you have TWO
> > processes sharing the same thread and all that entails (if it
> > doesn't overlay the forked process with a new executable, they are
> > sharing the thread until the thread exits).
> 
> In the parent you have 1 or more threads.
> 
> After fork the new process has 1 thread, which is not shared with the
> parent. Any extra threads are not in the new process. But the memory
> in the new process will have data structures from the parents other
> threads.
> 
> So no you never have two processes sharing an threads.
> 
> This leads to problems with locks.
> 
> Barry
> 
> 
> > 
> > https://stackoverflow.com/questions/54466572/how-to-properly-multithread-in-opencv-in-2019
> > (which points to)
> > https://answers.opencv.org/question/32415/thread-safe/?answer=32452#post-id-32452
> > """
> > The library itself is thread safe in that you can have multiple
> > calls into the library at the same time, however the data is not
> > always thread safe. """
> > 
> > The sleep(1), when compounded with the overhead of starting
> > the thread, and then starting the process, likely means the thread
> > had exited before the process actually is started. Try replacing
> > the sleep(2) in the work code with something like sleep(30) -- I
> > hypothesize that you'll get the same error condition even with the
> > sleep(1) in place.
> > 
> > 
> > -- 
> > Wulfraed Dennis Lee Bieber AF6VN
> > wlfr...@ix.netcom.com
> > http://wlfraed.microdiversity.freeddns.org/
> > 
> > -- 
> > https://mail.python.org/mailman/listinfo/python-list
> > 
> 

Thanks to all who replied. I'm still none the wiser about the error, but
I learned something about multiprocessing!

John
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What kind of magic do I need to get python to talk to Excel xlsm file?

[Peter Otten]

Steve wrote:

> Glutton for punishment, I am looking into designing another .py program. 
> I would like to pull two columns of information from Excel but the more I
> look
> into coding on the 'net, the more confusing it looks.  I don't understand
> what I need to import or install to get the link.

openpyxl should be fine to to extract data from an Excel table:

https://openpyxl.readthedocs.io/en/stable/usage.html

Of course this cannot magically turn Python into Basic, and there are 
probably other limitations.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Richard Damon]

On 8/31/20 6:05 PM, Chris Angelico wrote:
> On Tue, Sep 1, 2020 at 5:08 AM Richard Damon  wrote:
>> The file descriptor could remember the path used to get to it. chroot
>> shows that .. needs to be somewhat special, as it needs to go away for
>> anyone that . is their current root.
> But my point is that there might not actually *be* a valid path that
> gets you to a file descriptor. It can't remember something that
> doesn't exist. (And it's pretty impractical to do that even if it
> does.)

Remember, we are talking about a hypothetical OS that handles hardlinks
to directories, and defines that .. will point to the parent used to
come to it, NOT just having current *NIX allowing hardlinks to
directories with no remediation of the issues cause.

One result of the definition, is that when you open a file/directory, if
it might be a directory, the system WILL need to remember the path to it
(so as to provide a value for ..) and that memory will provide a
'reference' for the directories so they can't go away (just like an
unlinked file stays around will someone has it open). The normal way to
get a file descriptor starts from a path, so the path had to exist in
the first place, and since we are assuming that to get .., it keep a
reference to that path, it can't truly go away.

>
>> I see no problem with it being a hardlink, and in fact, executables know
>> the name they were executed by, so directories  knowing the path isn't
>> that different.
> Actually no, they don't. They are all taught, and being taught,
> believe, that their first argument is their name.
>
>> The key differnce between a hardlink and a symlink is
>> that hardlinks maintain existance, and always point to something that
>> exists (things know how many hardlinks refer to them). symlinks don't
>> reference the actual file object, but the symbolic path to it, which may
>> or may not actually exist, and who doesn't know such a link exists.
> Symlinks refer to a path, which may be relative. Hardlinks refer to an
> inode (or whatever other way you choose to identify an actual file's
> contents). It's entirely possible to have an open file or directory
> that no longer has any actual path referring to it; in fact, things
> don't know how many hardlinks refer to them, just how many references
> there are.
>
> ChrisA


-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Chris Angelico]

On Tue, Sep 1, 2020 at 10:57 PM Richard Damon  wrote:
>
> On 8/31/20 6:05 PM, Chris Angelico wrote:
> > On Tue, Sep 1, 2020 at 5:08 AM Richard Damon  
> > wrote:
> >> The file descriptor could remember the path used to get to it. chroot
> >> shows that .. needs to be somewhat special, as it needs to go away for
> >> anyone that . is their current root.
> > But my point is that there might not actually *be* a valid path that
> > gets you to a file descriptor. It can't remember something that
> > doesn't exist. (And it's pretty impractical to do that even if it
> > does.)
>
> Remember, we are talking about a hypothetical OS that handles hardlinks
> to directories, and defines that .. will point to the parent used to
> come to it, NOT just having current *NIX allowing hardlinks to
> directories with no remediation of the issues cause.

Yes. I'm assuming that you can define things any way you like.

> One result of the definition, is that when you open a file/directory, if
> it might be a directory, the system WILL need to remember the path to it
> (so as to provide a value for ..) and that memory will provide a
> 'reference' for the directories so they can't go away (just like an
> unlinked file stays around will someone has it open). The normal way to
> get a file descriptor starts from a path, so the path had to exist in
> the first place, and since we are assuming that to get .., it keep a
> reference to that path, it can't truly go away.
>

But as I've pointed out, you don't always *have* a valid path. File
descriptors can be passed from process to process (most commonly by
being retained when you fork and not closed when you exec, but there
are other ways too, eg sockets), so even though starting with a path
is the most common, it isn't the only way, and you could easily have
an FD with no associated path, and then read it to find out what ".."
is.

Also, even if all that could be solved, I don't like the idea that
reading the same directory from two different sources leads to
different results. Is it really the same directory if reading it in
different ways gives different results?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What kind of magic do I need to get python to talk to Excel xlsm file?

[Akkana Peck]

Steve writes:
> Glutton for punishment, I am looking into designing another .py program.  I
> would like to pull two columns of information from Excel but the more I look

I've found the xlrd module easy to use for extracting data from
Excel files. But I've only used it on .xsl and .xsls. I've never
encountered a .xlsm, but a web search tells me that extension means
it has macros, and if you want macros to be executed, that may be
quite a bit more difficult.

One person says xlrd works for xlsm:
https://stackoverflow.com/questions/23554808/how-to-extract-sheet-from-xlsm-and-save-it-as-csv-in-python
and the discussion also includes an example using pandas. So now
you have three possible options (including the openpyxl someone
else suggested).

...Akkana
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Eryk Sun]

On 9/1/20, Chris Angelico  wrote:
>
> Also, even if all that could be solved, I don't like the idea that
> reading the same directory from two different sources leads to
> different results. Is it really the same directory if reading it in
> different ways gives different results?

What's your take on the following example in Linux?

"test2/spam" is a bind mount for "test1/spam", and note that `mount
--bind` in Linux is a namespace operation, i.e. it's not a new device:

>>> os.lstat('test1/spam').st_dev == os.lstat('test2/spam').st_dev
True
>>> os.lstat('test1/spam').st_ino == os.lstat('test2/spam').st_ino
True

According to POSIX (st_dev, st_ino), it's the same directory, yet the
".." entry evaluates depending on the path parsing context:

>>> os.lstat('test1/spam/..').st_ino == os.lstat('test1').st_ino
True
>>> os.lstat('test2/spam/..').st_ino == os.lstat('test2').st_ino
True
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Grant Edwards]

On 2020-09-01, Richard Damon  wrote:

> Remember, we are talking about a hypothetical OS that handles hardlinks
> to directories, and defines that .. will point to the parent used to
> come to it, NOT just having current *NIX allowing hardlinks to
> directories with no remediation of the issues cause.

I can testify from personal experience that SunOS 3/4 was in the
latter category.  After creating a hard-link to a directory, things
like fsck and the filesystem dump backup utility got very upset and
confused.  IIRC the only way to recover was to nuke the involved
inodes then let fsck try to pick up the pieces and put them in the
lost+found. IIRC, I managed to recover without losing any files, but
it wasn't a fun day.

--
Grant


-- 
https://mail.python.org/mailman/listinfo/python-list

How to set custom locals for function call?

[Andras Tantos]

All,

I'm new here, so please direct me to the right forum, if this is not the 
one...


What I'm trying to do is to call a function, but monitor all the local 
variable accesses within that function. What I thought I would need to 
do, is to |exec| the function with a custom |locals| dictionary. The 
code below attempts to do it.


|classMySymbolTable(dict):defset_type(self,t):self.type =t 
def__getitem__(self,key):print(f"Requesting key {key} from {self.type} 
table")returnsuper().__getitem__(key)def__setitem__(self,key,value):print(f"Setting 
key {key} from {self.type} table to value 
{value}")returnsuper().__setitem__(key,value)defmylocals(func):defwrapper(*args,**kwargs):loc 
=MySymbolTable()glob 
=MySymbolTable(globals())loc.set_type("local")glob.set_type("global")exec(func.__code__,glob,loc)returnwrapper 
@mylocalsdeffun1():print(f"fun1 with locals: {type(locals())} and 
globals: {type(globals())}")a =1b =2c =3a =b c =5fun1()|


However, when I run it, I get the following output:

|Requestingkey printfromglobaltable Requestingkey type fromglobaltable 
Requestingkey locals fromglobaltable Requestingkey type fromglobaltable 
Requestingkey globals fromglobaltable fun1 
withlocals:andglobals:|


That is to say, global accesses are redirected to my custom dict, but 
local assignments are not. You can even see that in the types of the two 
objects printed in the last line.


My hunch is that since I'm using the functions |__code__| member, I end 
up executing pre-compiled byte-code which has already assumptions about 
the locals dict built into it.


If I'm right (or even if I'm not), what is the right way to achieve my 
goal: that local assignments get redirected to the supplied dictionary?


Thanks,
Andras Tantos


--
https://mail.python.org/mailman/listinfo/python-list

Re: Repair was successful

[Lourival Beltrão Martins Jr]

Lourival Beltrão

Medical Physicist
Supervisor Radiation Protection
Nuclear Engineer

***
Go green, make sense


Please consider the environment before printing this e-mail.

The contents of this e-mail message (including any attachments) are 
confidential and are intended to be conveyed for the use of the recipient to 
whom it is addressed only. If you receive this transmission in error, please 
notify the sender of this immediately and delete the message from your system. 
Any distribution, reproduction, or use of this message by someone other than 
the recipient is not authorized and may be unlawful.


From: Lourival Beltrão Martins Jr
Sent: Tuesday, September 1, 2020 10:07
To: python-list@python.org 
Subject: Repair was successful




Hello,



Good Morning!



I have python 3.7 installed on my laptop and I did the download the 3.8.5 
version. The message "Repair installation" appears all times when I open Spyder 
4.0.



Could you help me to fix this issue?



Thank you very much!



Beltrão





Enviado do Correio para Windows 
10


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[MRAB]

On 2020-09-01 05:45, Andras Tantos wrote:

All,

I'm new here, so please direct me to the right forum, if this is not the
one...

What I'm trying to do is to call a function, but monitor all the local
variable accesses within that function. What I thought I would need to
do, is to |exec| the function with a custom |locals| dictionary. The
code below attempts to do it.

|classMySymbolTable(dict):defset_type(self,t):self.type =t
def__getitem__(self,key):print(f"Requesting key {key} from {self.type}
table")returnsuper().__getitem__(key)def__setitem__(self,key,value):print(f"Setting
key {key} from {self.type} table to value
{value}")returnsuper().__setitem__(key,value)defmylocals(func):defwrapper(*args,**kwargs):loc
=MySymbolTable()glob
=MySymbolTable(globals())loc.set_type("local")glob.set_type("global")exec(func.__code__,glob,loc)returnwrapper
@mylocalsdeffun1():print(f"fun1 with locals: {type(locals())} and
globals: {type(globals())}")a =1b =2c =3a =b c =5fun1()|

However, when I run it, I get the following output:

|Requestingkey printfromglobaltable Requestingkey type fromglobaltable
Requestingkey locals fromglobaltable Requestingkey type fromglobaltable
Requestingkey globals fromglobaltable fun1
withlocals:andglobals:|

That is to say, global accesses are redirected to my custom dict, but
local assignments are not. You can even see that in the types of the two
objects printed in the last line.

My hunch is that since I'm using the functions |__code__| member, I end
up executing pre-compiled byte-code which has already assumptions about
the locals dict built into it.

If I'm right (or even if I'm not), what is the right way to achieve my
goal: that local assignments get redirected to the supplied dictionary?

CPython is able to identify all of the local names of a function and, 
basically, for reasons of efficiency, it uses slots for the local names 
instead of an actual dict. 'locals()' just returns a dict that 
represents those local names and their current values, but modifying 
that dict has no effect on the actual local names. In short, there isn't 
really a local dict that you can replace.

--
https://mail.python.org/mailman/listinfo/python-list

RE: What kind of magic do I need to get python to talk to Excel xlsm file?

[Steve]

Ok, I think I ran into this about a year ago when this was suggested back then.

I entered “pip install pandas”

I am told to “install Anaconda which can be deleted later”   (huh?)
Then it tells me “Installing with “Miniconda” (huh? Huh?)
Then “Installing from PyPl which is what I used to get the same page that is 
telling me all this…

ActivePython, then finally Linux which I am not using.

 

Ok, then I see:

Installing from source

See the contributing guide for complete instructions on building from the git 
source tree. Further, see creating a development environment if you wish to 
create a pandas development environment.

 

And the music goes round and round…..

 

 

 

 

FootNote:
If money does not grow on trees, then why do banks have branches?

 

From: Jeffrey Powell  
Sent: Tuesday, September 1, 2020 5:33 AM
To: Steve 
Subject: Re: What kind of magic do I need to get python to talk to Excel xlsm 
file?

 

Hi Steve,

 

Yes, you need to install pandas first.  I use pip install pandas


Kind regards,

Jeffrey Powell 

 

On Tue, 1 Sep 2020, 11:15 Steve, mailto:Gronicus@sga.ninja> > wrote:

ModuleNotFoundError: No module named 'pandas'

 

I take it that this is an installation.
Is there anything I need to know about this first?

 

Steve

 

 

FootNote:
If money does not grow on trees, then why do banks have branches?

 

From: Jeffrey Powell mailto:jparkerpow...@gmail.com> 
> 
Sent: Tuesday, September 1, 2020 4:30 AM
To: Steve mailto:Gronicus@sga.ninja> >
Subject: Re: What kind of magic do I need to get python to talk to Excel xlsm 
file?

 

Hi Steve,

 

Maybe something like this:

 

import pandas as pd

dt = pd.read_excel("myData.xls", sheet_name="mySheetName", skiprows=0, 
usecols=list(range(0,11)))

 

where usecols are the column numbers you want.

 

Kind regards,

Jeff

 

 

 

On Tue, Sep 1, 2020 at 10:24 AM Steve mailto:Gronicus@sga.ninja> > wrote:

Glutton for punishment, I am looking into designing another .py program.  I
would like to pull two columns of information from Excel but the more I look
into coding on the 'net, the more confusing it looks.  I don't understand
what I need to import or install to get the link.

Steve




Footnote:
Some mornings it just isn't worth chewing through the leather straps.

-- 
https://mail.python.org/mailman/listinfo/python-list




 

-- 

Jeffrey Powell, PhD.  Senior Data Scientist, Dataworkz.nl

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Chris Angelico]

On Wed, Sep 2, 2020 at 4:55 AM Eryk Sun  wrote:
>
> On 9/1/20, Chris Angelico  wrote:
> >
> > Also, even if all that could be solved, I don't like the idea that
> > reading the same directory from two different sources leads to
> > different results. Is it really the same directory if reading it in
> > different ways gives different results?
>
> What's your take on the following example in Linux?
>
> "test2/spam" is a bind mount for "test1/spam", and note that `mount
> --bind` in Linux is a namespace operation, i.e. it's not a new device:
>
> >>> os.lstat('test1/spam').st_dev == os.lstat('test2/spam').st_dev
> True
> >>> os.lstat('test1/spam').st_ino == os.lstat('test2/spam').st_ino
> True
>
> According to POSIX (st_dev, st_ino), it's the same directory, yet the
> ".." entry evaluates depending on the path parsing context:
>
> >>> os.lstat('test1/spam/..').st_ino == os.lstat('test1').st_ino
> True
> >>> os.lstat('test2/spam/..').st_ino == os.lstat('test2').st_ino
> True

When you add ".." to the end of a path, it is very frequently handled
textually. Can you do this exercise by actually opening the
directories and then using openat or statat?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Chris Angelico]

On Wed, Sep 2, 2020 at 5:00 AM Andras Tantos
 wrote:
>
> All,
>
> I'm new here, so please direct me to the right forum, if this is not the
> one...
>
> What I'm trying to do is to call a function, but monitor all the local
> variable accesses within that function. What I thought I would need to
> do, is to |exec| the function with a custom |locals| dictionary. The
> code below attempts to do it.
>
> |classMySymbolTable(dict):defset_type(self,t):self.type =t
> def__getitem__(self,key):print(f"Requesting key {key} from {self.type}
> table")returnsuper().__getitem__(key)def__setitem__(self,key,value):print(f"Setting
> key {key} from {self.type} table to value
> {value}")returnsuper().__setitem__(key,value)defmylocals(func):defwrapper(*args,**kwargs):loc
> =MySymbolTable()glob
> =MySymbolTable(globals())loc.set_type("local")glob.set_type("global")exec(func.__code__,glob,loc)returnwrapper
> @mylocalsdeffun1():print(f"fun1 with locals: {type(locals())} and
> globals: {type(globals())}")a =1b =2c =3a =b c =5fun1()|

Your code's messed up in the post, so I'm guessing on how this is
actually meant to be written. Most of your whitespace has been
destroyed.

> That is to say, global accesses are redirected to my custom dict, but
> local assignments are not. You can even see that in the types of the two
> objects printed in the last line.
>
> My hunch is that since I'm using the functions |__code__| member, I end
> up executing pre-compiled byte-code which has already assumptions about
> the locals dict built into it.

Yes; and furthermore, CPython generally doesn't even use a dictionary
for a function's locals. Instead, it uses cells in a special array;
you can poke around with this by disassembling the code (see the
built-in "dis" module for details) and looking at the
LOAD_FAST/STORE_FAST opcodes.

You might be able to mess with this by using a closure, but local
variable references are generally considered to be an implementation
detail of the function, and it's not going to be easy to mess with
them from the outside reliably. What's the goal here?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What kind of magic do I need to get python to talk to Excel xlsm file?

[Chris Angelico]

On Wed, Sep 2, 2020 at 6:17 AM Steve  wrote:
>
> Ok, I think I ran into this about a year ago when this was suggested back 
> then.
>
> I entered “pip install pandas”
>
> I am told to “install Anaconda which can be deleted later”   (huh?)
> Then it tells me “Installing with “Miniconda” (huh? Huh?)
> Then “Installing from PyPl which is what I used to get the same page that is 
> telling me all this…
>
> ActivePython, then finally Linux which I am not using.
>

You're on Windows? Get your pandas here:

https://www.lfd.uci.edu/~gohlke/pythonlibs/

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: What kind of magic do I need to get python to talk to Excel xlsm file?

[Steve]

Wow, that is quite a list
Here is what I narrowed it down to. What does that paragraph starting with 
"Requires" mean?

===
Pandas: a cross-section and time series data analysis toolkit.

Requires numpy, dateutil, pytz, setuptools, and optionally numexpr, bottleneck, 
scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, brotlipy, 
backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.

pandas‑1.1.1‑cp39‑cp39‑win_amd64.whl


Footnote: 
Mars is the only known planet in our solar system solely inhabited by 
functioning robots.

-Original Message-
From: Python-list  On Behalf 
Of Chris Angelico
Sent: Tuesday, September 1, 2020 4:44 PM
To: Python 
Subject: Re: What kind of magic do I need to get python to talk to Excel xlsm 
file?

On Wed, Sep 2, 2020 at 6:17 AM Steve  wrote:
>
> Ok, I think I ran into this about a year ago when this was suggested back 
> then.
>
> I entered “pip install pandas”
>
> I am told to “install Anaconda which can be deleted later”   (huh?)
> Then it tells me “Installing with “Miniconda” (huh? Huh?) Then 
> “Installing from PyPl which is what I used to get the same page that 
> is telling me all this…
>
> ActivePython, then finally Linux which I am not using.
>

You're on Windows? Get your pandas here:

https://www.lfd.uci.edu/~gohlke/pythonlibs/

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Eryk Sun]

On 9/1/20, Chris Angelico  wrote:
> On Wed, Sep 2, 2020 at 4:55 AM Eryk Sun  wrote:
>
>> "test2/spam" is a bind mount for "test1/spam", and note that `mount
>> --bind` in Linux is a namespace operation, i.e. it's not a new device:
>>
>> >>> os.lstat('test1/spam').st_dev == os.lstat('test2/spam').st_dev
>> True
>> >>> os.lstat('test1/spam').st_ino == os.lstat('test2/spam').st_ino
>> True
>>
>> According to POSIX (st_dev, st_ino), it's the same directory, yet the
>> ".." entry evaluates depending on the path parsing context:
>>
>> >>> os.lstat('test1/spam/..').st_ino == os.lstat('test1').st_ino
>> True
>> >>> os.lstat('test2/spam/..').st_ino == os.lstat('test2').st_ino
>> True
>
> When you add ".." to the end of a path, it is very frequently handled
> textually. Can you do this exercise by actually opening the
> directories and then using openat or statat?

The Python example doesn't process the input path to normalize or
resolve it, so I assume you mean text-based processing in the kernel
for the lstat system call. I can switch to using dir_fd, which is
implemented via fstatat in Linux.

Showing again that the bind mountpoint is the same directory:

>>> fd1 = os.open('test1/spam', 0)
>>> fd2 = os.open('test2/spam', 0)
>>> fs1, fs2 = os.fstat(fd1), os.fstat(fd2)
>>> (fs1.st_dev, fs1.st_ino) == (fs2.st_dev, fs2.st_ino)
True

In agreement with the previous example, the ".." entry evaluates
depending on the original path parsing context of the opened fd:

>>> os.lstat('..', dir_fd=fd1).st_ino == os.lstat('test1').st_ino
True
>>> os.lstat('..', dir_fd=fd2).st_ino == os.lstat('test2').st_ino
True
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What kind of magic do I need to get python to talk to Excel xlsm file?

[Chris Angelico]

On Wed, Sep 2, 2020 at 6:59 AM Steve  wrote:
>
> Wow, that is quite a list
> Here is what I narrowed it down to. What does that paragraph starting with 
> "Requires" mean?
>
> ===
> Pandas: a cross-section and time series data analysis toolkit.
>
> Requires numpy, dateutil, pytz, setuptools, and optionally numexpr, 
> bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, 
> brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.
>

It means you need all those things installed before you can install Pandas.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Richard Damon]

On 9/1/20 9:03 AM, Chris Angelico wrote:
> On Tue, Sep 1, 2020 at 10:57 PM Richard Damon  
> wrote:
>> On 8/31/20 6:05 PM, Chris Angelico wrote:
>>> On Tue, Sep 1, 2020 at 5:08 AM Richard Damon  
>>> wrote:
 The file descriptor could remember the path used to get to it. chroot
 shows that .. needs to be somewhat special, as it needs to go away for
 anyone that . is their current root.
>>> But my point is that there might not actually *be* a valid path that
>>> gets you to a file descriptor. It can't remember something that
>>> doesn't exist. (And it's pretty impractical to do that even if it
>>> does.)
>> Remember, we are talking about a hypothetical OS that handles hardlinks
>> to directories, and defines that .. will point to the parent used to
>> come to it, NOT just having current *NIX allowing hardlinks to
>> directories with no remediation of the issues cause.
> Yes. I'm assuming that you can define things any way you like.
>
>> One result of the definition, is that when you open a file/directory, if
>> it might be a directory, the system WILL need to remember the path to it
>> (so as to provide a value for ..) and that memory will provide a
>> 'reference' for the directories so they can't go away (just like an
>> unlinked file stays around will someone has it open). The normal way to
>> get a file descriptor starts from a path, so the path had to exist in
>> the first place, and since we are assuming that to get .., it keep a
>> reference to that path, it can't truly go away.
>>
> But as I've pointed out, you don't always *have* a valid path. File
> descriptors can be passed from process to process (most commonly by
> being retained when you fork and not closed when you exec, but there
> are other ways too, eg sockets), so even though starting with a path
> is the most common, it isn't the only way, and you could easily have
> an FD with no associated path, and then read it to find out what ".."
> is.
>
> Also, even if all that could be solved, I don't like the idea that
> reading the same directory from two different sources leads to
> different results. Is it really the same directory if reading it in
> different ways gives different results?
>
> ChrisA

But when you pass the file descriptors to another process, the OS knows
this, so the data that was pointed by the descriptor (which is where you
would keep it anyway) still has it.  There is no normal way that I know
of from user land to get to a directory except from a path or at least
an object that could have remembered a path. For a FD, that FD started
with a path, so it could still remember it.

Part of your issue is likely that you are used to thinking of the file
system as a pure tree, and *nix likes to do that too. Once you accept
hard links to directories as 'normal', suddenly the meaning of .. gets
funny, as there is not a unique parent, but getting the parent as you
got there could be useful for recursive file system searches (which will
need protection from getting stuck in a loop).

My research says that Unix System V allowed them, but restricted them to
super users, to avoid the bigger problems with them. I don't know how it
handle the issue of ..

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Andras Tantos]

On 9/1/2020 12:41 PM, MRAB wrote:

On 2020-09-01 05:45, Andras Tantos wrote:

All,

I'm new here, so please direct me to the right forum, if this is not the
one...

What I'm trying to do is to call a function, but monitor all the local
variable accesses within that function. What I thought I would need to
do, is to |exec| the function with a custom |locals| dictionary. The
code below attempts to do it.

|classMySymbolTable(dict):defset_type(self,t):self.type =t
def__getitem__(self,key):print(f"Requesting key {key} from {self.type}
table")returnsuper().__getitem__(key)def__setitem__(self,key,value):print(f"Setting 


key {key} from {self.type} table to value
{value}")returnsuper().__setitem__(key,value)defmylocals(func):defwrapper(*args,**kwargs):loc 


=MySymbolTable()glob
=MySymbolTable(globals())loc.set_type("local")glob.set_type("global")exec(func.__code__,glob,loc)returnwrapper 


@mylocalsdeffun1():print(f"fun1 with locals: {type(locals())} and
globals: {type(globals())}")a =1b =2c =3a =b c =5fun1()|

However, when I run it, I get the following output:

|Requestingkey printfromglobaltable Requestingkey type fromglobaltable
Requestingkey locals fromglobaltable Requestingkey type fromglobaltable
Requestingkey globals fromglobaltable fun1
withlocals:andglobals:|

That is to say, global accesses are redirected to my custom dict, but
local assignments are not. You can even see that in the types of the two
objects printed in the last line.

My hunch is that since I'm using the functions |__code__| member, I end
up executing pre-compiled byte-code which has already assumptions about
the locals dict built into it.

If I'm right (or even if I'm not), what is the right way to achieve my
goal: that local assignments get redirected to the supplied dictionary?

CPython is able to identify all of the local names of a function and, 
basically, for reasons of efficiency, it uses slots for the local 
names instead of an actual dict. 'locals()' just returns a dict that 
represents those local names and their current values, but modifying 
that dict has no effect on the actual local names. In short, there 
isn't really a local dict that you can replace.


While I'm sure you're right, it certainly is well hidden:

Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 
07:34:03) [MSC v.1916 64 bit (AMD64)] on win32

Type "help", "copyright", "credits" or "license" for more information.
>>> locals()
{'__name__': '__main__', '__doc__': None, '__package__': None, 
'__loader__': , '__spec__': 
None, '__annotations__': {}, '__builtins__': }

>>> a =3
>>> locals()
{'__name__': '__main__', '__doc__': None, '__package__': None, 
'__loader__': , '__spec__': 
None, '__annotations__': {}, '__builtins__': (built-in)>, 'a': 3}

>>> locals()['a'] = 4
>>> locals()
{'__name__': '__main__', '__doc__': None, '__package__': None, 
'__loader__': , '__spec__': 
None, '__annotations__': {}, '__builtins__': (built-in)>, 'a': 4}

>>>

In other words, I can indeed change the value of a local variable 
through the locals() dict. I've modified my code to see if I can at 
least introduce locals through the value passed in for exec, but even 
that seems to fail:


classMySymbolTable(dict):
defset_type(self,t):
self.type = t
def__getitem__(self,key):
print(f"Requesting key {key} from {self.type} table")
returnsuper().__getitem__(key)
def__setitem__(self,key,value):
print(f"Setting key {key} from {self.type} table to value {value}")
returnsuper().__setitem__(key, value)
defmylocals(func):
defwrapper(*args,**kwargs):
loc = MySymbolTable()
glob = MySymbolTable(globals())
loc.set_type("local")
loc["xxx"]=123
glob.set_type("global")
glob["yyy"]=42
exec(func.__code__, glob, loc)
return wrapper
@mylocals
deffun1():
print("locals at the beginning:")
print(", ".join(f"{name}: {value}"for name, value inlocals().items()))
print("globals at the beginning:")
print(", ".join(f"{name}: {value}"for name, value inglobals().items()))
print(f"fun1 with locals: {type(locals())} and globals: {type(globals())}")
a =1
b =2
c =3
a = b
c =5
print("locals at the end:")
print(", ".join(f"{name}: {value}"for name, value inlocals().items()))
print("globals at the end:")
print(", ".join(f"{name}: {value}"for name, value inglobals().items()))
fun1()

This however still fails:

Setting key xxx from local table to value 123
Setting key yyy from global table to value 42
Requesting key print from global table
locals at the beginning:
Requesting key print from global table
Requesting key locals from global table

Requesting key print from global table
globals at the beginning:
Requesting key print from global table
Requesting key globals from global table
__name__: __main__, __doc__: None, __package__: None, __loader__: 
<_frozen_importlib_external.SourceFileLoader object at 
0x01CA2DC22640>, __spec__: None, __annotations__: {}, __builtins__: 
, __file__: locals_test.py, __cached__: 
None, MySymbolTable:

Re: Symlinks already present

[Cameron Simpson]

On 01Sep2020 18:42, Grant Edwards  wrote:
>On 2020-09-01, Richard Damon  wrote:
>> Remember, we are talking about a hypothetical OS that handles 
>> hardlinks
>> to directories, and defines that .. will point to the parent used to
>> come to it, NOT just having current *NIX allowing hardlinks to
>> directories with no remediation of the issues cause.
>
>I can testify from personal experience that SunOS 3/4 was in the
>latter category.  After creating a hard-link to a directory, things
>like fsck and the filesystem dump backup utility got very upset and
>confused.  IIRC the only way to recover was to nuke the involved
>inodes then let fsck try to pick up the pieces and put them in the
>lost+found. IIRC, I managed to recover without losing any files, but
>it wasn't a fun day.

IIRC this is very old behaviour indeed, in that when I was using V7 UNIX 
the manual mentioned that only root could make a hard link to a 
directory, and I think there was some allusion to this being how rename 
was done for directories, just like files (no rename OS call, just link 
to new name, unlink old). That would mean "mv" was setuid root at that 
time, which does _not_ ring any bells. So I may be misremembering the 
details.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Chris Angelico]

On Wed, Sep 2, 2020 at 8:44 AM Andras Tantos
 wrote:
>
> While I'm sure you're right, it certainly is well hidden:
>
> Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020,
> 07:34:03) [MSC v.1916 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> locals()
> {'__name__': '__main__', '__doc__': None, '__package__': None,
> '__loader__': , '__spec__':
> None, '__annotations__': {}, '__builtins__': }
>  >>> a =3
>  >>> locals()
> {'__name__': '__main__', '__doc__': None, '__package__': None,
> '__loader__': , '__spec__':
> None, '__annotations__': {}, '__builtins__':  (built-in)>, 'a': 3}
>  >>> locals()['a'] = 4
>  >>> locals()
> {'__name__': '__main__', '__doc__': None, '__package__': None,
> '__loader__': , '__spec__':
> None, '__annotations__': {}, '__builtins__':  (built-in)>, 'a': 4}
>  >>>
>
> In other words, I can indeed change the value of a local variable
> through the locals() dict.

Not quite. What you're seeing there is that there aren't any locals,
and you're getting the globals:

>>> locals() is globals()
True

In this situation, locals() DOES return a real dictionary, and it can
be manipulated as one. But only because it's the same as globals(),
and you can do all that with globals.

> I've modified my code to see if I can at
> least introduce locals through the value passed in for exec, but even
> that seems to fail:
>
> classMySymbolTable(dict):
> defset_type(self,t):
> self.type = t
> def__getitem__(self,key):
> print(f"Requesting key {key} from {self.type} table")
> returnsuper().__getitem__(key)
> def__setitem__(self,key,value):
> print(f"Setting key {key} from {self.type} table to value {value}")
> returnsuper().__setitem__(key, value)
> defmylocals(func):
> defwrapper(*args,**kwargs):
>  loc = MySymbolTable()
>  glob = MySymbolTable(globals())
>  loc.set_type("local")
>  loc["xxx"]=123
>  glob.set_type("global")
>  glob["yyy"]=42
> exec(func.__code__, glob, loc)
> return wrapper

Your whitespace is getting messed up again, although not as badly now.
Try posting in plain text, not "rich text" or HTML or formatted text
or anything.

> It still appears to me that 'exec()' simply ignores the dict passed in
> as the local parameter: it doesn't even seem to initialize locals() from it.

I'm honestly not sure what's going on, because it's hard to follow
your code. But my suspicion is that when you exec a code object
(rather than compiling and executing a string), it's using cells/slots
rather than actually using your locals dict.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Terry Reedy]

On 9/1/2020 5:58 PM, Andras Tantos wrote:


On 9/1/2020 12:41 PM, MRAB wrote:


CPython is able to identify all of the local names of a function and, 


Note >>>of a function<<<.

basically, for reasons of efficiency, it uses slots for the local 
names instead of an actual dict. 'locals()' just returns a dict that 
represents those local names and their current values, but modifying 
that dict has no effect on the actual local names. In short, there 
isn't really a local dict that you can replace.


This applies to code in a function (def statement or lambda expression) 
and only that.


[Snip experiments with non-function code at >>> prompt.]

In other words, I can indeed change the value of a local variable 
through the locals() dict.


This is true for code that is not part of a function body, which is top 
level code and class code that is not part of a method body.



--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Symlinks already present

[Grant Edwards]

On 2020-09-01, Richard Damon  wrote:

> My research says that Unix System V allowed them, but restricted them to
> super users, to avoid the bigger problems with them. I don't know how it
> handle the issue of ..

SunOS-3/4 (a BSD 4.x derivative) allowed them, but restricted them to
root.  It did not handle the issue of '..' at all, and if you did much
of anything at all with the directory while the "extra" link was
present (which meant the filesystem was no longer a tree) it didn't go
well.  It wasn't something you did a second time.

--
Grant




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Andras Tantos]

On 9/1/20 1:42 PM, Chris Angelico wrote:

On Wed, Sep 2, 2020 at 5:00 AM Andras Tantos
 wrote:

All,

I'm new here, so please direct me to the right forum, if this is not the
one...

What I'm trying to do is to call a function, but monitor all the local
variable accesses within that function. What I thought I would need to
do, is to |exec| the function with a custom |locals| dictionary. The
code below attempts to do it.

|classMySymbolTable(dict):defset_type(self,t):self.type =t
def__getitem__(self,key):print(f"Requesting key {key} from {self.type}
table")returnsuper().__getitem__(key)def__setitem__(self,key,value):print(f"Setting
key {key} from {self.type} table to value
{value}")returnsuper().__setitem__(key,value)defmylocals(func):defwrapper(*args,**kwargs):loc
=MySymbolTable()glob
=MySymbolTable(globals())loc.set_type("local")glob.set_type("global")exec(func.__code__,glob,loc)returnwrapper
@mylocalsdeffun1():print(f"fun1 with locals: {type(locals())} and
globals: {type(globals())}")a =1b =2c =3a =b c =5fun1()|

Your code's messed up in the post, so I'm guessing on how this is
actually meant to be written. Most of your whitespace has been
destroyed.


Indeed. That's annoying, lemme see if I can copy-paste something in...

class MySymbolTable(dict):
    def set_type(self, t):
    self.type = t
    def __getitem__(self, key):
    print(f"Requesting key {key} from {self.type} table")
    return super().__getitem__(key)
    def __setitem__(self, key, value):
    print(f"Setting key {key} from {self.type} table to value {value}")
    return super().__setitem__(key, value)

def mylocals(func):
    def wrapper(*args, **kwargs):
    loc = MySymbolTable()
    glob = MySymbolTable(globals())
    loc.set_type("local")
    glob.set_type("global")
    exec(func.__code__, glob, loc)
    return wrapper

@mylocals
def fun1():
    print(f"fun1 with locals: {type(locals())} and globals: 
{type(globals())}")

    a = 1
    b = 2
    c = 3
    a = b
    c = 5

fun1()


That is to say, global accesses are redirected to my custom dict, but
local assignments are not. You can even see that in the types of the two
objects printed in the last line.

My hunch is that since I'm using the functions |__code__| member, I end
up executing pre-compiled byte-code which has already assumptions about
the locals dict built into it.

Yes; and furthermore, CPython generally doesn't even use a dictionary
for a function's locals. Instead, it uses cells in a special array;
you can poke around with this by disassembling the code (see the
built-in "dis" module for details) and looking at the
LOAD_FAST/STORE_FAST opcodes.

You might be able to mess with this by using a closure, but local
variable references are generally considered to be an implementation
detail of the function, and it's not going to be easy to mess with
them from the outside reliably. What's the goal here?

ChrisA


I did see these macros in the CPython source code. What it seems to 
imply is that if I wanted to do what I intend, that is to hook every 
local variable assignment, I would have to modify the AST. That seems 
rather more effort than simply supplying my own locals dictionary. I 
will try to look myself as well, but do you have an inkling as to how 
complex would it be to convince CPython to optimize locals only if no 
custom locals is provided during exec?


To my untrained eye, it seems to be an unsafe optimization that breaks 
the contract of the exec API.


Andras




--
https://mail.python.org/mailman/listinfo/python-list

Turtle Import

[Barnabas Harris]

My import function mysteriously stop working. Is there a way to restore it as I 
need the second screen to see the turtle move when I command it from python 
3.8.5 shell but when I hit enter nothing happens. 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Chris Angelico]

On Wed, Sep 2, 2020 at 10:23 AM Andras Tantos
 wrote:
> I did see these macros in the CPython source code. What it seems to
> imply is that if I wanted to do what I intend, that is to hook every
> local variable assignment, I would have to modify the AST. That seems
> rather more effort than simply supplying my own locals dictionary. I
> will try to look myself as well, but do you have an inkling as to how
> complex would it be to convince CPython to optimize locals only if no
> custom locals is provided during exec?
>
> To my untrained eye, it seems to be an unsafe optimization that breaks
> the contract of the exec API.

By the time you're calling exec, it's too late to decide whether to
optimize, since you're using a precompiled code object. If you want
that behaviour, try exec'ing a string of code instead.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Andras Tantos]

Snipping old parts...

On 9/1/20 4:01 PM, Chris Angelico wrote:

On Wed, Sep 2, 2020 at 8:44 AM Andras Tantos
 wrote:

While I'm sure you're right, it certainly is well hidden:

...

In other words, I can indeed change the value of a local variable
through the locals() dict.

Not quite. What you're seeing there is that there aren't any locals,
and you're getting the globals:


locals() is globals()

True

In this situation, locals() DOES return a real dictionary, and it can
be manipulated as one. But only because it's the same as globals(),
and you can do all that with globals.

That could very well be the case.



I've modified my code to see if I can at
least introduce locals through the value passed in for exec, but even
that seems to fail:

...

Your whitespace is getting messed up again, although not as badly now.
Try posting in plain text, not "rich text" or HTML or formatted text
or anything.


OK, and sorry about that. May third time a charm?

class MySymbolTable(dict):
    def set_type(self, t):
    self.type = t
    def __getitem__(self, key):
    print(f"Requesting key {key} from {self.type} table")
    return super().__getitem__(key)
    def __setitem__(self, key, value):
    print(f"Setting key {key} from {self.type} table to value {value}")
    return super().__setitem__(key, value)

def mylocals(func):
    def wrapper(*args, **kwargs):
    loc = MySymbolTable()
    glob = MySymbolTable(globals())
    loc.set_type("local")
    loc["xxx"] = 123
    glob.set_type("global")
    glob["yyy"] = 42
    exec(func.__code__, glob, loc)
    return wrapper

@mylocals
def fun1():
    print("locals at the beginning:")
    print(", ".join(f"{name}: {value}" for name, value in 
locals().items()))

    print("globals at the beginning:")
    print(", ".join(f"{name}: {value}" for name, value in 
globals().items()))
    print(f"fun1 with locals: {type(locals())} and globals: 
{type(globals())}")

    a = 1
    b = 2
    c = 3
    a = b
    c = 5
    print("locals at the end:")
    print(", ".join(f"{name}: {value}" for name, value in 
locals().items()))

    print("globals at the end:")
    print(", ".join(f"{name}: {value}" for name, value in 
globals().items()))


fun1()


It still appears to me that 'exec()' simply ignores the dict passed in
as the local parameter: it doesn't even seem to initialize locals() from it.

I'm honestly not sure what's going on, because it's hard to follow
your code. But my suspicion is that when you exec a code object
(rather than compiling and executing a string), it's using cells/slots
rather than actually using your locals dict.

ChrisA


Thanks for all the help!

Indeed, I'm executing a code object and not a string. Is there any way 
to get to the string version of the function and force a re-compile of 
it during an exec? Would it help?


I guess if I was doing that, I would need to execute the 'body' of the 
function within the exec and by passing in a local parameter I can 
pretend it has its own call-stack entry. But is that truly equivalent to 
calling the original function?


It seems to me that even it were all possible, the solution would be 
very brittle, for example, additional decorators would mess things up. 
Though that's probably the case independent of the 
code-object/source-code distinction.


Andras


--
https://mail.python.org/mailman/listinfo/python-list

RE: What kind of magic do I need to get python to talk to Excel xlsm file?

[Steve]

>> Requires numpy, dateutil, pytz, setuptools, and optionally numexpr, 
>> bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, 
>> brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.

>It means you need all those things installed before you can install Pandas.

If that is the case, then it is just not worth it.  However, I am seeing it 
more as a poorly constructed sentence.

How about:

>> Requires numpy, dateutil, pytz, setuptools , 
and optionally:  
numexpr, bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, 
brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.

Does it suggest that the first four are required and the rest are optional?
Even then, it is still ambiguous.



FootNote:
If money does not grow on trees, then why do banks have branches?

-Original Message-
From: Chris Angelico  
Sent: Tuesday, September 1, 2020 5:08 PM
To: Steve 
Cc: Python 
Subject: Re: What kind of magic do I need to get python to talk to Excel xlsm 
file?

On Wed, Sep 2, 2020 at 6:59 AM Steve  wrote:
>
> Wow, that is quite a list
> Here is what I narrowed it down to. What does that paragraph starting with 
> "Requires" mean?
>
> ===
> Pandas: a cross-section and time series data analysis toolkit.
>
> Requires numpy, dateutil, pytz, setuptools, and optionally numexpr, 
> bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, 
> brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.
>

It means you need all those things installed before you can install Pandas.

ChrisA

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Chris Angelico]

On Wed, Sep 2, 2020 at 11:29 AM Andras Tantos
 wrote:
> OK, and sorry about that. May third time a charm?

Thank you! Whatever you did this time, that worked :)

> Indeed, I'm executing a code object and not a string. Is there any way
> to get to the string version of the function and force a re-compile of
> it during an exec? Would it help?

Eee normally you'd do that by, well, *not* compiling
it. But maybe the inspect module can help here:

https://docs.python.org/3/library/inspect.html#inspect.getsource

There's no guarantee that it will work though.

> I guess if I was doing that, I would need to execute the 'body' of the
> function within the exec and by passing in a local parameter I can
> pretend it has its own call-stack entry. But is that truly equivalent to
> calling the original function?

I think it is getting an actual call stack entry anyway.

Now, if your goal is to do something with an expression evaluator,
then there are LOTS of options. Let's say you want to allow people to
write code that implicitly calls on some data source, so typing "x **
2 + y ** 2" would look up x and y in some way and then do arithmetic
with them. If you get the original source as a string, you can most
certainly do things with the globals as you compile it. Or if you want
more reliability, compile it to AST, walk the tree, and replace nodes
as required, before compiling it the rest of the way for execution. Or
run the code at global scope, with the globals doing your magic and
the builtins doing everything else. Lots of ways to do things, as long
as you start with the source code.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What kind of magic do I need to get python to talk to Excel xlsm file?

[Chris Angelico]

On Wed, Sep 2, 2020 at 11:43 AM Steve  wrote:
>
> >> Requires numpy, dateutil, pytz, setuptools, and optionally numexpr, 
> >> bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, snappy, 
> >> brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and whatnot.
>
> >It means you need all those things installed before you can install Pandas.
>
> If that is the case, then it is just not worth it.  However, I am seeing it 
> more as a poorly constructed sentence.
>
> How about:
>
> >> Requires numpy, dateutil, pytz, setuptools ,
> and optionally:
> numexpr, bottleneck, scipy, matplotlib, pytables, lxml, xarray, blosc, 
> snappy, brotlipy, backports.lzma, statsmodels, sqlalchemy, psycopg2, and 
> whatnot.
>
> Does it suggest that the first four are required and the rest are optional?
> Even then, it is still ambiguous.
>

I believe you are correct, but I usually let pip do all the work.

(But I don't use Windows.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set custom locals for function call?

[Andras Tantos]

On 9/1/20 5:31 PM, Chris Angelico wrote:

On Wed, Sep 2, 2020 at 10:23 AM Andras Tantos
 wrote:

I did see these macros in the CPython source code. What it seems to
imply is that if I wanted to do what I intend, that is to hook every
local variable assignment, I would have to modify the AST. That seems
rather more effort than simply supplying my own locals dictionary. I
will try to look myself as well, but do you have an inkling as to how
complex would it be to convince CPython to optimize locals only if no
custom locals is provided during exec?

To my untrained eye, it seems to be an unsafe optimization that breaks
the contract of the exec API.

By the time you're calling exec, it's too late to decide whether to
optimize, since you're using a precompiled code object. If you want
that behaviour, try exec'ing a string of code instead.

ChrisA


I guess, it depends. I can think of several ways of dealing with this.

Probably the highest performant version would be to generate byte-code 
along these lines for every function body:


    if <>:
     <>
    else:
     <>

This of course makes the byte-code about twice as large as today.

One could also patch up the SETLOCAL/GETLOCAL macros to test for this 
condition, in which case no byte-code changes are needed, but 
performance would degrade because of all the - normally unnecessary - 
checks.


There's also a possibility of detecting this condition within the 
underlying C implementation of exec() and trigger a re-generation of the 
byte-code from source.


I don't feel confident enough to even attempt anything but the middle 
approach, but even then: this is a change to CPython, obviously 
something that needs community vetting and approval.


Reacting to your last sentence: I have investigated how to get the 
source code, given a function object, but the best I could come up with is:


    1. Retrieve the source code for the function, using inspect.getsource()

    2. massage it to contain only the function body (that is to remove 
the def ... line)


    3. pass that to exec.

With that, I finally have been able to get locals work, but here, I'm 
back to the other case, where I essentially promoted the body of the 
function into the global namespace, so now locals() and globals() are 
the same thing. This is hardly the same context in which the function 
would originally execute.


Andras


--
https://mail.python.org/mailman/listinfo/python-list