Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[rocky]

Paul Moore writes:
 > 2008/12/23 Rocky Bernstein :
 > > Now that there is a package mechanism (are package mechanisms?) like
 > > zipimporter that bundle source code into a single file, should the
 > > notion of a "file" location should be adjusted to include the package
 > > and/or importer?
 > 
 > Check PEP 302 (http://www.python.org/dev/peps/pep-0302/) specifically
 > the get_source (optional) method. 

Yes, that's one of the things I was thinking when I wrote:

  As best as I can tell, PEP 302 which discussed importer hooks and
  suggests a standard way to get file data.

And by "suggests" I meant was implying that yes I know this is
optional.


 > It's not exactly what you describe,
 > but it may help. 

Yes, it's not exactly what is desired. 

 > Please note that it's optional - if you loaded the
 > code from a zipfile containing only bytecode files, there is no source
 > to get, so you have to be prepared for that case. But if the source is
 > available, this should give you a way of getting to it.

What is wanted is a uniform way get and describe a file location
from a code object that takes into account the file might be a member
of an archive. 

Are there even guidelines for saying what string goes into a code
object's co_filename? Clearly it should be related to the source code
that generated the code, and there are various conventions that seem
to exist when the code comes from an "eval" or an "exec". 

But empirically it seems as though there's some variation. It could be
an absolute file or a file with no root directory specified. (But is
it possible to have things like "." and ".."?). And in the case of a
member of a package what happens? Should it be just the member without
the package? Or should it include the package name like
   /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/tracer.py ? 

Or be unspecified? If left unspecified as I gather it is now, it makes
it more important to have some sort of common routine to be able to
pick out the archive part in a filesystem from the member name inside 
the archive.


 > 
 > Paul.
 > 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[rocky]

Paul Moore writes:
 > 2008/12/23 R. Bernstein :
 > >  A use case here I am thinking of here is in a stack trace or a
 > >  debugger, or a tool which wants to show in great detail, information
 > >  from a code object obtained possibly via a frame object.
 > 
 > Thanks for the clarifications. I see what you're after much better now.
 > 
 > > I find it kind of sucky to see in a traceback: "" as opposed
 > > to the text (or prefix of the text) of the actual string that was
 > > passed. Or something that has been referred to as a "pseudo-file" like
 > > /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/foo/bar.py
 > > when it is really member foo/bar.py of zipped egg
 > > /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg.
 > 
 > Fair comment. That points to a "human readable" type of string. It's
 > not available at the moment, but I guess it could be.
 > 
 > But see below.
 > 
 > >  > -  xxx.get_source_code(token) is a function (I don't know where,
 > >  > xxx is a placeholder for some "suitable" module) which, given such a
 > >  > token, returns the source, or None if there's no viable concept of
 > >  > "the source".
 > >
 > > There always is a viable concept of a source. It's whatever was done
 > > to get the code. For example, if it was via an eval then the source
 > > was the eval function and a string, same for exec. If it's via
 > > database access, well that then and some summary info about what's
 > > known about that.
 > 
 > Hmm, "source" colloquially, yes "bytecode loaded from \xxx.pyc",
 > for example. But not "source" in the sense of "source code". Some
 > applications run with only bytecode shipped, no source code available
 > at all.
 > 
 > > There are two problems. One is displaying location information in an
 > > unambiguous way -- the pseudo-file above is ambiguous and so is
 > >  since there's no guarentee that OS's make to not name a file
 > > that. The second problem is programmatically getting information such
 > > as a debugger or an IDE might do so that the information can be
 > > conveyed back to a user who might want to inspect surrounding source
 > > code or modules.
 > 
 > This is more than you were asking for above.
 > 
 > The first problem is addressed with a "human readable" (narrative)
 > description, as above.
 > 
 > The second, however, requires machine-readable access to source code
 > (if it exists). That's what the loader get_source() call does for you.
 > But you have to be prepared for the fact that it may not be possible
 > to get source code, and decide what you want to happen in that case.

I'm missing your point here. 

When one uses information from a traceback, or is in a debugger, or is
in an IDE, it is assumed that in order to use the information given
you'll need access to the source code. And IDE's and debuggers have
had to deal with the fact that source code is not available from day
one, even before there was zipimporter.

In order to get the strings of source text that linecache.getlines()
gives, it has to prowl around for other information, possibly looking
for a loader along the protocol defined in PEP 302 and/or others. And
its that information that a debugger, IDE or some tool of that ilk
might need.

Many IDE's and debuggers nowadays open a socket and pass information
back and forth over that. An obvious advantage is that it means you
can debug remotely. But in order for this to work, some information is
generally passed back and for regarding the location of the source
text. In the Java world and Eclipse for example, it is possible for
the jar to be in a different location from on the machine which you
might be debugging on. And probably too often that jar isn't the same
one. So it is helpful in this kind of scenario to break out a location
into the name of a jar and the member inside the jar. Perhaps also
some information about that jar.

It is possible that instead of passing around locations, debuggers and
such tools instead use get_source() instead, because that's what
Python has to offer.  :-)

I jest here, but honestly I've been surprised that there is no IDE
that I know of that in fact works this way. The machine running the
code clearly may have more accurate access to the source than a
front-end IDE. Undeterred by the harsh facts of reality, I have hope
that someday there *might* be an IDE that has provision for this. So
in a Ruby debugger (ruby-debug) one can request checksum information
on the files the debugger things are loaded in order to facilitate
checking that the source one an IDE might be showing in fact matches
the source for that part of the code that one is currently under
investigation.


 > 
 > >  > I hope this is of some help,
 > >
 > > Yes, thanks. At least I now have a clearer idea of the state of
 > > where things stand.
 > 
 > Good. Sorry it's not better news :-)
 > 
 > Paul
 > 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/li

Re: [Python-Dev] Should execv() call _run_exitfuncs()? If not, should _run_exitfuncs() be private?

[rocky]

Guido van Rossum writes:
 > Depending on the use for the exit function you might or might not want
 > it run at the occasion of exec*(). E.g. I imagine that in a typical
 > fork() + exec*() scenario, calling the exit functions in the child
 > process would be a mistake.
 > 
 > So I don't think the hooks should be called by default. However you
 > are welcome to call the function (leading underscore and all) if it
 > helps in your case.

Ok - got it. Thanks, everyone, for the clarification(s).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should there be a source-code checksum in module objects?

[rocky]

Brett Cannon writes:
 > On Mon, Feb 2, 2009 at 00:52, Rocky Bernstein  wrote:
 > > As I've mentioned, I've been re-examining from ground up the whole
 > > state of affairs in writing a debugger.
 > >
 > > One of the challenges of a debugger or any source-code analysis tool
 > > is verifying that the source-code that the tool is reporting on
 > > corresponds to the compiled object under execution. (For debuggers,
 > > this problem becomes more likely to occur when you are debugging on a
 > > computer that isn't the same as the computer where the code is
 > > running.)
 > >
 > > Is there a checksum of the source text computed and stored in
 > > compilation?
 > 
 > No, only the bytecode version used, the mtime of the source the
 > bytecode is derived from, and the bytecode itself.
 > 
 > > If not, should there be one?
 > 
 > If you are planning to read this directly from the .pyc file then it
 > is not needed as the mtime timestamp in the .pyc should in general be
 > enough to detect changes to the source.

I'm not sure I understand. I am debugging program P on computer A
which may have (probably did?) do a compile. I am on computer B which
I have the source code. Maybe it is checked out from a version control
system same as on A. Maybe it has bytecodes, maybe not. But the
expectation is that the programmer thinks it matches what is currently
on A that the programmer is debuggging. Can I tell for certain?

Suppose I'm writing a code coverage tool which can accumulate
statistics over many runs. I notice that the date on the Python file
changes, again one way it might is that it might be checked out fresh
and subversion for example will put in a current timestamp. How can
the code coverage tool know that the source hasn't changed even though
the file may have disappeared or maybe was modified several times but
in the end stays the same?

Thanks.

 > 
 > -Brett
 > 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should there be a source-code checksum in module objects?

[rocky]

Guido van Rossum writes:
 > I suggest that you move this discussion to python-ideas to ferret out
 > a possible implementation and API; or to find out work-arounds.

Okay. Done. 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Can I introspect/reflect to get arguments exec()?

[Rocky Bernstein]

[asked on comp.lang.python but no takers. So I'm bumping it up a notch.]

I have ported my Python debugger pydbgr to Python3. See [1] or [2].

Inside the debugger, when there is an exec() somewhere in the call stack,
I'd like to be able to retrieve the string parameter. With this, the
debugger can show part of the string in a call stack. Or it can show the
text when the frame is set to that exec() frame.

Going further, the debugger could write the exec string out to a temporary
file. And when reporting locations, it could report not just something like
" line 4", but also give that temporary file name which a front-end
could use as well.

So consider this code using inspect.getargvalues() and
inspect.currentframe():

import inspect
def my_exec(string):
show_args(inspect.currentframe()) # simulate exec(string)

def show_args(frame):
print(inspect.getargvalues(frame))

my_exec("show_args(inspect.currentframe())")
exec("show_args(inspect.currentframe())")


When run this is the output:

python3 exec-args.py
ArgInfo(args=['string'], varargs=None, keywords=None, locals={'string':
'show_args(inspect.currentframe())'})
ArgInfo(args=[], varargs=None, keywords=None, locals={'my_exec':
,, ...


In a different setting, CPython byte-code assembly that gets generated for
running exec() is:

 25   88 LOAD_GLOBAL10 (exec)
  91 LOAD_CONST  4
('show_args(inspect.currentframe())')
   -->94 CALL_FUNCTION   1
  97 POP_TOP

What's going on?

Also, I have the same question for CPython 2.6 or Python 2.7.

Thanks!

[1] http://code.google.com/p/pydbgr/
[2] http://code.google.com/p/python3-trepan/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[Rocky Bernstein]

On Tue, Mar 26, 2013 at 10:18 PM, Benjamin Peterson wrote:

> 2013/3/26 Rocky Bernstein :
> > [asked on comp.lang.python but no takers. So I'm bumping it up a notch.]
> >
> > I have ported my Python debugger pydbgr to Python3. See [1] or [2].
> >
> > Inside the debugger, when there is an exec() somewhere in the call stack,
> > I'd like to be able to retrieve the string parameter. With this, the
> > debugger can show part of the string in a call stack. Or it can show the
> > text when the frame is set to that exec() frame.
> >
> > Going further, the debugger could write the exec string out to a
> temporary
> > file. And when reporting locations, it could report not just something
> like
> > " line 4", but also give that temporary file name which a
> front-end
> > could use as well.
> >
> > So consider this code using inspect.getargvalues() and
> > inspect.currentframe():
> >
> > import inspect
> > def my_exec(string):
> > show_args(inspect.currentframe()) # simulate exec(string)
> >
> > def show_args(frame):
> > print(inspect.getargvalues(frame))
> >
> > my_exec("show_args(inspect.currentframe())")
> > exec("show_args(inspect.currentframe())")
> >
> >
> > When run this is the output:
> >
> > python3 exec-args.py
> > ArgInfo(args=['string'], varargs=None, keywords=None,
> locals={'string':
> > 'show_args(inspect.currentframe())'})
> > ArgInfo(args=[], varargs=None, keywords=None, locals={'my_exec':
> > ,, ...
> >
> >
> > In a different setting, CPython byte-code assembly that gets generated
> for
> > running exec() is:
> >
> >  25   88 LOAD_GLOBAL10 (exec)
> >   91 LOAD_CONST  4
> > ('show_args(inspect.currentframe())')
> >-->94 CALL_FUNCTION   1
> >   97 POP_TOP
> >
> > What's going on?
>
> execing something is not the same as calling it, so there are no arguments.
>

Okay. But is the string is still somewhere in the CPython VM stack? (The
result of LOAD_CONST 4 above). Is there a way to pick it up from there? At
the point that we are stopped the exec action hasn't taken place yet.




>
>
> --
> Regards,
> Benjamin
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?

[Rocky Bernstein]

Thank you for your very thoughtful and detailed explanation of what is
going on and for your considerations as to how to get this done (as opposed
to why it *can't *be done).

It looks like DecoratorTools or more likely a customized version of it is
the way to go!

(The important info is above. The rest are just some geeky details)

You see, there are some places where additional care may be needed in my
setting.

The debuggers I write use sometimes don't use just the getline module but
also use my own pyficache module. I want to also cache things like file
stat() info, provide a SHA1 for the text of the file, and provide colorized
syntax-highlighted versions of the text when desired. But since I control
pyficache, I can mirror the changes made to getline.

Of course the debugger uses sys.settrace too, so the evil-ness of that is
definitely not a concern. But possibly I need to make sure that since the
DecoratorTools and the debugger both hook into trace hooks they play nice
together and fire in the right order. And for that I created another module
called tracer() which takes into account that one might want to specify
priorities in the chain hook order, and that one might want a to filter out
(i.e. ignore) certain calls to the hook function for specific hooks.

It may be a while before I seriously get to this, but again, it is good to
have in mind an approach to take. So thanks again.

On Thu, Mar 28, 2013 at 1:45 AM, PJ Eby  wrote:

> On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein  wrote:
> > Okay. But is the string is still somewhere in the CPython VM stack? (The
> > result of LOAD_CONST 4 above). Is there a way to pick it up from there?
>
> Maybe using C you could peek into the frame's value stack, but that's
> not exposed to any Python API I know of.  But that still doesn't help
> you, because the value will be removed from the stack before exec() is
> actually called, which means if you go looking for it in code called
> from the exec (e.g. the call event itself), you aren't going to see
> the data.
>
> > At the point that we are stopped the exec action hasn't taken place yet.
>
> That doesn't help if you're using line-level tracing events.  At the
> beginning of the line, the data's not on the call stack yet, and by
> the time you enter the frame of the code being exec'd, it'll be off
> the stack again.
>
> Basically, there is no way to do what you're asking for, short of
> replacing the built-in exec function with your own version.  And it
> still won't help you with stepping through the source of functions
> that are *compiled* using an exec() or compile(), or any other way of
> ending up with dynamically-generated code you want to debug.
>
> (Unless you use something like DecoratorTools to generate it, that is
> -- DecoratorTools has some facilities for caching
> dynamically-generated code so that it works properly with debuggers.
> But that has to be done by the code doing the generation, not the
> debugger.  If the code generator uses DecoratorTools' caching support,
> then any debugger that uses the linecache module will Just Work.  It
> might be nice for  the stdlib should have something like this, but you
> could also potentially fake it by replacing the builtin eval, exec,
> compile, etc. functions w/versions that cache the source.)
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] code.interact() locals + globals?

[Rocky Bernstein]

I notice  code.interact() in Python 2.6 has a parameter for locals but not
globals. Should it?

Ultimately,  *exec* is used underneath so I don't see a technical reason why
it couldn't add a global parameter. Also, although one could pass in a
dictionary that is the merger or update() of locals() and globals(),  this
has a couple of downsides.

First, if the intention is to allow update of local and global variables,
combining the two into a single dictionary wreaks havoc. Even if one doesn't
care about this, when the underlying *exec* is performed it will see
different and wrong values for the functions locals() and globals() among
other problems of this kind.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] License for code extracted from a Python library?

[Rocky Bernstein]

Hi -

I'm not sure if this is the right place to ask this, but I don't know of a
more appropriate place.

Sometime around 2007 I first extracted a function that was in Python's Cmd
class and used that. Since then, I have modified and generalized it a bit
more and turned it into a Python egg called
columnize.
I also ported it to Ruby and made Ruby package or gem. A couple of
people/companies have asked about redistributing this and have asked for a
more MIT-like and less GPL-like license. I guess they will bundle this with
their proprietary code.

For the Python package, I used PSL Version 2, but is this legitimate?
code.google.com doesn't seem to recognize in their list of licenses so that
suggests what I did might not be the right thing. (So instead,
code.google.com I selected the MIT License.)

Can a MIT License be used for code extracted from Python's standard library?
Other comments or suggestions?

Thanks.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[Rocky Bernstein]

Now that there is a package mechanism (are package mechanisms?) like
zipimporter that bundle source code into a single file, should the
notion of a "file" location should be adjusted to include the package
and/or importer?

Is there a standard API or routine which can extract this information
given a code object?

A use case here I am thinking of here is in a stack trace or a
debugger, or a tool which wants to show in great detail information
from a code object possibly via a frame. For example does this come
from a zipped egg? And if so, which one?

For concreteness, here is what I did and here's what I saw.  Select
one of the zipimporter eggs at http://code.google.com/p/pytracer and
install one of these.

I did this on GNU/Linux and Python 2.5 and I look at the co_filename
of one of the methods:

>>> import tracer
>>> tracer.__dict__['size'].func_code.co_filename
'build/bdist.linux-i686/egg/tracer.py'

But there is no file called "build/bdist.linux-686/egg/tracer.py" in
the filesystem. Instead there is a member "tracer.py" inside
/usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg'.

It's possible I caused this egg to get built incorrectly or that
setuptools has a bug which entered that misleading information.
However, shouldn't there be a standard way to untangle package
location, loader and member inside the package?

As best as I can tell, PEP 302 which discussed importer hooks and
suggests a standard way to get file data. But it doesn't address a
standard way to get container package and/or loader information.

Also I'm not sure there *is* a standard print string way to show
member inside a package. zipimporter may insert co_filename strings
like:

  /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/tracer.py

but the trouble with this is that it means file routines have to scan
the path and notice say that
/usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg is a *file*,
not a directory. And a file stat/reading routine needs to understand
what kind of packager that is in order to get tracer.py information.

(Are there any file routines in place for doing this?)

Thanks.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should execv() call _run_exitfuncs()? If not, should _run_exitfuncs() be private?

[Rocky Bernstein]

As a hobby I've been writing a debugger. One of the
commands,"restart", works by calling an execv(). You may need to do
this when
the program you are debugging is threaded or when one needs to ensure
that all program state is reinitialized.

Recently, I added remote debugging via TCP sockets and noticed that
execv() in Python doesn't arrange exit hooks to get called. Should it?

One can run _run_exitfuncs() from the atexit module, but the leading
underscore of the function call suggests it is private. Should it be?

Thanks
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should there be a source-code checksum in module objects?

[Rocky Bernstein]

As I've mentioned, I've been re-examining from ground up the whole
state of affairs in writing a debugger.

One of the challenges of a debugger or any source-code analysis tool
is verifying that the source-code that the tool is reporting on
corresponds to the compiled object under execution. (For debuggers,
this problem becomes more likely to occur when you are debugging on a
computer that isn't the same as the computer where the code is
running.)

Is there a checksum of the source text computed and stored in
compilation? If not, should there be one?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com