Advanced ways to get object information from within python
Hello,
I would like to significantly increase my abilities to find the information
I am seeking about any Python object I am using from within Python. I find
this to be a really essential skill set. After reading documentation, it
really helps to get under the hood at the command line and start testing
your own competence by examining all the methods and classes, and their
arguments and return types and so on.
I was hoping someone could help me fill in more details about what I
currently know.
I'd like to use Scrapy as an example, since it's a library I'm currently
learning.
import scrapy
I assume I'll start with "dir", as it's the most convenient.
dir(scrapy) shows this:
['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
'__all__', '__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__path__', '__spec__',
'__version__', '_txv', 'exceptions', 'http', 'item', 'link',
'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
'utils', 'version_info']
I wish there was a convenient way for me to know what all of these are. I
understand "dir" shows everything from the namespace - so that includes
methods which are present only because they are imported from other
modules, by this module.
Let's assume at minimum I know that I should be able to call all these
"attributes" (I believe this is what Python calls them - an attribute can
be anything, a method, a variable, etc. But then, how to distinguish
between this general notion of an "attribute" vs a specific attribute of a
class? Or is that called a "property" or something?)
I can confirm that every single name in the above list works when I call it
from scrapy, like this:
>>> scrapy.Field
>>> scrapy.utils
But I can't conveniently iterate over all of these to see all their types,
because dir() returns a list of strings. How can I iterate over all
attributes?
I can't use "getattr" because that requires you to enter the name of what
you're looking for. I would like to spit out all attributes with their
types, so I can know what my options are in more detail than dir() provides.
This is basically a dead-end for me until someone can illuminate this
strategy I'm pursuing, so now I'd like to focus on inspect and help.
inspect.getmembers is useful in principle, but I find the results to give
information overload.
This is just an excerpt of what it returns:
pprint.pprint(inspect.getmembers(scrapy))
[('Field', ),
('Selector', ),
('Spider', ),
('__all__',
['__version__',
'version_info',
'twisted_version',
'Spider',
Why does it just list the name and type for some classes, but for others
goes on to a sublist? __all__ does not list any type in adjacent angular
brackets, it just goes on to list some attributes without any information
about what they are. Can I suppress sublists from being printed with
inspect.getmethods? Or can I recursively require sublists also display
their type?
Lastly, the "help" function.
I find "help" to similarly be a situation of information overload. Again,
it starts with a list of "package contents". I'm not sure I see the use of
this long list of names, without much description of what they are. Next,
it lists "classes", but I don't understand:
builtins.dict(builtins.object)
scrapy.item.Field
parsel.selector.Selector(builtins.object)
scrapy.selector.unified.Selector(parsel.selector.Selector,
scrapy.utils.trackref.object_ref)
What determines the order of these classes - the order in which they appear
in the source code? What about the indentation? builtins.dict() is a Python
builtin. Then why is it listed inside of Scrapy's "help" - are all builtins
necessarily listed inside a class or just the builtins it specifically
imported or inherited?
My best guess is the most indented lines are what is actually written in
the class, the lines above are just listing the inheritance? So
scrapy.item.Field inherits the Python dictionary class, and it does this
because that way you can treat the class like a dictionary sometimes, using
dictionary methods and so on?
class Field(builtins.dict)
| Container of field metadata
|
| Method resolution order:
| Field
| builtins.dict
| builtins.object
|
| Data descriptors defined here:
What are data descriptors?
I understand IDE's tend to just print the docstring of a method as a sort
of overlay while you are writing with it, but I'm not able to use the
__docstring__ variable well - scrapy.__docstring__,
scrapy.Spider.__docstring__, and so on, return "object has no attribute
__docstring__".
I'm really fond of inspect.getsource(), all else failing, though - that's
very clear and insightful.
There's more to learn but that's enough questions for now. I'd really
appreciate anybody helping me find effective ways of investigating modules,
classes and methods from the command line.
Thanks very much,
Julius
--
https://mail.python.o
Can Python call and use FME modules and functions such as StreamOrderCalculator?
Can we do something like import an fme.something and make use of FME modules and functions? Regards, David -- https://mail.python.org/mailman/listinfo/python-list
Re: Advanced ways to get object information from within python
Julius Hamilton wrote at 2021-12-23 12:01 +0100: >I would like to significantly increase my abilities to find the information >I am seeking about any Python object I am using from within Python. I find >this to be a really essential skill set. After reading documentation, it >really helps to get under the hood at the command line and start testing >your own competence by examining all the methods and classes, and their >arguments and return types and so on. > >I was hoping someone could help me fill in more details about what I >currently know. Look at the `inspect` module. -- https://mail.python.org/mailman/listinfo/python-list
Sir jupyter note book is not working on my laptop?
Sent from [1]Mail for Windows References Visible links 1. https://go.microsoft.com/fwlink/?LinkId=550986 -- https://mail.python.org/mailman/listinfo/python-list
Re: Advanced ways to get object information from within python
Julius Hamilton wrote: > dir(scrapy) shows this: > > ['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider', > '__all__', '__builtins__', '__cached__', '__doc__', '__file__', > '__loader__', '__name__', '__package__', '__path__', '__spec__', > '__version__', '_txv', 'exceptions', 'http', 'item', 'link', > 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version', > 'utils', 'version_info'] > > I wish there was a convenient way for me to know what all of these are. ['%s: %s' % (x, type(getattr(scrapy, x))) for x in dir(scrapy)] -- https://mail.python.org/mailman/listinfo/python-list
Re: Sir jupyter note book is not working on my laptop?
Hi, On Thu, Dec 23, 2021 at 1:42 PM IMTIAZ AHMAD wrote: > > Well, my program just crashed. Can you help? Thank you. > > > >Sent from [1]Mail for Windows > > > > References > >Visible links >1. https://go.microsoft.com/fwlink/?LinkId=550986 > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
email.message_from_file & quoted printable
I have a bunch of old email archives I'm messing around with, for example,
rendering them on-demand in HTML. Some of the files use quoted printable
content transfer encoding. Here's one (with a number of headers elided):
>From classicrendezvous-admin Mon Dec 4 15:29:22 2000
Message-ID: <027801bfcdaa$fa175230$268cbcd4@stewart>
Reply-To: "Stewart Barrie"
From: "Stewart Barrie"
To:
Date: Sat, 3 Jun 2000 23:28:00 +0100
MIME-Version: 1.0
X-StripMime: Non-text section removed by stripmime
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset="iso-8859-1"
Subject: {ClassicRend]Frame dent
Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it = seems
fine.=20
Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?
Cheers
Stewart
--- StripMime Report -- processed MIME parts ---
multipart/alternative
text/plain (text body -- kept)
text/html
---
This message is stored in a file. I read it using email.message_from_file.
In the example above, the payload still retains the quoted printable bits.
Maybe it's not correctly encoded (no "=20" at the end of the second
paragraph, for example), but I would have thought the email package would
do what it could to decode things, but nope:
>>> print(msg.get_payload())
Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it =
seems fine.=20
Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?
Cheers
Stewart
--- StripMime Report -- processed MIME parts ---
multipart/alternative
text/plain (text body -- kept)
text/html
---
Am I expecting too much from the email package when munching on crufty
20+yo archived email messages?
Thx,
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can Python call and use FME modules and functions such as StreamOrderCalculator?
On 12/23/21 09:38, Shaozhong SHI wrote: > Can we do something like import an fme.something and make use of FME > modules and functions? And what, pray tell, is FME? -- https://mail.python.org/mailman/listinfo/python-list
Re: email.message_from_file & quoted printable
On 23Dec2021 14:50, Skip Montanaro wrote: >I have a bunch of old email archives I'm messing around with, for example, >rendering them on-demand in HTML. Some of the files use quoted printable >content transfer encoding. Here's one (with a number of headers elided): [...] >This message is stored in a file. I read it using >email.message_from_file. >In the example above, the payload still retains the quoted printable bits. >Maybe it's not correctly encoded (no "=20" at the end of the second >paragraph, for example) That seems legit, as it is a space in the message text. >, but I would have thought the email package would >do what it could to decode things, but nope: > print(msg.get_payload()) From the docs: get_payload(i=None, decode=False) Return the current payload, which will be a list of Message objects when is_multipart() is True, or a string when is_multipart() is False. [...] Optional decode is a flag indicating whether the payload should be decoded or not, according to the Content-Transfer-Encoding header. When True and the message is not a multipart, the payload will be decoded if this header’s value is quoted-printable or base64. Try decode=True. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Advanced ways to get object information from within python
On 23/12/2021 11:01, Julius Hamilton wrote: > Lastly, the "help" function. > > I find "help" to similarly be a situation of information overload. I assume you know that you can target help() to the specific attribute or function you need not just the top level classes? So combined with dir() you can call help on each of the names that dir() reveals. That usually produces a much more focused form of documentation. So in your example: > dir(scrapy) shows this: >['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_txv', 'exceptions', 'http', 'item', 'link', 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version', 'utils', 'version_info'] > I wish there was a convenient way for me to know what > all of these are. help(scrapy.http) help(scrapy.spiders) etc... And if it turns out they are not functions or classes you can use [p]print to get the values. You can also use type() to clarify what kind of thing an attribute is. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos -- https://mail.python.org/mailman/listinfo/python-list
Re: email.message_from_file & quoted printable
> > From the docs: > > get_payload(i=None, decode=False) ... Try decode=True. :dopeslap: Thanks. Never been all that consistent reading documentation. Skip -- https://mail.python.org/mailman/listinfo/python-list
