Re: ESR "Waning of Python" post

[Peter J. Holzer]

On 2018-10-12 14:07:56 -0500, Tim Daneliuk wrote:
> On 10/11/2018 12:15 AM, Gregory Ewing wrote:
> > But it's not like that at all. As far as I know, all the
> > attempts that have been made so far to remove the GIL have
> > led to performance that was less than satisfactory. It's a
> > hard problem that we haven't found a good solution to yet.
> 
> Do you happen to have a reference that explains what the issues are
> for GIL removal?

I'm certainly not an expert on CPython internals, but what I've gathered
from talks and discussions on the topic is that the CPython interpreter
accesses shared state a lot (the reference count fields are an obvious
example, but there are others), so to remove the GIL you would have to
replace it with a myriad of small locks which are taken and released all
the time - this adds a lot of overhead compared to a single lock which
is basically always taken and just released before blocking syscalls. 

(If you ask your favourite search engine for "gilectomy", you'll
probably find lots of info)

It might be better to start from scratch: Design a new VM suitable for
Python which can run mostly without locks. But this is of course a lot
of work with no guarantee of success. (JPython and IronPython did
something similar, although they didn't design new VMs, but reused VMs
designed for other languages).

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python indentation (3 spaces)

[Peter J. Holzer]

On 2018-10-08 20:13:38 +, Grant Edwards wrote:
> On 2018-10-08, Peter J. Holzer  wrote:
> > Theoretically I would agree with you: Just use a single tab per
> > indentation level and let the user decide whether that's displayed
> > as 2, 3, 4, or 8 spaces or 57 pixels or whatever.
> >
> > In practice it doesn't work in my experience.
> 
> Nor in mine.  On problem is that under Unix is that there isn't just
> one place where you would need to configure the tab width.  There are
> dozens of text-processing tools that get used on all sorts of text,
> including program sources in a variety of languages.
> 
> For "just use tabs" to work, all of those tools would have to
> magically recognize that they're looking at Python source and adjust
> the tab size accordingly.  That isn't going to happen.

Well, no. The idea of "just use tabs" isn't have a different tab width
per language, but a different tab width per user. So I - being a fan of
fixed-width fonts - would set the tab width to 5 spaces (just to pick a
somewhat unusual number), and Python, Perl, C, TypeScript, HTML, CSS,
etc. code containing tabs would be displayed and printed with that tab
width. You set your tab width to half an inch. And all your text files
are displayed and printed with that tab width. Gene uses 3 spaces, etc.

There aren't that many tools which would have to be configured. For me
that would be vim (the text editor), the tty driver and/or the terminal
emulator (not sure what the right place is), less (the pager), and paps
(for printing). Maybe a few others, but the vast majority of tools
doesn't care about the tab width. For them a tab is just a single byte.

The problem is in my experience that a "use only tabs" policy is hard to
enforce. People will type spaces when they should use tabs (and use tabs
when they should use spaces), their software will autoconvert between
spaces and tabs (with a different number of spaces per tab for each
user), etc. The result is an awful mess. "Use only spaces" is much
easier to enforce: Just reject any file (except binaries and makefiles)
with a 0x09 byte at commit time.

(Now that I think of it, "use tabs" might work better in Python than in
other languages because in Python indentation has syntactic meaning. If
your editor replaces some tabs with spaces or vice versa, your program
won't compile any more and you will (hopefully) fix it before committing
it.)

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python indentation (3 spaces)

[Peter J. Holzer]

On 2018-10-09 09:55:34 +0200, Antoon Pardon wrote:
> On 08-10-18 19:43, Peter J. Holzer wrote:
> > On 2018-10-08 10:36:21 +1100, Chris Angelico wrote:
> >> How wide my indents are on my screen shouldn't influence your screen
> >> or your choices.
> > Theoretically I would agree with you: Just use a single tab per
> > indentation level and let the user decide whether that's displayed as 2,
> > 3, 4, or 8 spaces or 57 pixels or whatever. 
> >
> > In practice it doesn't work in my experience. There is always someone in
> > a team who was "just testing that new editor" and replaced all tabs
> > with spaces (or vice versa) or - worse - just some of them.
> 
> Isn't that caugth in the process of commiting to version control?

Tabs are easy to catch. If a file contains a tab, reject it.

Spaces aren't, because spaces are everywhere. So you would have to check
whether a particular run of spaces should really be a tab, which is a
much harder problem. At the very least you will have to parse enough of
the language to find multi-line strings and indented blocks.

So it's easier to mandate "no tabs" than "only tabs for indentation".
Especially if you are dealing with more than one language. (I think
Python would be relatively easy. Perl is somewhere between hard and
impossible, but there is a library which can handle most
non-pathological code, C is somewhere between, etc. But the point is
that you would need a different checking tool for every language.)

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[jfine2358]

On Friday, October 12, 2018 at 8:41:12 PM UTC+1, Paul Rubin wrote:

> 1) If you keep the existing refcount mechanism, you have to put locks
> around all the refcounts, which kills performance since refcounts are
> updated all the time.

I think BUFFERED multi-core reference count garbage collection is possible. If 
so, then locks are not needed. I explain this in this thread:

[Python-ideas] Multi-core reference count garbage collection
https://groups.google.com/forum/#!topic/python-ideas/xRPdu3ZGeuk
https://mail.python.org/pipermail/python-ideas/2018-July/052054.html

-- 
Jonathan
-- 
https://mail.python.org/mailman/listinfo/python-list

socket: Too many open files

[Shakti Kumar]

Hello,
I’m running a script which basically does a traceroute to the list of hosts
provided, and then pulls up some info by logging in to gateways in the path.
I am running this script for a list of almost 40k hosts in our data centers.
Also, I am using commands module to get the traceroute output.

out = commands.getstatusoutput('traceroute ' + ip)

However I observe that this particular line is failing with socket error
after I reach some 5k to 6k hosts.
I know commands module is using pipes to execute the given command and this
is one reason for exhaustion of file descriptors.
Any suggestions for improving this and getting a workaround?

Thanks,
Shakti
-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: socket: Too many open files

[Shakti Kumar]

>Hello,
>I’m running a script which basically does a traceroute to the list of
hosts provided, and then pulls up some info by logging in to gateways in
the path.
>I am running this script for a list of almost 40k hosts in our data
centers.
>Also, I am using commands module to get the traceroute output.

>out = commands.getstatusoutput('traceroute ' + ip)

>However I observe that this particular line is failing with socket error
after I reach some 5k to 6k hosts.
>I know commands module is using pipes to execute the given command and
this is one reason for exhaustion of file descriptors.
>Any suggestions for improving this and getting a workaround?

>Thanks,
>Shakti

I dont understand how I should close the socket created by commands module
after I do traceroute to one host.
Also, if I’m not wrong, isn’t commands module supposed to do that on its
own?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[Marko Rauhamaa]

dieter :
> Marko Rauhamaa  writes:
>> However, I challenge the notion that creating hundreds of thousands of
>> temporary objects is stupid. I suspect that the root cause of the
>> lengthy pauses is that the program maintains millions of *nongarbage*
>> objects in RAM (a cache, maybe?).
>
> Definitely. The application concerned was a long running web application;
> caching was an important feature to speed up its typical use cases.

As an optimization technique, I suggest turning the cache into a "binary
blob" opaque to GC, or using some external component like SQLite.
Keeping the number of long-term objects low is key.

Note that Python creates a temporary object every time you invoke a
method. CPython removes them quickly through reference counting, but
other Python implementations just let GC deal with them, and that's
generally ok.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ESR "Waning of Python" post

[Marko Rauhamaa]

Paul Rubin :
> Note that Java has a lot of [GC] options to choose from:
> https://docs.oracle.com/javase/9/gctuning/available-collectors.htm

I'm all for GC, but Java's GC tuning options are the strongest
counter-argument against it. The options just shift the blame from the
programming language to the operator of the software.

For GC to be acceptable, you shouldn't ever have to tune it. And I've
seen it in action. A customer complains about bad performance. The
system engineer makes a tailored GC recipe to address the issue, which
may help for a short while.

Here's my rule of thumb. Calculate how much memory you need for
long-term objects. Don't let the application exceed that amount.
Multiply the amount by 10 and allocate that much RAM for your
application.

> Another approach is Erlang's, where the application is split into a
> lot of small lightweight processes, each of which has its own GC and
> heap. So while some of them are GC'ing, the rest can keep running.

So the lightweight processes don't share any data. That may be a fine
approach.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Overwhelmed by the Simplicity of Python. Any Recommendation?

[Alister via Python-list]

On Fri, 12 Oct 2018 09:12:03 -0700, Rob Gaddi wrote:

> On 10/11/2018 11:29 PM, Kaan Taze wrote:
>> Hi everyone,
>> 
>> Since this is my first post to mail-list I'm kind of hesitant to ask
>> this question here but as many of you spend years working with Python
>> maybe some of you can guide me.
>> 
>> What I trouble with is not a logical error that exist on a program I
>> wrote.
>> It's the Python itself. Well, I'm 22 years old CS student -from Turkey-
>> and what they showed us at university was C Language and Java but I
>> mainly use C in school projects etc. So it's been few months that I
>> started to use Python for my personal side-projects. There are lots of
>> resources to learn language. I do what I need to do with Python too but
>> I was kinda shocked when I solve Python questions at Hackerrank. Even
>> with list comprehensions you can implement in very smart way to get
>> things done and easy. Iterations, string operations. The codes I see on
>> the Internet using basics in a very clever way which I couldn't come up
>> with the same solution if I tried to for some time. I do understand
>> this ways but coming from ANSI C makes it hard to see this flexibility.
>> I probably do things in a both inefficient and hard way in my projects.
>> 
>> How do I get used to this? Is this just another "practice, practice,
>> practice" situation? Anything you can recommend?
>> 
>> 
>> All the best.
>> 
>> Kaan.
>> 
>> 
> A) Yes, it's practice practice practice.
> 
> B) Don't get hung up on finding the clever solution.  Comprehensions and
> generators and lots of other things are great under some circumstances
> for making the code clearer and easier to read, but they too can become
> the hammer that makes everything look like a nail.  The most important
> thing is that your code is logical, clean, and easy to understand.  If
> it doesn't take full advantage of the language features, or if the
> performance isn't optimized to within an inch of its life, well so be
> it.

To re-enforce this someone much wiser than me wrote (& I a paraphrasing a 
bit)

It takes more skill to debug code than to write it so if you write code 
at the limits of your ability yo do not have the skills needed to debug 
it.

KISS - Keep It Simple Stupid



-- 
Even if you do learn to speak correct English, whom are you going to speak
it to?
-- Clarence Darrow
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Overwhelmed by the Simplicity of Python. Any Recommendation?

[Abdur-Rahmaan Janhangeer]

become wiser in python

me i came from c/java and was doing

for i in range(0, len(list)):
# get list item by index

instead of

for item in list:


XD

well the more you are exposed to py, the better you knoe hoe things work.

reading source of popular projects is really great, and ... read the docs,
ask your doubts on the mailing list, not to be afraid to read technical
stuffs about py etc oh and ... do some projects (500+ lines) in py

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius
-- 
https://mail.python.org/mailman/listinfo/python-list

Is this mailbox manipulation working by luck, or can't I understand my own code?

[Chris Green]

I use a Python script (called directly by '| ' in
.forward) which routes incoming mail to various mailboxes according to
the mailing list it's from (plus a few other criteria).  The first
lines of the program are:-

#!/usr/bin/python 
# 
# 
# Mail filtering script 
# 
import email
import mailbox
import os
import sys
import time
import mailLib
import shlex
# 
# 
# Redirect any exceptions to a file 
# 
sys.stderr = open("/home/chris/tmp/mail.err", 'a')
# 
# 
# Some constants (i.e. configuration) 
# 
home = "/home/chris"
logfile = home + "/tmp/mail.log"
filtfile = home + "/.mutt/filter"
mldir = home + "/Mail/"
indir = mldir + "In/"
judir = mldir + "Ju/"
# 
# 
# Set to log to mail.log in ~/tmp with name 'filter' and the
envelope/from 
# 
log = mailLib.initLog("filter")
# 
# 
# Initialise destination mailbox name to empty 
# 
dest = ""
# 
# 
# Read the message from standard input and make a message object from
it 
# 
# msg = email.message_from_string(sys.stdin.read()) 
msg = mailbox.mboxMessage(sys.stdin.read())
# 
# 
# Extract the To:, Cc: and Subject: headers and the envelope/from 
# 
msgcc = msg.get("Cc", "unknown").lower()
msgto = msg.get("To", "unknown").lower()
msgsb = msg.get("Subject", "unknown")
msgfm = msg.get("From", "unknown").lower()

I then do various things according to the values in msgcc, msgto,
msgsb and msgfm.  The program works well and I've been using it for
quite a few years.

Now I want to do some minor changes and I can't for the life of me see
how those lines with msg.get(... in them work.  The mailbox.mboxMessage()
class doesn't have a get() method and the get() method for the parent
class mailbox() doesn't seem to do what it appears to be doing here.

Can anyone suggest how this might be working?  What will those
msg.get() calls return?  Will it simply be the whole message (there's
always only one message fed into the program from .froward) or is
there something going on that means it does work as I intended by
getting the content of the individual header lines.

Looking at the rest of the code it *might* work OK even if all of
msgcc, msgto, msgsb and msgfm contain the whole message.  It just
wouldn't be working quite as I intended! :-)

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: socket: Too many open files

[jfine2358]

Hi Shakti

You wrote:

> out = commands.getstatusoutput('traceroute ' + ip)

The page

https://docs.python.org/3/library/subprocess.html#legacy-shell-invocation-functions

describes subprocess.getstatusoutput as one of the "legacy functions from the 
2.x commands module. These operations implicitly invoke the system shell and 
none of the guarantees described above regarding security and exception 
handling consistency are valid for these functions."

I suggest you use subprocess.run or Popen.communicate instead. Once you've done 
that, perhaps your problem will go away.

And even if it does not, I think you're better placed for getting support, when 
you're not using legacy commands.

I hope this helps, and good luck.

-- 
Jonathan


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[Chris Green]

Stefan Ram  wrote:
> Chris Green  writes:
> >msg.get
> 
>   You can get some information about »get«:
> 
> print( "msg.get.__doc__ =", msg.get.__doc__ )
> print( "msg.get.__func__ =", msg.get.__func__ )
> print( "msg.get.__self__ =", msg.get.__self__ )
> print( "msg.get.__str__() =", msg.get.__str__() )
> print( "msg.get.__repr__() =", msg.get.__repr__() )
> print( "msg.get.__name__ =", msg.get.__name__ )
> import dis
> dis.dis( msg.get )
> dis.show_code( msg.get )
> 
Thanks, yes, this tells me a little:-

('msg.get.__doc__ =', 'Get a header value.\n\nLike
__getitem__() but return failobj instead of None when the field\n
 is missing.\n')

However it isn't mentioned *anywhere* in the documentation that I can
see.  There's only __getitem__() and get() for the mailbox class which
operate on a whole mailbox and return a whole message.`

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[Peter J. Holzer]

On 2018-10-13 17:28:27 +0100, Chris Green wrote:
> ('msg.get.__doc__ =', 'Get a header value.\n\nLike
> __getitem__() but return failobj instead of None when the field\n
>  is missing.\n')
> 
> However it isn't mentioned *anywhere* in the documentation that I can
> see.  There's only __getitem__() and get() for the mailbox class which
> operate on a whole mailbox and return a whole message.`

The documentation for mailbox.mboxMessage refers to mailbox.Message,
which refers to email.message.Message, which documents the get method:

https://docs.python.org/3/library/email.compat32-message.html#email.message.Message.get

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[Thomas Jollans]

On 13/10/2018 18:28, Chris Green wrote:

Stefan Ram  wrote:

Chris Green  writes:

msg.get


   You can get some information about »get«:

print( "msg.get.__doc__ =", msg.get.__doc__ )
print( "msg.get.__func__ =", msg.get.__func__ )
print( "msg.get.__self__ =", msg.get.__self__ )
print( "msg.get.__str__() =", msg.get.__str__() )
print( "msg.get.__repr__() =", msg.get.__repr__() )
print( "msg.get.__name__ =", msg.get.__name__ )
import dis
dis.dis( msg.get )
dis.show_code( msg.get )


Thanks, yes, this tells me a little:-

 ('msg.get.__doc__ =', 'Get a header value.\n\nLike
 __getitem__() but return failobj instead of None when the field\n
  is missing.\n')

However it isn't mentioned *anywhere* in the documentation that I can
see.  There's only __getitem__() and get() for the mailbox class which
operate on a whole mailbox and return a whole message.`



https://docs.python.org/3.7/library/mailbox.html#message-objects does 
say that all the different message types are subclasses of 
email.message.Message.


... and these are the docs you're looking for:

https://docs.python.org/3.7/library/email.compat32-message.html#email.message.Message.get
--
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[Chris Green]

Chris Green  wrote:
> Stefan Ram  wrote:
> > Chris Green  writes:
> > >msg.get
> > 
> >   You can get some information about »get«:
> > 
> > print( "msg.get.__doc__ =", msg.get.__doc__ )
> > print( "msg.get.__func__ =", msg.get.__func__ )
> > print( "msg.get.__self__ =", msg.get.__self__ )
> > print( "msg.get.__str__() =", msg.get.__str__() )
> > print( "msg.get.__repr__() =", msg.get.__repr__() )
> > print( "msg.get.__name__ =", msg.get.__name__ )
> > import dis
> > dis.dis( msg.get )
> > dis.show_code( msg.get )
> > 
> Thanks, yes, this tells me a little:-
> 
> ('msg.get.__doc__ =', 'Get a header value.\n\nLike
> __getitem__() but return failobj instead of None when the field\n
>  is missing.\n')
> 
> However it isn't mentioned *anywhere* in the documentation that I can
> see.  There's only __getitem__() and get() for the mailbox class which
> operate on a whole mailbox and return a whole message.`
> 
I've worked it out, sort of, by looking at the actual Python code.

The mailbox.mboxMessage() class is a sub-class of email.message and
*that* has the get() function I'm calling.  It does tell you this
in the documentation but it's not immediately obvious.  In particular
one isn't really expecting an object in the mailbox class to be a
sub-class of a different 'parent' class, or at least I wasn't! :-)

-- 
Chris Green
·
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[Roel Schroeven]

Chris Green schreef op 13/10/2018 om 17:15:

I use a Python script (called directly by '| ' in
.forward) which routes incoming mail to various mailboxes according to
the mailing list it's from (plus a few other criteria).  The first
lines of the program are:-

> ...

 msg = mailbox.mboxMessage(sys.stdin.read())
 #
 #
 # Extract the To:, Cc: and Subject: headers and the envelope/from
 #
 msgcc = msg.get("Cc", "unknown").lower()
 msgto = msg.get("To", "unknown").lower()
 msgsb = msg.get("Subject", "unknown")
 msgfm = msg.get("From", "unknown").lower()



Can anyone suggest how this might be working?  What will those
msg.get() calls return?


I think your question is answered by this quote from the documentation 
(https://docs.python.org/3/library/email.message.html):


"*The conceptual model provided by an EmailMessage object is that of an 
ordered dictionary of headers coupled with a payload that represents the 
RFC 5322 body of the message, which might be a list of sub-EmailMessage 
objects*. *In addition to the normal dictionary methods for accessing 
the header names and values*, there are methods for accessing 
specialized information from the headers (for example the MIME content 
type), for operating on the payload, for generating a serialized version 
of the message, and for recursively walking over the object tree.


The EmailMessage dictionary-like interface is indexed by the header 
names, which must be ASCII values. The values of the dictionary are 
strings with some extra methods. Headers are stored and returned in 
case-preserving form, but field names are matched case-insensitively. 
Unlike a real dict, there is an ordering to the keys, and there can be 
duplicate keys. Additional methods are provided for working with headers 
that have duplicate keys."


I.e. email.message instances provide the get() method because it is one 
of the "normal dictionary methods" you can use to access the headers.


More specifically, get() is documented as 
(https://docs.python.org/3/library/email.message.html#email.message.EmailMessage.get): 

"Return the value of the named header field. This is identical to 
__getitem__() except that optional failobj is returned if the named 
header is missing (failobj defaults to None)."


Watch out for headers with duplicate keys (like multiple Received: 
headers): use get_all() for those 
(https://docs.python.org/3/library/email.message.html#email.message.EmailMessage.get_all).


--
"Honest criticism is hard to take, particularly from a relative, a
friend, an acquaintance, or a stranger."
-- Franklin P. Jones

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: Is this mailbox manipulation working by luck, or can't I understand my own code?

[MRAB]

On 2018-10-13 16:15, Chris Green wrote:

I use a Python script (called directly by '| ' in
.forward) which routes incoming mail to various mailboxes according to
the mailing list it's from (plus a few other criteria).  The first
lines of the program are:-

 #!/usr/bin/python
 #
 #
 # Mail filtering script
 #
 import email
 import mailbox
 import os
 import sys
 import time
 import mailLib
 import shlex
 #
 #
 # Redirect any exceptions to a file
 #
 sys.stderr = open("/home/chris/tmp/mail.err", 'a')
 #
 #
 # Some constants (i.e. configuration)
 #
 home = "/home/chris"
 logfile = home + "/tmp/mail.log"
 filtfile = home + "/.mutt/filter"
 mldir = home + "/Mail/"
 indir = mldir + "In/"
 judir = mldir + "Ju/"
 #
 #
 # Set to log to mail.log in ~/tmp with name 'filter' and the
 envelope/from
 #
 log = mailLib.initLog("filter")
 #
 #
 # Initialise destination mailbox name to empty
 #
 dest = ""
 #
 #
 # Read the message from standard input and make a message object from
 it
 #
 # msg = email.message_from_string(sys.stdin.read())
 msg = mailbox.mboxMessage(sys.stdin.read())
 #
 #
 # Extract the To:, Cc: and Subject: headers and the envelope/from
 #
 msgcc = msg.get("Cc", "unknown").lower()
 msgto = msg.get("To", "unknown").lower()
 msgsb = msg.get("Subject", "unknown")
 msgfm = msg.get("From", "unknown").lower()

I then do various things according to the values in msgcc, msgto,
msgsb and msgfm.  The program works well and I've been using it for
quite a few years.

Now I want to do some minor changes and I can't for the life of me see
how those lines with msg.get(... in them work.  The mailbox.mboxMessage()
class doesn't have a get() method and the get() method for the parent
class mailbox() doesn't seem to do what it appears to be doing here.

Can anyone suggest how this might be working?  What will those
msg.get() calls return?  Will it simply be the whole message (there's
always only one message fed into the program from .froward) or is
there something going on that means it does work as I intended by
getting the content of the individual header lines.

Looking at the rest of the code it *might* work OK even if all of
msgcc, msgto, msgsb and msgfm contain the whole message.  It just
wouldn't be working quite as I intended! :-)

'mailbox.mboxMessage' is class 'mboxMessage' in file 'mailbox.py' in the 
stdlib.


In file 'mailbox.py' in the stdlib, class 'mboxMessage' inherits from 
'_mboxMMDFMessage', which inherits from '_mboxMMDFMessage', which 
inherits from 'Message', which inherits from 'email.message.Message' 
(class 'Message' from file 'email/message.py' in the stdlib.


Those msg.get() calls return is the contents (a string) of the relevant 
field in the message's header.


This:

msg.get("Cc", "unknown").lower()

returns the contents of the "Cc" field if it exists, or the default 
value "unknown" if it doesn't exist.


That string is then converted to lowercase.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Encounter issues to install Python

[Anthony Flury via Python-list]

Olivier,

Welcome to the list - before we can help you, we need some more 
information :


 * What Operating system are you using - Windows/Mac/Linux/Raspberry
   Pi/Android for something else ?
 * What command or installer did you use to try to install Python.
 * What issues did you have during installation - if any ?
 * What interface are you trying to access, and how are you doing that ?
 * Do you get error messages?

Unless you tell us what the problem is we can't possibly help.

On 08/10/18 20:21, Olivier Oussou via Python-list wrote:

Hi!I downloaded and installed python 3.6.4 (32-bit) on my computer but I have 
problems and can not access the python interface.
I need your technical assistance to solve this matter.

Best regard!

Olivier OUSSOUMedical entomologist, Benin


--
Anthony Flury
*Email* : anthony.fl...@btinternet.com 
*Twitter* : @TonyFlury 
--
https://mail.python.org/mailman/listinfo/python-list

Re: snakify issues

[bob gailer]

5:50 AM Dec 8, 2016 a post was made to this list - subject "Snakify - 
free introductory Python online course with exercises"


Recently I was engaged by a student seeking help with some of the 
exercises. I found a number of issues at the snakify web site. Thus 
began a conversation between me and the Site Maintainer Here is the latest:


On 9/28/2018 1:02 PM, Vitaly Pavlenko wrote:

Hi Bob,

Thanks for your email. I’d be very happy if you could describe what 
sort of issues were you running into.
I drafted a rather lengthy reply which I sent to you on October 1. I 
have not received any response.


Did you get it? Was it helpful? I don't see any changes of the website. 
Please let me know.


As I continue to assist my student I run into more and more issues. I 
will be glad to send them after I hear from you. Examples:

- misspelled words
- unclear statements
- information missing from tracebacks
- no way to interrupt an infinite loop.

My worst fear is that the list was overwhelming or seen as trivia or 
nit-pickingl. I have had other publishers refuse to talk to me after I 
had pointed out legitimate concerns. I don't understand that, but it 
happens.


Bob Gailer
--
https://mail.python.org/mailman/listinfo/python-list

Re: snakify issues

[Ben Bacarisse]

bob gailer  writes:

> 5:50 AM Dec 8, 2016 a post was made to this list - subject "Snakify - 
> free introductory Python online course with exercises"
>
> Recently I was engaged by a student seeking help with some of the
> exercises. I found a number of issues at the snakify web site. Thus
> began a conversation between me and the Site Maintainer Here is the
> latest:
>
> On 9/28/2018 1:02 PM, Vitaly Pavlenko wrote:
>> Hi Bob,
>>
>> Thanks for your email. I’d be very happy if you could describe what
>> sort of issues were you running into.
> I drafted a rather lengthy reply which I sent to you on October 1. I
> have not received any response.

I looked at the "landing page" and they clearly need a proof-reader.
It's intelligible, but the English is clunky.  In the body of the course
there's quite a lot of incorrect terminology.

But the material itself is so dull -- numbers, arithmetic, more numbers,
factorials, more arithmetic, numbers...  And when a string exercise came
up (capitalise a word) the hint was to use ord and chr.

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

[RELEASE] Python 3.7.1rc2 and 3.6.7rc2 now available for testing

[Ned Deily]

Python 3.7.1rc2 and 3.6.7rc2 are now available. 3.7.1rc2 is a release
preview of the first maintenance release of Python 3.7, the latest
feature release of Python. 3.6.7rc2 is a release preview of the next
maintenance release of Python 3.6, the previous feature release of
Python. Assuming no further critical problems are found prior to
2018-10-20, no code changes are planned between these release
candidates and the final releases. These release candidates are
intended to give you the opportunity to test the new security and bug
fixes in 3.7.1 and 3.6.7. We strongly encourage you to test your
projects and report issues found to bugs.python.org as soon as
possible. Please keep in mind that these are preview releases and,
thus, their use is not recommended for production environments.

You can find these releases and more information here:
https://www.python.org/downloads/release/python-371rc2/
https://www.python.org/downloads/release/python-367rc2/

--
  Ned Deily
  n...@python.org -- []

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: socket: Too many open files

[Cameron Simpson]

On 13Oct2018 14:10, Shakti Kumar  wrote:
I’m running a script which basically does a traceroute to the list of 
hosts

provided, and then pulls up some info by logging in to gateways in the path.
I am running this script for a list of almost 40k hosts in our data centers.
Also, I am using commands module to get the traceroute output.

out = commands.getstatusoutput('traceroute ' + ip)

However I observe that this particular line is failing with socket error
after I reach some 5k to 6k hosts.
I know commands module is using pipes to execute the given command and this
is one reason for exhaustion of file descriptors.
Any suggestions for improving this and getting a workaround?


I'd figure out where your file descriptors are going.

Is traceroute leaving sockets littering your system? If you're on Linux 
this command:


 netstat -anp

will show you all the sockets, their state, and the pids of the 
processes which own them. Does your script cause sockets to accrue after 
the traceroutes?


If you write a trivial shell script to do the traceroutes:

 while read ip
 do
   traceroute $ip
 done The if doesn't, then traceroute may not be the problem and something 
else is leaking file descriptors.


In fact, given that it is file descriptors, maybe sockets are not what 
is leaking?


From another terminal, see what your Python programme has open when this 
happens with "lsof -n -p pid-of-python-programme". Maybe the leaks are 
pipes, or connections from your "logging in to gateways in the path" 
code. It may be as simple as you not closing files or connections.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Single DB connection during class's lifetime. Metaclass,singleton and __new__() examples and references.

[Cameron Simpson]

On 12Oct2018 13:28, Ryan Johnson  wrote:

Thanks for the clarification.

If I am creating a class variable, are you suggesting I perform the “if it 
exists, great, otherwise make it” logic in the __init__ block or in the class 
definition block? Will that even run in a class definition?


The class definition code runs when the class is defined (as Python 
reads it in your code).


The __init__ block runs once each time a new instance of the class is 
initialised.


When do you _want_ this logic to run? That dictates where the logic 
goes.


If you run this in the class definition code it pretty much will 
unconditionally make a db connection. But in reality (a) you usually 
want to defer making the connection until you need it to reduce resource 
usage and (b) you often don't know enough to make the connection at 
class definition time i.e. you don't know the database host, the 
credentials, etc - they are often supplied to the initialiser (directly 
or via some config file).


I never see examples do anything besides assignment operations and flow 
control, although it would follow that if the block allows creation of 
strings (an object), it would allow creation of connection objects. On 
the other hand, the __init__ block seems like a natural place to put 
the cursor instantiation.


You can do anything that is sensible in the __init__ block - it is just 
code. Your goal is to decide what is sensible.


Normally the initialiser mosts sets up various arrtributes to sane 
initial values. It can do complex things, but is usually relatively 
basic.


I'd note, as Thomas did, that the cursor is a control object associated 
with a query. You can have multiple cursors on a connection, and you 
often make one from a query, process the query results, then discard the 
cursor. SO you routinely use several during the lifetime of a 
connection.


Therefore you don't make cursors when you set up the connection; you 
make them in association with queries.



From this answer ( 
https://stackoverflow.com/questions/25577578/python-access-class-variable-from-instance/25577642#25577642
 ) the pythonic way to access the class variable is using type(self), but 
doesn’t this access the local class’s variable, if someone subclasses my class? 
Can I specify my class variable via the class name itself? Example: instancevar 
= ClassName.classvar .
I am hoping that because of the way python points labels to objects, this 
should give my instance an instance var that refers to the class var. Am I 
right?


_Usually_ I access class attributes (which you're calling variables, I 
believe - they're not) via the instance:


 def foo(self, blah=None):
   if blah is None:
 blah = foo.DEFAULT_BLAH_VALUE
   ... work with blah ...

As you suggest, this will find DEFAULT_BLAH_VALUE from the subclass 
before it finds it from the superclass. Usually that is what I want - 
the purpose of subclassing is to (possibly) override the aspects of the 
superclass.


However, you _can_ always reach directly to a specific class to get a 
value:


 blah = MySuperCLass.DEFAULT_BLAH_VALUE

if that is sensible. All you're doing is changing the way in which the 
name "DEFAULT_BLAH_VALUE" is found: do I use the instance's name lookup 
or go somewhere direct?


In your case with a persistent database connection associated with a 
class it would be best to make the "get a connection" logic a class 
method because the "is there a connection" attribute is associated with 
the class, not the instance.


Methods are, by default, "instance" methods: they are defined like this:

 def method(self, ...):

and you largely work through "self", being the current instance. That is 
its "context".


Class method are defined like this:

 @classmethod
 def method(cls, ...)

and instead of having an instance as context (with the conventional name 
'self"), you have the class (with the conventional name "cls"). These 
are for methods which _do_ _not_ care about the instance, just the 
class. So in the case of your database connection, made on demand once 
per class, you might go:


 @classmethod
 def conn(cls):
   c = cls.connection
   if c is None:
 c = connect_to_the_db(.)
 cls.connection = c
   return c

See that there's no "self" in here?

Then your instance methods can look like this:

 def lookup(self, ):
   conn = self.conn()
   cursor = conn.select()
   ... use the cursor to process the result ...

The instance finds the "conn" class method in the usual way: look in the 
instance, then look in the class hierarchy.


This in itself is an argument against making the connection in the 
__init__ block.


Does this help?

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 10442: character maps to

[pjmclenon]

On Wednesday, June 13, 2018 at 7:14:06 AM UTC-4, INADA Naoki wrote:
> ​> 1st is this script is from a library module online open source
> 
> If it's open source, why didn't you show the link to the soruce?
> I assume your code is this:
> 
> https://github.com/siddharth2010/String-Search/blob/6770c7a1e811a5d812e7f9f7c5c83a12e5b28877/createIndex.py
> 
> And self.collFile is opened here:
> 
> https://github.com/siddharth2010/String-Search/blob/6770c7a1e811a5d812e7f9f7c5c83a12e5b28877/createIndex.py#L91
> 
> You need to add `encoding='utf-8'` argument.



hello i used this recommandtion in one of my projects in python and it worked 
fine im wrting today cuz i have this same unicode error in a slighty differn 
file code line and i added encoding utf 8 but i still get the same error

here is my line of code

with open(join("docs", path)) as f:

where can i add the encoding="utf8" line??
does anyone on this forum happen to know??

ok thank you jessica
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 10442: character maps to

[MRAB]

On 2018-10-14 00:13, pjmcle...@gmail.com wrote:

On Wednesday, June 13, 2018 at 7:14:06 AM UTC-4, INADA Naoki wrote:

​> 1st is this script is from a library module online open source

If it's open source, why didn't you show the link to the soruce?
I assume your code is this:

https://github.com/siddharth2010/String-Search/blob/6770c7a1e811a5d812e7f9f7c5c83a12e5b28877/createIndex.py

And self.collFile is opened here:

https://github.com/siddharth2010/String-Search/blob/6770c7a1e811a5d812e7f9f7c5c83a12e5b28877/createIndex.py#L91

You need to add `encoding='utf-8'` argument.




hello i used this recommandtion in one of my projects in python and it worked 
fine im wrting today cuz i have this same unicode error in a slighty differn 
file code line and i added encoding utf 8 but i still get the same error

here is my line of code

with open(join("docs", path)) as f:

where can i add the encoding="utf8" line??
does anyone on this forum happen to know??

ok thank you jessica


with open(join("docs", path), encoding="utf-8") as f:
--
https://mail.python.org/mailman/listinfo/python-list

Re: how to replace line on particular line in file[no need to write it back whole file again]

[Grant Edwards]

On 2018-10-13, Dennis Lee Bieber  wrote:

>   However -- my point was that those formats were supported natively at
> the OS level, not some language utility library working on top of the basic
> streams.
>
>   A more recent (my age shows) example would be the features in DEC VMS
> Record Management Services. Again the format was handled at the system
> service level by any language -- stream, fixed length with counts, ISAM...

Yep, and when writing C programs under VMS, you had to pay attention
to which of the underlying file "formats" you were dealing with.  Only
the "stream" format worked the way that a Unix user expected.  ISTR
that "stream" wasn't commonly used by programs written in other
languages (e.g. Fortran).  I think "varible length record" was the one
that I most often ran into.  But, it's been a _long_ time since I
worked with VMS (almost 30 years)...

--
Grant




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python indentation (3 spaces)

[Grant Edwards]

On 2018-10-13, Peter J. Holzer  wrote:
>
>> For "just use tabs" to work, all of those tools would have to
>> magically recognize that they're looking at Python source and adjust
>> the tab size accordingly.  That isn't going to happen.
>
> Well, no. The idea of "just use tabs" isn't have a different tab width
> per language, but a different tab width per user.

You work in a different environment than I do.  for me, tab width
varies from one project to another (sometimes even in the same
language).  I don't get to pick just one tab width.

-- 
Grant


-- 
https://mail.python.org/mailman/listinfo/python-list